cloudtomography / at3d

Retrieves 3D cloud properties from multi-angle images of reflected solar radiation

Atmospheric Tomography with 3D Radiative Transfer (AT3D)

AT3D performs 3D reconstruction of cloud/aerosol microphysical properties from multi-angle, multi-spectral solar reflected radiation using a non-linear optimization procedure [1,2,3]. The core radiative transfer routines are sourced from the Fortran SHDOM (Spherical Harmonic Discrete Ordinate Method for 3D Atmospheric Radiative Transfer) code by Frank Evans [4,5].

The python package was created by Aviad Levis, Amit Aides (Technion - Israel Institute of Technology) and Jesse Loveridge (University of Illinois). Code contributions have been made so far by Linda Forster and Vadim Holodovsky.

Usage

The AT3D software is built around SHDOM, which is freely distributed online. Please contact Frank Evans if you have concerns about the licensing of his code as it appears in this package. This package (AT3D) is distributed under the GNU General Public License (see the LICENSE file). If you want to acknowledge the use of this repository in a publication (e.g. scientific journal article), then please cite the appropriate release, or the most recent release, which is available at the following DOI. See the CITATION.cff file for how to reference this repository.

If you want to acknowledge the scientific origin of a particular feature of this software in a publication, then please cite the appropriate journal or conference articles in which the feature originates [1,2,3]. If you use a feature of this repository that is not published elsewhere but is scientifically important to your study then please reach out to discuss the possibility of co-authorship. See the Contact section below.

This work relies on the generosity of Frank Evans in making his code publicly available, for which we are very grateful. Please acknowledge his work appropriately. In particular, use of the SHDOM solver as a part of the AT3D software in a scientific work should cite [5].

Any publications using the synthetic les clouds in the ./data/synthetic_cloud_fields/jpl_les directory which is included in the distribution must cite the following work [7].

Contact

If you find this package useful and/or would like to contribute code please let us know: [email protected]; [email protected].

 

Features

Forward (RTE solver):

AT3D is a python wrapper for polarized SHDOM and can be used to compute radiative quantities for a variety of atmospheric configurations. The key features of polarized SHDOM are included

1. Solar/Thermal/Combined sources
2. A variety of (spatially variable) surface BRDFs
3. Vector or scalar radiative transfer.
4. Open or periodic boundary conditions.

Note that each RTE solution is serial (unlike SHDOM) but independent wavelengths and pixel radiance calculations are parallelized using either MPI or a multi-threading shared memory framework. Other key features that are implemented are:

• Several sensor configurations (e.g. Perspective, Orthographic) and arbitrary observation geometries.
• Mie & Rayleigh scattering optical property calculations including OPAC aerosols. Optical properties of other species (e.g. non-spherical ice or aerosol or absorbing gases) can be included but must be calculated externally.
• Microphysical/optical properties can be generated or be read from netCDF or the SHDOM/I3RC file format.

Inverse (remote-sensing):

AT3D recovers microphysical/optical properties of atmospheric constituents that fit measured radiances. In contrast to most availble codes, AT3D can recover 3D variable atmospheric properties. This is achieved by local optimization procedures which employ an approximation to the Frechet derivatives of the RTE developed by Levis et al. [3].

Future Improvements

Future improvement include:

• Add additional sensor geometries (cross-track scan, push-broom).
• Parallelize RTE solution with MPI.
• Include retrieval of surface BRDF.

To contribute to the development effort, contact us! see Contact section above.

 

Updates in AT3D 4.0

• Data is represented using xarray objects.
• Wide field of fiew radiances are now modeled.

 

Installation

Compilation of this package requires Fortran & C compilers (e.g. GCC 9.3.0_1) to be available and correctly linked. Installation has been tested on Mac and Linux using using anaconda package management.

The treatment of legacy Fortran code has changed from GCC 9.X to 10.X+ so currently there is a flag in the setup.py script which needs to be commented if trying to install using GCC 9.X or earlier versions. The flag is extra_f77_compile_args=["-fallow-argument-mismatch"]. There is additional discussion of this point written as comments in setup.py but please feel raise an issue or discussion if you run into issues with the compiler version. The default version of the package should compile with GCC v11.3.

Clone the repository into your local machine

git clone https://github.com/CloudTomography/AT3D.git
cd AT3D

Start a clean virtual environment and setup environment variables

conda create -n at3d python=3.10.4
conda activate at3d

Install required packages

pip install -r requirements.txt

Install AT3D distribution. This should be run from within the folder containing setup.py. For development mode add the flag -e.

pip install .

 

Running Tests

After successful installation, run the tests using Python's nosetests package and make sure they all succeed:

cd tests
nose2 -v

This command will execute all files starting with test_*.py. To execute only one specific test file, test.py use

nose2 -v test

Don't panic if the tests involving the SHDOM solver has an error (Verify_Solver.test_solver). This test uses a static reference generated on a particular computer with a particular compiler. Due to the use of the adaptive grid, small numerical differences due to differences in computer systems may be amplified to a level that tests can fail. Run the test again with the verbose option (nose2 test_shdom -vv) and check to see whether the number of grid points are the same between the reference and the test.

If they are the same but the test still fails then there is problem and it might be worth regenerating the static reference on your own machine using the original SHDOM code. The input data are supplied in the tests/data/ folder as shdom_verification_polarized. The test uses custom output from SHDOM which requires modifying the original distribution. If you have trouble reproducing this test and have some questions then please raise an issue on GitHub or email me ([email protected]).

 

Basic usage

For basic usage follow the following jupyter notebook tutorials under the notebooks directory:

• MakeMieTablesExample
• MakePolydisperseMie
• MakeOpticalProperties
• MakeSensors
• SolveRTE
• SimulatingRadiances
• SimpleInverseProblem

 

Main scripts

For generating rendering and optimization scripts see the list below. The scripts folder contains another readme file with examples of how to run each script. TODO

at3d's Issues

About the question to create optical properties for ice particles

I wonder whether I can create optical properties dataset for ice particles which is not directly supported by at3d like how can I create xarray dataset for non-spherical particles when I have obtained single scattering albedo and extinction data externally?

Error raised on negative phase function for tabulated phase function

I tried to use phase function tables outside and I have already converted the original phase functions to the Wigner expansion by following the example of 'convert_libradtran_phase_to_at3d.py'. However, it still raised this error and I don't know whether it is related to the issue that I could not convert phase functions successfully.

segfaults in core.optical_depth

Occasional segmentation faults when calling core.optical_path. Issue may also affect space_carve and min_optical_path.

Array shape error in phase function mixing in gradient

maxnmicro is used as shape for optical property derivatives instead of deriv_max_num_micro. Need to fix all related shapes.

https://github.com/CloudTomography/AT3D/blob/bcb15626cec1d21996ea5213fc2c99c36b9a27a1/at3d/solver.py#LL1342C37-L1342C37

This issue can be sidestepped by using species that all have the same maxnmicro.

Segfault related to NPHI

When NPHI is a bit larger than 2 * NUM_MU_BINS there are segfaults when solution_iterations is called before the completion of one iteration. This was tested for NUM_MU_BINS = 8 and NUM_MU_BINS = 16 in 1D and 3D modes with adaptive grid splitting.

I am guessing it is due to this misspecification of the size of some working array in the discrete ordinates integration.

Gradient bugs

Some gradient tests fail with the appearance of NaNs in the gradient. Seems to be linked to boundary columns. Use with caution till fixed.

Could not locate fortran compiler

During the install process, the wrong fortran compiler might be selected. E.g., a defunct compiler like g77 might be selected as the fortran77 compiler (when it should be gfortran). If this happens, or the compilation command fails with exit status 127 (command not found), then help might be required to find the correct compiler.

Assuming that GCC is installed, the gfortran compiler for fortran77 (F77), fortran 90 (F90), which process the .f and .f90 files, respectively, can be selected by running
export F77=gfortran, and/or export F90=gfortran prior to running pip install .

If running gofrtran --version produces no output, then you have other problems with the install/linking of the compiler.

Don't use boundary grid points for optimization

Very small inconsistencies in the calculation of the sensitivity of the direct beam optical path to the extinction, typically in boundary voxels. Until this is resolved, I suggest not using these grid points for retrievals.

The errors includes some negative values in the predicted sensitivity. Currently, the tests for the direct beam fail but the accuracy loss is very small (maximum_error < 1e-4) for typical values ranging two orders of magnitude larger. Derivative calculations within the volume should still meet the specified accuracy requirement in the tests.

Import Error on Mac M1 (arm64)

AT3D installs but fails to import on Mac with M1 chip with the error in the screenshot.

Another user had a similar issue that was found to be that anaconda was not the arm64 version, but instead the x86_64 version. They suggested the following fix:

To find out what you have for python, I would try the following command:
(first activate the python environment that you are using, then type)

file -L which python3

This should tell you what kind of binary anaconda is. If it is x86_64, then I’d recommend installing the arm64 version of anaconda.

I would delete whatever folder anaconda is installed in. Typically it is ~/opt/anaconda3. With anaconda, all the envs are contained within that directory. If you created any envs with pip, which puts the in the folder of whatever project you are working on, you'll have to remove them separately. Then reinstall the correct version of anaconda and recreate your envs from scratch.

should this be just a warning?

https://github.com/CloudTomography/pyshdom/blob/f6b00e57b95c338456947ec582db4ebc25047ea3/pyshdom/solver.py#L1587

If this should be a warning, maybe we should add an argument to check_range: raise='exception' with the options 'exception'/'warning'?

Sensor list saving in seperate util function

https://github.com/CloudTomography/pyshdom/blob/6215c180ad2e55ed3193511cdf02b53c73e6d398/pyshdom/util.py#L615

Enhancement:
Currently when working on real data it would be useful to have a function just to save the sensor list without needing to create the solver list.

Bug:
I don't follow the logic here - there is an if/else condition with the identical consequences.

gradient call hangs with thermal with delta-M scaling

Not sure why at this point.

Tests fail because `np.int` and `np.float` were deprecated

Hello,

Running the tests in tests/ produces a few errors like the following:

======================================================================
ERROR: _ErrorHolder
----------------------------------------------------------------------
Traceback (most recent call last):
  File "at3d\AT3D\tests\test_derivatives.py", line 838, in setUpClass
    solvers, Sensordict,cloud_poly_tables,final_step,rte_grid = cloud_solar(mie_mono_table,ext,veff,reff,ssalb,solarmu,surfacealb,ground_temperature,
  File "at3d\AT3D\tests\test_derivatives.py", line 726, in cloud_solar
    Sensordict.get_measurements(solvers, maxiter=200, n_jobs=4, verbose=False)
  File "at3d\at3d\at3d\containers.py", line 226, in get_measurements
    keys, ray_start_end, pixel_start_end = at3d.parallel.subdivide_raytrace_jobs(rte_sensors, n_jobs)
  File "at3d\at3d\at3d\parallel.py", line 142, in subdivide_raytrace_jobs
    render_jobs[key] = max(np.ceil(render_job/ray_count * n_jobs * job_factor).astype(np.int), 1)
  File "miniconda3\envs\at3d\lib\site-packages\numpy\__init__.py", line 338, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'int'.
`np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'inf'?

======================================================================

For both np.int and np.float, that were deprecated since numpy 1.20.0 (see https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations ).

Changing all references to python's built-in int and float solves the errors, but while being previously aliased by np.int and np.float, perhaps a lower level revision of the dtypes is necessary to make sure they are numerically appropriate?

I'm using conda 23.9.0, python 3.10.13, and numpy 1.26.2 (can't install version 1.21.2 from the requirements in python 3.10).

Thank you

Cannot compile the code on Ubuntu 18 or 16

Hi dear pyshdom developers,
I didn't succeed to compile the code due to strange Fortran compilation errors:

src/polarized/shdomsub4.f:1339.22:
INTEGER DOEXACT(IDR)
Error: Variable 'idr' cannot appear in the expression at (1)

src/polarized/shdomsub4.f:2684.30:
REAL DLEG(NSTLEG,0:NLEG,DNUMPHASE)
Error: Variable 'dnumphase' cannot appear in the expression at (1)

src/polarized/shdomsub4.f:2683.31:
REAL LEGEN(NSTLEG,0:NLEG,NUMPHASE)
Error: Variable 'numphase' cannot appear in the expression at (1)

I see that the code is OK so I guess I maybe use a "bad" gfortan version.
I followed every step in the readme.
It happens in Ubuntu 18.04.5 and Ubuntu 16.04.7.

Can you tell me what gfortan and Ubuntu version (if relevant) do you use?

Thanks

Support for density variable

https://github.com/CloudTomography/pyshdom/blob/7591db4bcac48205d19dd6b94d13fb09c88e7f3b/pyshdom/medium.py#L252
Crashed with one of the variables as density because self._exact_size_distribution_parameters does not have a density key.

Is this an intended behavior? Should only size distribution parameters and extinction be used?

Bug fixes in airmspi branch

Minor bug fixes for micro-physical optimization (should be integrated to main at some point): 5f813dd

mask.astype(np.bool) was moved to StateToGridMask.add_transoform for easier support of transforms that are not set externally as a numpy array e.g.:

state_to_grid = pyshdom.transforms.StateToGridMask()
state_to_grid.add_transform('cloud', 'density', mask.values, fill_value=0.0)
state_to_grid.add_transform('cloud', 'reff', mask.values, fill_value=10.0)

Jesse I assigned you so you could verify these bug fixes

edge case where mask and grid are not of same dimensions

https://github.com/CloudTomography/pyshdom/blob/f6b00e57b95c338456947ec582db4ebc25047ea3/pyshdom/medium.py#L1136

Segfaults in radiance calculation

Error in the subpixel ray averaging in util.f90 seems to cause segfaults. Likely linked to #7

Windows Install

The numpy version in the requirements.txt (1.21.2) is not necessarily supported on windows for python=3.10.4.

https://numpy.org/doc/stable/release/1.21.2-notes.html

If install fails you can try unofficial wheels discussed here

https://stackoverflow.com/questions/69458399/numpy-1-21-2-may-not-yet-support-python-3-10

or switch to a linux/MacOS machine.