arthur-e / unmixing

Interactive tools for spectral mixture analysis of multispectral raster data in Python

Interactive Unmixing Tools

Overview

This is a library of interactive tools and functions for performing linear spectral mixture analysis (LSMA) and spatially adaptive spectral mixture analysis (SASMA). It supports parallel fully constrained least-squares (FCLS) mixture analysis over multiple processes, allowing for very efficient mapping of endmember abundances, both in the spatially adaptive approach and in regular LSMA. In detail, the unmixing module includes tools and functions to support:

• Stacking raster bands from any GDAL file format, including HDF;
• Applying a mask or a combination of masks to a raster;
• Generating raster masks from saturated pixels or using density slicing (e.g., to mask water pixels);
• Applying CFMask output and other quality assurance layers from the USGS;
• Band-wise compositing of reflectance (or fraction/ abundance images);
• Tasseled cap transformation and other indices including NDVI and RNDSI;
• Dimensionality reduction through the Minimum Noise Fraction (MNF);
• Radiometric rectification of a raster time series;
• Visualizing the mixing space of a moderate resolution raster;
• Interactive selection of pixels in the mixing space, which are then captured in a KML file;
• Endmember induction and plotting of endmembers for linear spectral mixture analysis (LSMA);
• Fully constrained least squares (FCLS) unmixing;
• Spatial interpolation of endmembers for spatially adaptive spectral mixture analysis (SASMA);
• Learning of endmember candidates for SASMA using classification and regression trees (CART);
• Validation of unmixing through a forward model of reflectance;

For an overview and tutorial on how to use this library, check out the iPython Notebook(s) in the docs/ folder:

• <https://github.com/arthur-e/unmixing/blob/master/docs/Overview_LSMA_in_Python.ipynb>
• <https://github.com/arthur-e/unmixing/blob/master/docs/Example_Spatially_Adaptive_Spectral_Mixture_Analysis_SASMA.ipynb>

Installation and Setup

Because this is a scientific library, there are complex dependencies that may be difficult to install. For GNU/Linux systems, particularly Ubuntu, look at install.sh for a guide on installing the system dependencies required for the Python dependencies.

At this time, installation is intended for development purposes only. As such, unmixing should be installed in "editable" mode using pip; see <https://packaging.python.org/distributing/#working-in-development-mode>. From the unmixing directory, where setup.py is found:

$ pip install -e .

Dependencies

• numpy
• scipy
• matplotlib
• pysptools
• GDAL
• pykml
• scikit-learn (For SASMA)

Use System Dependencies

Many of the core packages, particularly NumPy and SciPy, have wide adoption and use. They also take a lot of time and clock cycles to compile into a virtual environment. Consequently, it is recommended that these libraries be installed globally (system-wide).

Other packages should be installed only in the virtual environment. To use both system-wide and local packages within a virtual environment, the virtual environment must be set up with the --system-site-packages option:

sudo apt-get install python3-numpy python3-scipy python3-matplotlib python3-zmq
virtualenv -p /usr/bin/python3.5 --system-site-packages <virtualenv_path>

The local packages can be installed within the virtual environment via pip:

source /my/virtualenv/bin/activate
pip install -r REQUIREMENTS

Documentation

Python Dependencies

Some resources on the dependencies can be found:

• matplotlib <http://matplotlib.org/contents.html>
• PySptools <http://pysptools.sourceforge.net/index.html>
• PySptools on PyPI <https://pypi.python.org/pypi/pysptools/0.13.0>

Third-Party Tools

Documentation on related (but not required) third-party tools can be found:

• Fmask <https://code.google.com/p/fmask/>
• CFmask <https://code.google.com/p/cfmask/> (A C version of Fmask)
• LEDAPS <https://code.google.com/p/ledaps/>

Foundations

Hyperspectral Imaging (HSI) Cubes

Many of the tools in this library are designed to work with HSI cubes. While the raster arrays read in from GDAL are p x m x n arrays, HSI cubes are n x m x p arrays, where m is the row index, n is the column index, and p is the spectral band number. It is important to note that the row index n corresponds to the latitude while the column index m corresponds to the longitude. Thus, the coordinates (n, m) can be converted directly to a longitude-latitude pair.

unmixing's Issues

visualize.FeatureSpace has RuntimeError with plot_feature_space() when colorbar used

from unmixing.visualize import FeatureSpace
vis_fs = FeatureSpace(path = '/home/arthur/Desktop/detroit.tiff', nodata = -9999,
                      feature_limit = None)
vis_fs.plot_feature_space(c = 2) # Anytime `c` is set

RuntimeError: No mappable was found to use for colorbar creation. First define a mappable such as an image (with imshow) or a contour set (with contourf).

Vectorize forward model of reflectance

Currently, the function to predict reflectance from abundances operates on an entire matrix.

https://github.com/arthur-e/unmixing/blob/master/unmixing/lsma.py#L523

It's pretty fast, but I think it would be more appropriate to vectorize this function. It can then be applied against only the subset of samples that are used in calculating the RMSE between actual and predicted reflectance.

https://github.com/arthur-e/unmixing/blob/master/unmixing/lsma.py#L621

visualize.FeatureSpace doesn't work well with MNF-transformed data

Though visualize.FeatureSpace can accept MNF-transformed data (by setting transform=False to indicate the data is already transformed), the resulting plots of the mixing space don't look right... I think that something is going wrong with the indexing (e.g., indexing on m,n axes instead of the p-axis).

visualize.FeatureSpace needs setters for raster data, spatial ref. info

Currently, visualize.FeatureSpace() expects to be given a path to a raster file, which it then reads in, setting its internal data structure and attributes. Need to provide setters so that:

• An existing gdal.Dataset or np.ndarray (with gt and wkt also provided) can be used to initialize a FeatureSpace instance;
• Either an MNF-transformed or non-transformed raster/ array can be provided and plot_spectral_profile() still works;

Update lsma.FCLSAbundance, other classes with best practices from management module

I wrote a lot of extensions to *Abundance clases in the missing management module; translate these into methods on the FCLSAbundance or NNLSAbundance classes.

unable to view or download the Overview_LSMA_in_Python.ipynb

I am unable to view Overview_LSMA_in_Python.ipynb in github or even open the downloaded file in jupyter. I want to try this out

the work in this session https://github.com/arthur-e/unmixing/blob/master/docs/Overview_LSMA_in_Python.ipynb tells how to process hdf image i have .hdr image which is already stacked but if use that i couldnt move on to next step so i converted that .hdr image to .tiff format after that i couldnt stack the image and get the desired output. how to resolve this issue.?? If i get the mail ID i can send the datas which have so that you can verify it once. I have attached the data which i have in Re: [arthur-e/unmixing] visualize.FeatureSpace doesn't work well with MNF-transformed data (#2) in which you have mailed me. Also i can mail it again to get this issue resolved

Update installation procedure with distutils

Time to figure out how to do this again!

Switch to using affine transformations with affine

Long ago, I wrote functions like xy_to_pixel() that do the job that is now seamlessly accomplished by a nice new-ish library, affine. See on Github.

Classification with GIS data on endmember locations

I am wondering is it possible to implement soil classification with SASMA, where the classification parameters are parcels in .SHP/.GeoJSON format?

Installation issue

How to install the unmixing library in window 10?