loicdtx / lsru

:mag: :globe_with_meridians:Query and Order Landsat Surface Reflectance data via ESPA

Landsat-ESPA-util

Interface to USGS and ESPA APIs for Landsat surface reflectance data ordering

Online doc at lsru.readthedocs.io

Before, downloading Landsat surface reflectance data for a given area meant:

• Manually querying the sceneIDs on Earth Explorer
• Saving these lists of sceneIDs to text files
• Manually uploading these files to ESPA to place the order
• Downloading the processed data with a download manager

Now, thanks to USGS API, and espa API it can all be done programtically.

Example

Send a spatio-temporal query to the Usgs API

Used to retrieve a list of available scenes metadata

from lsru import Usgs
import datetime

# Define query extent
bbox = (3.5, 43.4, 4, 44)

# Instantiate Usgs class and login
usgs = Usgs()
usgs.login()

# Query the Usgs api to find scene intersecting with the spatio-temporal window
scene_list = usgs.search(collection='LANDSAT_8_C1',
                         bbox=bbox,
                         begin=datetime.datetime(2013,1,1),
                         end=datetime.datetime(2016,1,1),
                         max_results=10,
                         max_cloud_cover=40)

# Extract Landsat scene ids for each hit from the metadata
scene_list = [x['displayId'] for x in scene_list]

Place a processing order to Espa

The scene list can be used to send a processing order to Espa via the Espa API. Many options are available (full scene, pixel resizing, reprojection, cropping).

Order full scenes

from lsru import Espa
from pprint import pprint

# Instantiate Espa class
espa = Espa()

# Place order (full scenes, no reprojection, sr and pixel_qa)
order = espa.order(scene_list=scene_list, products=['sr', 'pixel_qa'])
print(order.orderid)

# [email protected]'

Check current orders status

for order in espa.orders:
    # Orders have their own class with attributes and methods
    print('%s: %s' % (order.orderid, order.status))

# [email protected]: ordered
# [email protected]: complete
# [email protected]: complete
# [email protected]: complete
# [email protected]: complete

Download completed orders. When Espa finishes pre-processing an order, its status changes to complete, we can then download the processed scenes.

for order in espa.orders:
    if order.is_complete:
        order.download_all_complete('/media/landsat/download/dir')

It is also possible order processing with reprojection, cropping, resizing, etc

# Inspect aea projection parameters
pprint(espa.projections['aea'])
# Define projection parameters
proj_params = {'aea': {'central_meridian': 3.8,
                       'datum': 'wgs84',
                       'false_easting': 0,
                       'false_northing': 0,
                       'latitude_of_origin': 43.7,
                       'standard_parallel_1': 43,
                       'standard_parallel_2': 44}}
# Place order
order_meta = espa.order(scene_list=scene_list, products=['sr', 'pixel_qa'],
                        note='cropped order with resampling', projection=proj_params,
                        extent=bbox, resolution=60)

Installation

Activate a virtualenv (optional but preferable) and run:

pip install lsru

Setup

The package requires a configuration file in which usgs credentials are written. By default the file is called ~/.lsru (this can be modified if you want to join this configuration with the configuration of another project) and has the following structure.

[usgs]
username=your_usgs_username
password=your_very_secure_password

Why can't I just retrieve my Landsat data from Earth Explorer, Amazon or Google cloud?

You can but it will be top of atmosphere (TOA) radiance, not surface reflectance. If you aim to have a scientific use of the data, you probably want to have surface reflectance.

The critical part between TOA radiance and surface reflectance is the atmospheric correction. That means that surface reflectance data are corrected for atmospheric effects, therefore providing accurate measurements of the target's spectral properties.

Today (October 2016), there are a few ways to obtain Landsat surface reflectance data (All of them have been processed by LEDAPS, the reference high level Landsat processing tool):

1. By ordering them via the ESPA system

• this is what this utility helps you to do
• ESPA does on demand pre-processing of full (reprojected) scenes or subsets
• A cloud mask (fmask) and vegetation indices can also be added to the order

1. By ordering them from Google Earth Engine

• GEE has ingested the entire ESPA surface reflectance collection to make it available via its platform

1. By downloading TOA data from any source and processing with a local installation of LEDAPS

• Not necessarily trivial

lsru's Issues

conf not passed to Order when file in not standard location

Espa.order() returns an instance of Order. conf= is not explicitely passed so that it fails when config file is in non standard location

earthexplorer json search deprecation

Looks like usgs API is about to be deprecated (by December 2020)
See migration guide to switch to v1.5

Handle date restricted property in available-products results

The following query to available-products

curl  --user username:password \
                  -X GET \
                  -d '{"inputs":["LE07_L1GT_020049_20160611_20161209_01_T2",
                                      "LE07_L1GT_021049_20200105_20200131_01_T2",
                                      "LE07_L1TP_021049_20200206_20200303_01_T1"]}' \
                        https://espa.cr.usgs.gov/api/v0/available-products

returns

{"etm7_collection": {"products": ["source_metadata", "l1", "toa", "bt", "stats", "pixel_qa", "et"], 
                                "inputs": ["LE07_L1GT_020049_20160611_20161209_01_T2", "LE07_L1GT_021049_20200105_20200131_01_T2", "LE07_L1TP_021049_20200206_20200303_01_T1"]},
                                "date_restricted": {"sr": ["LE07_L1GT_020049_20160611_20161209_01_T2"],
                                                             "sr_ndvi": ["LE07_L1GT_020049_20160611_20161209_01_T2"],
                                                              "sr_evi": ["LE07_L1GT_020049_20160611_20161209_01_T2"],
                                                              "sr_savi": ["LE07_L1GT_020049_20160611_20161209_01_T2"],
                                                              "sr_msavi": ["LE07_L1GT_020049_20160611_20161209_01_T2"],
                                                              "sr_ndmi": ["LE07_L1GT_020049_20160611_20161209_01_T2"],
                                                              "sr_nbr": ["LE07_L1GT_020049_20160611_20161209_01_T2"],
                                                              "sr_nbr2": ["LE07_L1GT_020049_20160611_20161209_01_T2"]}}

A query without date restricted scenes return

curl  --user username:password \
                  -X GET \
                  -d '{"inputs":["LE07_L1GT_021049_20200105_20200131_01_T2", 
                                 "LE07_L1TP_021049_20200206_20200303_01_T1"]}' \
                        https://espa.cr.usgs.gov/api/v0/available-products

{"etm7_collection": {"products": ["source_metadata", "l1", "toa", "bt", "sr", "sr_ndvi", "sr_evi", "sr_savi", "sr_msavi", "sr_ndmi", "sr_nbr", "sr_nbr2", "stats", "pixel_qa", "et"], "inputs": ["LE07_L1GT_021049_20200105_20200131_01_T2", "LE07_L1TP_021049_20200206_20200303_01_T1"]}}

So products appear either in products or in date_restricted, but not in both.
What need to be done is; if any of the products ordered appear in date_restricted, it needs to be removed from the inputs

Improve warning when a scene is not downloaded

Right now, it says Missing scene downloaded! when the downloader hits an un-completed scene.

Ideas:

• Print ESPA justification of incompleteness instead
• ...

PathRow and Sensor as Parameters

Allow a call to send a path/row in lieu of a BBox as well as specify sensor (LS4, LS5, LS7, LS8). I built this same exact solution in NodeJS and love that I found a Python alternative. I also added in the Search capability to find the scenes to order from https://earthexplorer.usgs.gov/inventory/json/v/1.4.0/search.

Catch failed requests, filter list and re-send

Some requests fail because of scenes not available on espa.

The json returned in r.text:

{
 "message": {
   "Inputs Not Available": [
     "LT52310672004112CUB00"
   ]
 },
 "status": 400
}

Remote sensing

Hi do you can guide me please

More of a comment: 429 Errors

For downloading, if you try to download more than one scene at a time, you get a 429 Error code and a 162 byte download. How does this library handle retry for that situation?