eodms-sgdot / py-eodms-rapi Goto Github PK

Request to add feature to have a maximum number of attempts the download query will try to attempt to download. Where a timetout error is given when the maximum number of attempts is reached.
Use case: We use this script operationally and sometimes server space is needed after say x ammount of time. We need to have the option to stop the script after x ammount of time is passed so other activities can be started.

in line 1389

def download(self, items, dest, wait=60.0, attempts=5):

and in line 1447 - 1455

` attempt = 1
complete_items = []
while len(unique_items) > len(complete_items) and attempt <= attempts:
time.sleep(wait)
attempt += 1

        # start, end = self._get_dateRange(unique_items)
        # orders = self.get_orders(dtstart=start, dtend=end)
        orders = self.get_orders()`

Although most entries seem to be formed well, I have a list of entries that aren't. All of these entries are missing a processingDate. I'm not sure how to interpret a missing processingDate. For my current application I'm just dropping these entries, but I wanted to let yall know this existed in case it was in error.

{'recordId': '12717949', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12718059', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12718064', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12717945', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12717951', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12718040', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12718032', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12718021', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12718027', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12717943', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12718078', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12717942', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12717994', 'collectionId': 'RCMImageProducts', 'processingDate': ''}
{'recordId': '12718036', 'collectionId': 'RCMImageProducts', 'processingDate': ''}

e.g. PREVIEW_AVAILABLE for NAPL images doesn't work

Syntax error on line 1822 in eodms.py.

Remove the submission of search requests1000 records at a time. Instead add firstResult to search method for use by the user.

Issue

Cause

User Report

the API does not work... always same timeout problem

When I place an order and specify the packagingFormat as described in the documentation, the parameter is ignored. I Also never received an email for the order when overloading the response destination with NOTIFICATION_EMAIL_ADDRESS

...

>>> params = [
        {"packagingFormat": "TARGZ"},
        {"NOTIFICATION_EMAIL_ADDRESS": "[email protected]"}
]
>>> _ = api.order(orders, parameters=params)
>>> api.download(_, Path('output'))
| EODMSRAPI | Getting list of current orders...
| EODMSRAPI | Downloading image with Record Id 12392382 (RCM3_OK1670273_PK1761090_3_SC50MA_20210901_054854_HH_HV_GRD.zip).
| EODMSRAPI | RCM3_OK1670273_PK1761090_3_SC50MA_20210901_054854_HH_HV_GRD.zip: 100%|███████████████████| 1.28G

Intermittent authentication error when querying through EODMSRAPI.

By intermittent, I mean that a few weeks ago, I was getting an authentication error. I set it aside, did other work, but didn't change anything with regards to my computer system, installations, or EODMS account. Yesterday my query worked, but today I am getting an authentication error again. Sometimes the authentication error is preceded by timeouts; in this instance, it is not.

Throughout this, I have always been able to log in to 'the actual website, https://www.eodms-sgdot.nrcan-rncan.gc.ca/index-en.html', using the same credentials I use through EODMSRAPI, which is provided in the error message.

Occurs for all collections; 'Radarsat2RawProducts', 'Radarsat2', 'Radarsat1', and 'RCMImageProducts' are the ones I'm most interested in.

A screenshot of error messages and my roi .geojson file are attached.

This might be a separate issue, and it's something that I can work around, but - if a query fails, and I try to re-submit the query, i.e. two sequential rapi.search() queries, with a failed first attempt, no error message is provided after the second attempt. I have to re-login using rapi=EODMSRAPI(user,pass) after a failed rapi.serach() attempt in order for the new search to be sent through. This is demonstrated in the third rapi.search() attempt in the screenshot.

geojson file:

{
"type": "FeatureCollection",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
"features": [
{ "type": "Feature", "properties": { }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -83.477629563048367, 42.904063909639284 ], [ -81.0, 42.904063909639284 ], [ -81.0, 41.382130631893006 ], [ -83.477629563048367, 41.382130631893006 ], [ -83.477629563048367, 42.904063909639284 ] ] ] } }
]
}

When I am making api calls and there is a "connection reset by peer" the library code continues to execute, resulting in a non-informative error from inside the library. The error 'NoneType' object is not subscriptable is underspecified, making error handling very difficult.

Traceback (most recent call last):
  File "/Users/philiprobinson/git/sar-processor/pkg/gfwsar/eodms/rcm/meta/_api.py", line 212, in <module>
    test()
  File "/Users/philiprobinson/git/sar-processor/pkg/gfwsar/eodms/rcm/meta/_api.py", line 206, in test
    filepath = run(gc, first, last, destination)
  File "/Users/philiprobinson/git/sar-processor/pkg/gfwsar/eodms/rcm/meta/_api.py", line 140, in run
    paths = gather_data_csvs(gc, first, last, destination)
  File "/Users/philiprobinson/git/sar-processor/venv/lib/python3.9/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/Users/philiprobinson/git/sar-processor/venv/lib/python3.9/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/Users/philiprobinson/git/sar-processor/venv/lib/python3.9/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/Users/philiprobinson/git/sar-processor/venv/lib/python3.9/site-packages/six.py", line 719, in reraise
    raise value
  File "/Users/philiprobinson/git/sar-processor/venv/lib/python3.9/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/Users/philiprobinson/git/sar-processor/pkg/gfwsar/eodms/rcm/meta/_api.py", line 116, in gather_data_csvs
    raw = gather_raw(api, start, end)
  File "/Users/philiprobinson/git/sar-processor/pkg/gfwsar/eodms/rcm/meta/_api.py", line 53, in gather_raw
    api.search(
  File "/Users/philiprobinson/git/sar-processor/venv/lib/python3.9/site-packages/eodms_rapi/eodms.py", line 2813, in search
    params['query'] = self._parse_query(filters, features, dates)
  File "/Users/philiprobinson/git/sar-processor/venv/lib/python3.9/site-packages/eodms_rapi/eodms.py", line 910, in _parse_query
    field_id = self._get_field_id('Acquisition Start Date')
  File "/Users/philiprobinson/git/sar-processor/venv/lib/python3.9/site-packages/eodms_rapi/eodms.py", line 619, in _get_field_id
    fields = self.get_available_fields(name_type='all')[field_type]
TypeError: 'NoneType' object is not subscriptable

When I run this code to search the RCMImageProducts collection for catalogs with the Medium Resolution 30m beam mode type

from eodms_rapi import EODMSRAPI
rapi = EODMSRAPI(user_name, user_pass)
searchFilters = {"Beam Mode Type": ("=", ["Medium Resolution 30m"])}
searchCollection = "RCMImageProducts"
rapi.search(
    collection=searchCollection,
    filters=searchFilters,
    dates=[{'start': '20200825_195108', 'end': '20230901_195108'}],
    max_results=5
)
res = rapi.get_results("full")

beamModeTypes = [x['beamModeType'] for x in res]
print(beamModeTypes)

beamModeTypes: ['Low Resolution 100m', 'Low Noise', 'Medium Resolution 16m', 'Low Noise', 'Low Resolution 100m']

I get results back but their beam mode type do not match what I filtered for.

IMPACT

Get HTTP-400 errors if don’t use stringified integer RecordIds

Network interruption when downloading causes Python API to fail. The order is still processed and sent to my email to download via other means.

Queries/Thoughts:

1. Perhaps I should be only passing one productID at a time instead of filtered search and check for failure? Through it would be great if the downloader would recover from network interruptions i.e. I disconnect from vpn.
2. If the downloading can not recover, then the order should be cancelled. I mistakenly restarted the script and had two identical orders waiting in my email. I do not want to consume more resources than necessary from the system.

Thanks so much for this API!

@keballantyne, Can you kindly update the readme of this repo upgrade instructions i.e. pip install py-eodms-rapi -U. Thanks.

Issue

In the below example, the EODMS REST API is returning the wrong url in the destinations. The one in manifest is right.

Impact

After submitting an order, a single image from that order is repeatedly downloaded and no other images in the order are downloaded.

Expectation

Construct the url from manifest and destinations until EODMS is fixed.

Sample

{
  "items": [
    {
      "orderId": 188705,
      "manifest": {
        "2e720468-c128-4cf3-8c71-22e75e2325e9/2239431/RCM2_OK2195019_PK2421176_1_SC50MB_20230119_145532_HH_HV_GRD.zip": "1050308566"
      },
      "destinations": [
        {
          "stringValue": "<a href='https://data.eodms-sgdot.nrcan-rncan.gc.ca/rcm/carts/6f641a71-e6c7-4ca4-8070-f0b22bdf805f/1461195/0afeadf1-cfb0-431a-8fb4-abdb3f50364a'>https://data.eodms-sgdot.nrcan-rncan.gc.ca/rcm/carts/6f641a71-e6c7-4ca4-8070-f0b22bdf805f/1461195/0afeadf1-cfb0-431a-8fb4-abdb3f50364a</a></br>",
          "type": "Download"
        }
      ],
...

The following error occurs in eodms.py when logging the download of an image:
(SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape)

This error occurs when the image is downloaded to the C:\Users folder.

Issue with muilt-geometries not working for files without the GDAL python package.

The following error occurs when query NAPL images:

Traceback (most recent call last):
  File "eodms_cli.py", line 1680, in cli
    prmpt.prompt()
  File "eodms_cli.py", line 1214, in prompt
    self.eod.search_order_download(self.params)
  File "C:\GRA\RaD\10-satellitePython\eodms-cli\scripts\utils.py", line 1601, in search_order_download
    max_images=max_images)
  File "C:\GRA\RaD\10-satellitePython\eodms-cli\scripts\utils.py", line 1309, in query_entries
    result_fields, max_images)
  File "C:\Users\frase\miniconda3\envs\satellite_env\lib\site-packages\eodms_rapi\eodms.py", line 3091, in search
    params['query'] = self._parse_query(filters, features, dates)
  File "C:\Users\frase\miniconda3\envs\satellite_env\lib\site-packages\eodms_rapi\eodms.py", line 1233, in _parse_query
    if val[0].lower().find('t') > -1 \
AttributeError: 'bool' object has no attribute 'lower'

Attempting to replicate EODMS GUI search via Python API.

Can query collections and available fields and choices.

Running into issue when attempting to use geometry filter (was not able to run without a geometry filter). The spatial predicate within or intersects seems to not be parsed correctly; tracebacks refer to the second letter of each predicate I have tried. Attempted to contain predicate in a list but that failed too.

Windows 11 running WSL 2 Ubuntu 22.04 with Python 3.11. Latest install of py-eodms-rapi.

import os

from dotenv import load_dotenv
from eodms_rapi import EODMSRAPI

load_dotenv()

username=os.getenv("EODMS_EMAIL")
password=os.getenv("EODMS_PASSWORD")

rapi = EODMSRAPI(username, password)

#%%
desired_product = 'WorldView-3'

# List all products available to user
#print(rapi.get_collections(as_list=True))

dates = [{"start": "20230501_000000", "end": "20230823_000000"}]
filter = {"Sensor Mode": ["PSH"]}

geographic_area = ("intersects", {"type": "Polygon",
        "coordinates": [
          [
            [
              -80.98348190613918,
              70.55936098953137
            ],
            [
              -128.08040511581584,
              73.66999943625751
            ],
            [
              -138.38572103373428,
              68.87269436214862
            ],
            [
              -124.16519435207523,
              67.7870395850222
            ],
            [
              -114.69037016788643,
              66.69986515771265
            ],
            [
              -99.25854956956725,
              66.11648758438096
            ],
            [
              -83.04517551683422,
              65.48014739072471
            ],
            [
              -75.59319320583784,
              67.65281907013141
            ],
            [
              -80.98348190613918,
              70.55936098953137
            ]
          ]
        ]
                              
      }
)

# List all fields and choices
#print(rapi.print_results(rapi.get_available_fields(desired_product)))
#print(rapi.print_results(rapi.get_field_choices(desired_product)))

# Submit a search
rapi.search(desired_product, filters=filter, features=geographic_area, dates=dates)
results = rapi.get_results()
# Pretty print of results for human reading:)
rapi.print_results(results)

Snippet

from eodms_rapi import EODMSRAPI

# Create the EODMSRAPI object
rapi = EODMSRAPI('eodms-username', 'eodms-password')

# Add a point to the search
feat = [('intersects', "POINT (-96.47 62.4)")]

# Create a dictionary of query filters for the search
filters = {'Beam Mnemonic': ('=', ['16M11', '16M13']), 'Incidence Angle': ('>=', '35')}

# Set a date range for the search
dates = [{"start": "20200513_120000", "end": "20200613_150000"}]

# Submit the search to the EODMSRAPI, specifying the Collection
rapi.search("RCMImageProducts", filters=filters, features=feat, dates=dates)

# Get the results from the search
res = rapi.get_results('full')

# Print results
rapi.print_results()

Cause

Exception has occurred: TypeError
sequence item 6: expected str instance, NoneType found
File "C:\code.py.", line 36, in <module> res = rapi.get_results('full')

The package cannot convert multi-geometry (MultiPoint, MultiLineString, MultiPolygon) to separate features for use in the RAPI.

The following error can occur when using the py-eodms-rapi:

File "/home/kb/.local/lib/python3.6/site-packages/eodms_rapi/geo.py", line 473, in get_features driver = ogr.GetDriverByName(ogr_driver) AttributeError: module 'ogr' has no attribute 'GetDriverByName'

For Special Handling Required, querying 'true' returns a Bad Request error.

Example of Python code:

filters = {'Special Handling Required': ('=', 'true')}

I wanted to try using the API by first searching products (dates and coordinates), and then making an order of these product. So, I first tried using the concerned example in the documentation (here : https://py-eodms-rapi.readthedocs.io/en/latest/). But it doesn't run correctly. Here is the returned error :

Then I tried to replace 'filters' by None and it runs correctly.