asfadmin / discovery-asf_search Goto Github PK
View Code? Open in Web Editor NEWLicense: BSD 3-Clause "New" or "Revised" License
discovery-asf_search's People
Stargazers
Watchers
Forkers
jhkennedy jondho dibit123 sugeradi985 sirbastiano gitter-badger nmoghaddam timatcambrio geogubd selabinc spicygarlicalbacoreroll jiangyuinsar oliveiralenon yenyi-wu nickohl invisiblenzw dfsanchezvi scalet98 luciananieto amanjainblr carmine1990 shahmed huayue21 geonamsoil efall19 sophiezang boltayevjahongir pinsar-py ssharifi1994 ceciliamasemola kevinxmorales chrissot lpcalderonc scottstanie lpsienes akashspunnayil nikojj ganduan zachhoppinen akshaykk12discovery-asf_search's Issues
JSON Decode Error in geo_search
Hi all, I'm getting a JSON decode error (see below) when trying to use asf_search.geo_search().
Here's a formerly working MWE:
import asf_search as asf
dct_kw = {'platform': 'Sentinel-1', 'processingLevel': 'SLC', 'relativeOrbit': [4], 'frame': None, 'start': None, 'end': None, 'lookDirection': None, 'intersectsWith': None}
asf.geo_search(**dct_kw)
Tested on Linux and MacOS, with current asf_search conda version v3.0.6.
I've also tested with my previous version, but this also fails, suggesting something changed upstream of the asf_search routines.
(P.S. I'm not sure what asf_search version I had previously working, as asf.__version__ returns 0.0.0.
Thanks in advance.
Traceback (most recent call last): File "/home/buzzanga/BB_LIB/bin/ASF_DL-search.py", line 238, in <module> Downloader(inps)() File "/home/buzzanga/BB_LIB/bin/ASF_DL-search.py", line 161, in __call__ scenes = self.query_asf() File "/home/buzzanga/BB_LIB/bin/ASF_DL-search.py", line 232, in query_asf scenes = asf.geo_search(**dct_kw) File "/u/leffe-data2/buzzanga/Miniconda3/envs/FRInGE/lib/python3.9/site-packages/asf_search/search/geo_search.py", line 61, in geo_search return search(**data) File "/u/leffe-data2/buzzanga/Miniconda3/envs/FRInGE/lib/python3.9/site-packages/asf_search/search/search.py", line 147, in search raise ASFSearch5xxError(f'HTTP {response.status_code}: {response.json()["error"]["report"]}') File "/u/leffe-data2/buzzanga/Miniconda3/envs/FRInGE/lib/python3.9/site-packages/requests/models.py", line 910, in json return complexjson.loads(self.text, **kwargs) File "/u/leffe-data2/buzzanga/Miniconda3/envs/FRInGE/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/u/leffe-data2/buzzanga/Miniconda3/envs/FRInGE/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/u/leffe-data2/buzzanga/Miniconda3/envs/FRInGE/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
IPF Versions for Sentinel-1 SLC
Is your feature request related to a problem? Please describe.
Want IPF versions (in CMR) in asf search
Describe the solution you'd like
From @asjohnston-asf:
- Echo10: https://cmr.earthdata.nasa.gov/search/granules.echo10?provider=ASF&granule_ur=S1A_IW_SLC__1SDH_20230208T083928_20230208T083955_047142_05A7F0_0F03-SLC
- umm-json: https://cmr.earthdata.nasa.gov/search/granules.umm_json?provider=ASF&granule_ur=S1A_IW_SLC__1SDH_20230208T083928_20230208T083955_047142_05A7F0_0F03-SLC
Additional context
Two different IPF versions for different dates cannot be used to generate an interferogram.
[Bug] Specific datetime fails search queries on sentinel-1
Describe the bug
asf_search.exceptions.ASFSearch4xxError thrown when asf.search(platform=asf.SENTINEL1, start=start, end=end) spans datetime.datetime(year=2021, month=12, day=7, hour=10, minute=55)
To Reproduce
Steps to reproduce the behavior:
import pytest
from datetime import datetime, timedelta
import asf_search as asf
# I'm located in Portland Oregon, which may effect time localization
START = datetime.utcfromtimestamp(1638864000)
DT = timedelta(days=1)
ASF_MAGIC_BUG_DATE = datetime.utcfromtimestamp(1638864000) + timedelta(hours=2, minutes=55)
@pytest.mark.commit
def test_sentinel_1_all():
end = START + DT
asf.search(
platform=asf.SENTINEL1, start=START, end=end
)
@pytest.mark.commit
def test_sentinel_1_split():
end = START + DT
asf.search(
platform=asf.SENTINEL1, start=START, end=ASF_MAGIC_BUG_DATE
)
asf.search(platform=asf.SENTINEL1B, start=ASF_MAGIC_BUG_DATE, end=end)
@pytest.mark.commit
def test_sentinel_1_split_2():
end = START + DT
asf.search(
platform=asf.SENTINEL1, start=START, end=ASF_MAGIC_BUG_DATE
)
asf.search(platform=asf.SENTINEL1A, start=ASF_MAGIC_BUG_DATE, end=end)
@pytest.mark.commit
def test_sentinel_1_all_2():
start = START + DT
end = start + DT
asf.search(
platform=asf.SENTINEL1, start=start, end=end
)Expected behavior
This search should complete without throwing an error
Desktop (please complete the following information):
Running on Mac OSX
% uname -a
Darwin Philips-MBP.hsd1.or.comcast.net 22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:52:24 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T6000 arm64(gfwsat-dev) philiprobinson@Philips-MBP gfwsat % python --version
Python 3.10.11(gfwsat-dev) philiprobinson@Philips-MBP gfwsat % pip freeze
affine @ file:///Users/ktietz/demo/mc3/conda-bld/affine_1629890479180/work
aiohttp @ file:///Users/runner/miniforge3/conda-bld/aiohttp_1686375944694/work
aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1667935791922/work
-e git+ssh://[email protected]/probinso/antimeridian_splitter.git@139822244014b156c013d05128afe722aeec626c#egg=antimeridian_splitter
anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1685486400275/work/dist
appdirs==1.4.4
appnope @ file:///home/conda/feedstock_root/build_artifacts/appnope_1649077682618/work
argcomplete @ file:///home/conda/feedstock_root/build_artifacts/argcomplete_1686587382193/work
argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1640817743617/work
argon2-cffi-bindings @ file:///Users/runner/miniforge3/conda-bld/argon2-cffi-bindings_1666850813776/work
arrow==1.2.3
asciitree==0.3.3
asf-search==5.0.2
asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1670263926556/work
async-lru==2.0.2
async-timeout @ file:///home/conda/feedstock_root/build_artifacts/async-timeout_1640026696943/work
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1683424013410/work
autobahn==23.6.1
Automat==22.10.0
Babel==2.12.1
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work
beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1680888073205/work
black==22.12.0
bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1674535352125/work
blessed==1.20.0
blinker @ file:///home/conda/feedstock_root/build_artifacts/blinker_1681349778161/work
bpython==0.24
branca @ file:///home/conda/feedstock_root/build_artifacts/branca_1667848736831/work
cachetools @ file:///home/conda/feedstock_root/build_artifacts/cachetools_1674482203741/work
Cartopy @ file:///Users/runner/miniforge3/conda-bld/cartopy_1670926209927/work
certifi @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_3bweqb8byn/croot/certifi_1690232244779/work/certifi
cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1671179893800/work
cfgv==3.3.1
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1678108872112/work
click @ file:///home/conda/feedstock_root/build_artifacts/click_1666798198223/work
click-plugins==1.1.1
cligj @ file:///home/conda/feedstock_root/build_artifacts/cligj_1633637764473/work
cmocean==2.0
colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1666700638685/work
colorspacious==1.1.2
comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1679481329611/work
constantly==15.1.0
coverage==7.2.7
cryptography @ file:///Users/runner/miniforge3/conda-bld/cryptography-split_1685659597707/work
curtsies==0.4.1
cwcwidth==0.1.8
cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1635519461629/work
darglint==1.8.1
dataclass-type-validator==0.1.2
dataclasses-json==0.5.8
dateparser==1.1.8
db-dtypes @ file:///home/conda/feedstock_root/build_artifacts/db-dtypes_1680245382075/work
debugpy @ file:///Users/runner/miniforge3/conda-bld/debugpy_1680755676047/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
distlib==0.3.6
dparse==0.6.2
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work
ephem==4.1.4
exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1678703645500/work
executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1667317341051/work
fastavro @ file:///Users/runner/miniforge3/conda-bld/fastavro_1683226863489/work
fasteners @ file:///home/conda/feedstock_root/build_artifacts/fasteners_1643971550063/work
fastjsonschema @ file:///home/conda/feedstock_root/build_artifacts/python-fastjsonschema_1684761244589/work/dist
filelock==3.12.2
Fiona @ file:///Users/runner/miniforge3/conda-bld/fiona_1671017708511/work
flake8==6.0.0
Flask==2.3.2
flit_core @ file:///home/conda/feedstock_root/build_artifacts/flit-core_1684084314667/work/source/flit_core
folium @ file:///home/conda/feedstock_root/build_artifacts/folium_1670956290671/work
fqdn==1.5.1
frozenlist @ file:///Users/runner/miniforge3/conda-bld/frozenlist_1667935502123/work
fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1668082755814/work
ftpretty==0.4.0
gcsfs @ file:///home/conda/feedstock_root/build_artifacts/gcsfs_1668089545087/work
GDAL @ file:///Users/runner/miniforge3/conda-bld/gdal-split_1670808086647/work/build/swig/python
geographiclib==2.0
geojson==3.0.1
geomet==1.0.0
geopandas @ file:///home/conda/feedstock_root/build_artifacts/geopandas_1686057576800/work
geopy==2.3.0
geos==0.2.3
-e git+ssh://[email protected]/GlobalFishingWatch/gfwsat.git@288e693743dd159deb6ac626e353688965f87ed2#egg=gfwsat
Glymur @ file:///home/conda/feedstock_root/build_artifacts/glymur_1669641175421/work
google-api-core @ file:///home/conda/feedstock_root/build_artifacts/google-api-core-split_1665207588296/work
google-api-python-client @ file:///home/conda/feedstock_root/build_artifacts/google-api-python-client_1685447681973/work
google-auth==2.6.6
google-auth-httplib2 @ file:///home/conda/feedstock_root/build_artifacts/google-auth-httplib2_1654134467987/work
google-auth-oauthlib==0.5.2
google-cloud==0.34.0
google-cloud-bigquery==3.3.6
google-cloud-bigquery-storage @ file:///home/conda/feedstock_root/build_artifacts/google-cloud-bigquery-storage-split_1642183604146/work
google-cloud-core @ file:///home/conda/feedstock_root/build_artifacts/google-cloud-core_1658759191372/work
google-cloud-storage==2.9.0
google-crc32c @ file:///Users/runner/miniforge3/conda-bld/google-crc32c_1666827019506/work
google-resumable-media==2.3.3
googleapis-common-protos==1.56.4
greenlet==2.0.2
grpcio @ file:///Users/runner/miniforge3/conda-bld/grpc-split_1684119215020/work
grpcio-status @ file:///home/conda/feedstock_root/build_artifacts/grpcio-status_1683994707350/work
h11==0.14.0
h3==3.7.6
haversine @ file:///home/conda/feedstock_root/build_artifacts/haversine_1662299098807/work
hkdf==0.0.3
html2text==2020.1.16
httpcore==0.17.3
httplib2 @ file:///home/conda/feedstock_root/build_artifacts/httplib2_1679483503307/work
httpx==0.24.1
humanize==4.6.0
hyperlink==21.0.0
identify==2.5.24
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1663625384323/work
imageio==2.31.1
imgkit==1.2.3
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1682176699712/work
importlib-resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1676919000169/work
incremental==22.10.0
iniconfig==2.0.0
ipykernel @ file:///Users/runner/miniforge3/conda-bld/ipykernel_1684162997309/work
ipython @ file:///Users/runner/miniforge3/conda-bld/ipython_1685727999785/work
ipython-genutils==0.2.0
ipywidgets @ file:///home/conda/feedstock_root/build_artifacts/ipywidgets_1655973868664/work
isoduration==20.11.0
itsdangerous==2.1.2
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1669134318875/work
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1654302431367/work
jinja2-cli==0.8.2
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1663332044897/work
json5==0.9.14
jsonpointer==2.3
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1669810440410/work
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events @ file:///home/conda/feedstock_root/build_artifacts/jupyter_events_1673559782596/work
jupyter-lsp==2.2.0
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1681432441054/work
jupyter_core @ file:///Users/runner/miniforge3/conda-bld/jupyter_core_1678994238709/work
jupyter_server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1685051725700/work
jupyter_server_terminals @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_terminals_1673491454549/work
jupyterlab==4.0.2
jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1649936611996/work
jupyterlab-widgets @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_widgets_1680041524590/work
jupyterlab_server==2.22.1
jupytext==1.14.6
kiwisolver @ file:///Users/runner/miniforge3/conda-bld/kiwisolver_1666805765141/work
kml2geojson==5.1.0
LatLon==1.0.2
lazy_loader==0.2
libcst @ file:///Users/runner/miniforge3/conda-bld/libcst_1686240048692/work
Logbook==1.5.3
lxml @ file:///Users/runner/miniforge3/conda-bld/lxml_1671013522951/work
magic-wormhole==0.12.0
mapclassify @ file:///home/conda/feedstock_root/build_artifacts/mapclassify_1673861555770/work
markdown-it-py==2.2.0
MarkupSafe @ file:///Users/runner/miniforge3/conda-bld/markupsafe_1685769224421/work
marshmallow==3.19.0
marshmallow-enum==1.5.1
matplotlib @ file:///Users/runner/miniforge3/conda-bld/matplotlib-suite_1636786736839/work
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work
mccabe==0.7.0
mdit-py-plugins==0.4.0
mdurl==0.1.2
mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1675771498296/work
mpmath==1.3.0
msgpack @ file:///Users/runner/miniforge3/conda-bld/msgpack-python_1678312968551/work
multidict @ file:///Users/runner/miniforge3/conda-bld/multidict_1672339514000/work
munch @ file:///home/conda/feedstock_root/build_artifacts/munch_1684150905574/work
mypy==1.3.0
mypy-extensions==1.0.0
nbclassic @ file:///home/conda/feedstock_root/build_artifacts/nbclassic_1683202081046/work
nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1684790896106/work
nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert-meta_1683636231832/work
nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1685541374434/work
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1664684991461/work
networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1680692919326/work
nodeenv==1.8.0
notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1680870634737/work
notebook_shim @ file:///home/conda/feedstock_root/build_artifacts/notebook-shim_1682360583588/work
numcodecs @ file:///Users/runner/miniforge3/conda-bld/numcodecs_1654108155270/work
numpy @ file:///Users/runner/miniforge3/conda-bld/numpy_1682210346059/work
oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1666056362788/work
overrides @ file:///home/conda/feedstock_root/build_artifacts/overrides_1666057828264/work
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1637239678211/work
pandas==1.3.5
pandas-gbq==0.17.9
pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work
pathspec==0.11.1
patsy @ file:///home/conda/feedstock_root/build_artifacts/patsy_1665356157073/work
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
Pillow @ file:///Users/runner/miniforge3/conda-bld/pillow_1666920704573/work
pipdeptree==2.9.0
pkgutil_resolve_name @ file:///home/conda/feedstock_root/build_artifacts/pkgutil-resolve-name_1633981968097/work
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1686403120743/work
plotly @ file:///home/conda/feedstock_root/build_artifacts/plotly_1654895488764/work
pluggy==1.0.0
pooch @ file:///home/conda/feedstock_root/build_artifacts/pooch_1679580333621/work
pre-commit==3.3.2
prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1684970902412/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1677600924538/work
proplot @ file:///home/conda/feedstock_root/build_artifacts/proplot_1677584975715/work
proto-plus @ file:///home/conda/feedstock_root/build_artifacts/proto-plus_1673334163294/work
protobuf==4.21.12
psutil @ file:///Users/runner/miniforge3/conda-bld/psutil_1681775207008/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work
py-eodms-rapi==1.5.5
pyarrow==9.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.7
pycodestyle==2.10.0
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
pydata-google-auth==1.4.0
pyflakes==3.0.1
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1681904169130/work
PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1683676063469/work
PyNaCl==1.5.0
pyobjc-core @ file:///Users/runner/miniforge3/conda-bld/pyobjc-core_1686129289943/work
pyobjc-framework-Cocoa @ file:///Users/runner/miniforge3/conda-bld/pyobjc-framework-cocoa_1686136016408/work
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1685514481738/work
pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1652235407899/work
pyperclip==1.8.2
pyproj @ file:///Users/runner/miniforge3/conda-bld/pyproj_1670929490271/work
pyrsistent @ file:///Users/runner/miniforge3/conda-bld/pyrsistent_1672681537313/work
pyseas @ git+https://github.com/GlobalFishingWatch/pyseas.git@86d48ff7cc38f7e9a189e9d830ff2c6a05454ba5
pyshape==0.1.1
pyshp @ file:///home/conda/feedstock_root/build_artifacts/pyshp_1659002966020/work
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
pytest==7.3.2
pytest-cov==4.1.0
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
python-json-logger @ file:///home/conda/feedstock_root/build_artifacts/python-json-logger_1677079630776/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1680088766131/work
pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/work
PyWavelets==1.4.1
pyxdg==0.28
PyYAML @ file:///Users/runner/miniforge3/conda-bld/pyyaml_1666772543719/work
pyzmq @ file:///Users/runner/miniforge3/conda-bld/pyzmq_1685519412652/work
qtconsole==5.4.3
QtPy==2.3.1
rasterio @ file:///Users/runner/miniforge3/conda-bld/rasterio_1668655036926/work
ray==2.6.3
regex==2023.6.3
Represent==1.6.0.post0
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1684774241324/work
requests-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/requests-oauthlib_1643557462909/work
retrying==1.3.4
rfc3339-validator @ file:///home/conda/feedstock_root/build_artifacts/rfc3339-validator_1638811747357/work
rfc3986-validator @ file:///home/conda/feedstock_root/build_artifacts/rfc3986-validator_1598024191506/work
rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1658328885051/work
Rtree @ file:///Users/runner/miniforge3/conda-bld/rtree_1666813320619/work
ruamel.yaml==0.17.31
ruamel.yaml.clib==0.2.7
rush==2021.4.0
safety==2.3.5
scikit-image==0.21.0
scikit-learn @ file:///Users/runner/miniforge3/conda-bld/scikit-learn_1685023781664/work
scipy==1.9.1
seaborn @ file:///home/conda/feedstock_root/build_artifacts/seaborn-split_1629095986539/work
Send2Trash @ file:///Users/runner/miniforge3/conda-bld/send2trash_1682601407921/work
sentinelsat==1.1.1
service-identity==21.1.0
Shapely @ file:///Users/runner/miniforge3/conda-bld/shapely_1668518218855/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1662051266223/work
snuggs @ file:///Users/ktietz/demo/mc3/conda-bld/snuggs_1629803886966/work
sortedcollections==2.1.0
sortedcontainers==2.4.0
soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1658207591808/work
spacetrack==0.16.0
spake2==0.8
stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work
statsmodels @ file:///Users/runner/miniforge3/conda-bld/statsmodels_1683305739900/work
sympy==1.12
tabulate==0.9.0
tenacity @ file:///home/conda/feedstock_root/build_artifacts/tenacity_1677600641219/work
terminado @ file:///Users/runner/miniforge3/conda-bld/terminado_1670254106711/work
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work
tifffile==2023.4.12
tinycss2 @ file:///home/conda/feedstock_root/build_artifacts/tinycss2_1666100256010/work
toml @ file:///home/conda/feedstock_root/build_artifacts/toml_1604308577558/work
tomli==2.0.1
tomlkit @ file:///home/conda/feedstock_root/build_artifacts/tomlkit_1683028165096/work
tornado @ file:///Users/runner/miniforge3/conda-bld/tornado_1684150425472/work
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1677948868469/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1675110562325/work
Twisted==22.10.0
txaio==23.1.1
txtorcon==23.5.0
types-requests==2.28.11.17
types-urllib3==1.26.25.13
typing-inspect @ file:///home/conda/feedstock_root/build_artifacts/typing_inspect_1685820062773/work
typing-utils @ file:///home/conda/feedstock_root/build_artifacts/typing_utils_1622899189314/work
typing_extensions==4.6.3
tzdata @ file:///home/conda/feedstock_root/build_artifacts/python-tzdata_1680081134351/work
tzlocal==5.0.1
uri-template==1.2.0
uritemplate @ file:///home/conda/feedstock_root/build_artifacts/uritemplate_1634152692041/work
urllib3==1.26.16
urlpath==1.2.0
viola==0.3.8
virtualenv==20.23.0
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1673864653149/work
webcolors==1.13
webencodings==0.5.1
websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1686311450661/work
Werkzeug==2.3.6
widgetsnbextension @ file:///home/conda/feedstock_root/build_artifacts/widgetsnbextension_1680041491747/work
xmltodict @ file:///home/conda/feedstock_root/build_artifacts/xmltodict_1652020822199/work
xyzservices @ file:///home/conda/feedstock_root/build_artifacts/xyzservices_1684571362432/work
yarl @ file:///Users/runner/miniforge3/conda-bld/yarl_1685191847849/work
yq @ file:///home/conda/feedstock_root/build_artifacts/yq_1682205044037/work
zarr @ file:///home/conda/feedstock_root/build_artifacts/zarr_1649285734910/work
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1677313463193/work
zope.interface==6.0
- Python Version [e.g. python3.11]
- Pip Environment ['python3 -m pip freeze']
Additional context
I originally thought this had to do with sentinel-1b failing in December of 2021, but that failure was on the 23rd.
[Bug] Error when Retrieving Granule metadata - Would be nice if this is more descriptive for ASF
Describe the bug
Suspect this is related to the data outage on October 7th from 3 - 7 AM UTC; we are not in the specified time range.
I see the HTTP error - maybe I have something wrong at my end with credential setup? I thought this portion of metadata didn't require any credentials.
When I say "more descriptive for ASF", I mean if this is an error related to some sort of expected outage, maybe a package specific error could be raised to alert users to the possible causes. But this issue will be a new pointer as well.
Anyways, thanks for your help!
Using asf_search version 2.0.2 installed with pip.
import asf_search as asf
response = asf.granule_search(['S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9'])
Outputs:
---------------------------------------------------------------------------
HTTPError Traceback (most recent call last)
~/opt/anaconda3/envs/rscube/lib/python3.8/site-packages/asf_search/search/search.py in search(absoluteOrbit, asfFrame, beamMode, collectionName, maxDoppler, minDoppler, end, maxFaradayRotation, minFaradayRotation, flightDirection, flightLine, frame, granule_list, groupID, insarStackId, instrument, intersectsWith, lookDirection, offNadirAngle, platform, polarization, processingDate, processingLevel, product_list, relativeOrbit, season, start, maxResults, host, cmr_token, cmr_provider)
141 try:
--> 142 response.raise_for_status()
143 except HTTPError:
~/opt/anaconda3/envs/rscube/lib/python3.8/site-packages/requests/models.py in raise_for_status(self)
952 if http_error_msg:
--> 953 raise HTTPError(http_error_msg, response=self)
954
HTTPError: 504 Server Error: Gateway Time-out for url: https://api.daac.asf.alaska.edu/services/search/param
During handling of the above exception, another exception occurred:
JSONDecodeError Traceback (most recent call last)
<ipython-input-4-f9e9a1c40764> in <module>
1 import asf_search as asf
2
----> 3 response = asf.granule_search(['S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9'])
~/opt/anaconda3/envs/rscube/lib/python3.8/site-packages/asf_search/search/granule_search.py in granule_search(granule_list, host, cmr_token, cmr_provider)
25 data = dict((k,v) for k,v in kwargs.items() if v is not None and v != '')
26
---> 27 return search(**data)
~/opt/anaconda3/envs/rscube/lib/python3.8/site-packages/asf_search/search/search.py in search(absoluteOrbit, asfFrame, beamMode, collectionName, maxDoppler, minDoppler, end, maxFaradayRotation, minFaradayRotation, flightDirection, flightLine, frame, granule_list, groupID, insarStackId, instrument, intersectsWith, lookDirection, offNadirAngle, platform, polarization, processingDate, processingLevel, product_list, relativeOrbit, season, start, maxResults, host, cmr_token, cmr_provider)
145 raise ASFSearch4xxError(f'HTTP {response.status_code}: {response.json()["error"]["report"]}')
146 if 500 <= response.status_code <= 599:
--> 147 raise ASFSearch5xxError(f'HTTP {response.status_code}: {response.json()["error"]["report"]}')
148 raise ASFServerError(f'HTTP {response.status_code}: {response.json()["error"]["report"]}')
149
~/opt/anaconda3/envs/rscube/lib/python3.8/site-packages/requests/models.py in json(self, **kwargs)
908 # used.
909 pass
--> 910 return complexjson.loads(self.text, **kwargs)
911
912 @property
~/opt/anaconda3/envs/rscube/lib/python3.8/site-packages/simplejson/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, use_decimal, **kw)
523 parse_constant is None and object_pairs_hook is None
524 and not use_decimal and not kw):
--> 525 return _default_decoder.decode(s)
526 if cls is None:
527 cls = JSONDecoder
~/opt/anaconda3/envs/rscube/lib/python3.8/site-packages/simplejson/decoder.py in decode(self, s, _w, _PY3)
368 if _PY3 and isinstance(s, bytes):
369 s = str(s, self.encoding)
--> 370 obj, end = self.raw_decode(s)
371 end = _w(s, end).end()
372 if end != len(s):
~/opt/anaconda3/envs/rscube/lib/python3.8/site-packages/simplejson/decoder.py in raw_decode(self, s, idx, _w, _PY3)
398 elif ord0 == 0xef and s[idx:idx + 3] == '\xef\xbb\xbf':
399 idx += 3
--> 400 return self.scan_once(s, idx=_w(s, idx).end())
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The SLC is on the DAAC as noted in this link.
[Feature] Skip over existing downloads
Hi all, great work you've done here.
I recently found that when I try and download an existing dataset (an ARIA product in this case) the code errors out:
asf_search.exceptions.ASFDownloadError: File already exists: ~/products/S1-GUNW-A-R-004-tops-20210805_20210712-230647-37654N_35777N-PP-8a73-v2_0_4.nc
Previously the bulk downloader would just skip and continue, which is preferable if you were say updating a time-series and wanted to download just the newest acquisitions for some area. Perhaps there could be a way to specify in calls to download whether to skip, re-download, or error when hitting the same filename.
Populate `frameNumber` for S1 ARIA GUNW search results
New S1 ARIA GUNW products published to CMR will soon include a FRAME_NUMBER additional attribute. @cmarshak would like this value populated in the frameNumber field for asf_search search results.
For more context, see ACCESS-Cloud-Based-InSAR/DockerizedTopsApp#136
To implement, I suspect we'd need to add the ASF_PLATFORM of Sentinel-1 Interferogram (BETA) the lists of platforms at:
[Bug]"perpendicularBaseline": null
when I search SLC๏ผI find some SLC'S perpendicularBaseline is null in properties['fileID'].
So, I could not to plot basline image.
[Bug] Why do I get duplicate products in my response of OPERA RTC-S1 data?
So, I am searching OPERA RTC-S1 data (see snippet below) and I am getting rows with the same product repeated twice or more. Am I doing something wrong (or even slightly dumb) or is there an actual issue?
from rasterio.crs import CRS
import geopandas as gpd
import asf_search as asf
from shapely.geometry import box
aoi_bounds = [-72.008,-33.889,-70.049, -32.672]
aoi = box(*aoi_bounds)
resp = asf.geo_search(intersectsWith=aoi.wkt,
processingLevel='RTC',
polarization=['VV', 'VH'],
platform=asf.constants.SENTINEL1)
df = gpd.GeoDataFrame.from_features(resp.geojson(), crs=CRS.from_epsg(4326))
df.sort_values(by='fileID').reset_index(drop=True)
df.fileID.iloc[:2].tolist()
Gives me the following:
['OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0',
'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0']
Partly sharing this in case my potential mistake helps others. It doesn't seem to be a duplicate product.
A full display of the records is here:
df.to_dict('records)[:2]
as
[{'geometry': <POLYGON ((-72.513 -33.613, -71.646 -33.397, -71.711 -33.214, -72.576 -33.42...>,
'centerLat': -33.41339684033409,
'centerLon': -72.11151852934287,
'stopTime': '2024-06-20T23:36:52Z',
'fileID': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0',
'flightDirection': 'ASCENDING',
'pathNumber': 91,
'processingLevel': 'RTC',
'url': 'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
'startTime': '2024-06-20T23:36:49Z',
'sceneName': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0',
'browse': ['https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png',
'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png',
'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png'],
'platform': 'Sentinel-1A',
'bytes': {'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5': {'bytes': 103992,
'format': 'HDF5'},
'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif': {'bytes': 6546248,
'format': 'GeoTIFF'},
'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif': {'bytes': 6477441,
'format': 'GeoTIFF'},
'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif': {'bytes': 51929,
'format': 'GeoTIFF'},
'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml': {'bytes': 161617,
'format': 'XML'}},
'md5sum': None,
'granuleType': None,
'orbit': 54413,
'polarization': ['VV', 'VH'],
'processingDate': '2024-06-21T03:26:22Z',
'sensor': 'C-SAR',
'groupID': 'S1A_IWDV_1063_1069_054413_091',
'pgeVersion': '2.1.1',
'operaBurstID': 'T091_195252_IW3',
'validityStartDate': None,
'subswath': 'IW3',
'fileName': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
'beamModeType': 'IW',
's3Urls': ['s3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png.md5',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png.md5',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png.md5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5.md5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml.md5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif.md5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif.md5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif.md5'],
'beamMode': 'IW',
'additionalUrls': ['https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5',
'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml',
'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif',
'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif'],
'bistaticDelayCorrection': 'TRUE',
'noiseCorrection': 'TRUE',
'postProcessingFilter': 'FALSE'},
{'geometry': <POLYGON ((-72.513 -33.613, -71.646 -33.397, -71.711 -33.214, -72.576 -33.42...>,
'centerLat': -33.41339684033409,
'centerLon': -72.11151852934287,
'stopTime': '2024-06-20T23:36:52Z',
'fileID': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0',
'flightDirection': 'ASCENDING',
'pathNumber': 91,
'processingLevel': 'RTC',
'url': 'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
'startTime': '2024-06-20T23:36:49Z',
'sceneName': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0',
'browse': ['https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png',
'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png',
'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png'],
'platform': 'Sentinel-1A',
'bytes': {'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5': {'bytes': 103992,
'format': 'HDF5'},
'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif': {'bytes': 6546248,
'format': 'GeoTIFF'},
'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif': {'bytes': 6477441,
'format': 'GeoTIFF'},
'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif': {'bytes': 51929,
'format': 'GeoTIFF'},
'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml': {'bytes': 161617,
'format': 'XML'}},
'md5sum': None,
'granuleType': None,
'orbit': 54413,
'polarization': ['VV', 'VH'],
'processingDate': '2024-06-21T03:26:22Z',
'sensor': 'C-SAR',
'groupID': 'S1A_IWDV_1063_1069_054413_091',
'pgeVersion': '2.1.1',
'operaBurstID': 'T091_195252_IW3',
'validityStartDate': None,
'subswath': 'IW3',
'fileName': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
'beamModeType': 'IW',
's3Urls': ['s3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png.md5',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png.md5',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png.md5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5.md5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml.md5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif.md5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif.md5',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif',
's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif.md5'],
'beamMode': 'IW',
'additionalUrls': ['https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5',
'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml',
'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif',
'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif'],
'bistaticDelayCorrection': 'TRUE',
'noiseCorrection': 'TRUE',
'postProcessingFilter': 'FALSE'}]
[Bug] `search()` only accepts `list[str]` for options marked with `Iterable[str]`
Describe the bug
Most of the arguments of .search are typed such that they accept either a string, or an Iterable of strings. In my example, I was searching for operaBurstID:
https://github.com/asfadmin/Discovery-asf_search/blob/master/asf_search/search/search.py#L43
Right now, these options seem to break if you pass any Iterable except a list
To Reproduce
import asf_search as asf
asf.search(operaBurstID=('T078_165486_IW2', 'T078_165485_IW2'), processingLevel='CSLC-STATIC')This should find two results, but prints:
ERROR:root:HTTP 400: ["The maximum value [ 'T078_165485_IW2')] must be greater than the minimum value [('T078_165486_IW2']"]
ERROR:root:The asf-search module ecountered an error with CMR, and the following message was automatically reported to ASF:
"
Error Message: HTTP 400: ["The maximum value [ 'T078_165485_IW2')] must be greater than the minimum value [('T078_165486_IW2']"]
User Agent: Python/3.10.10; requests/2.28.2; asf_search/6.7.1
Search Options: {
processingLevel: ['CSLC-STATIC']
operaBurstID: ["('T078_165486_IW2', 'T078_165485_IW2')"]
}
"If you have any questions email [email protected]
Out[25]: ASFSearchResults([])
Expected behavior
Same as passing a list:
In [27]: asf.search(operaBurstID=['T078_165486_IW2', 'T078_165485_IW2'], processingLevel='CSLC-STATIC')
Out[27]:
ASFSearchResults([<asf_search.ASFProduct.ASFProduct at 0x1530ca4a0>,
<asf_search.ASFProduct.ASFProduct at 0x1530cb220>])Note:
Running the parsing which happens inside search:
https://github.com/asfadmin/Discovery-asf_search/blob/master/asf_search/search/search.py#L89-L90C28
ipdb> print(data)
{'processingLevel': 'CSLC-STATIC', 'operaBurstID': ('T078_165486_IW2', 'T078_165485_IW2')}
ipdb> sopts = ASFSearchOptions()
ipdb> sopts.merge_args(**data)
ipdb> print(sopts)
{
"processingLevel": [
"CSLC-STATIC"
],
"operaBurstID": [
"('T078_165486_IW2', 'T078_165485_IW2')"
]
}and I see this is happening in the validators part, since operaBurstID is running the parse_string_list validator
https://github.com/asfadmin/Discovery-asf_search/blob/master/asf_search/ASFSearchOptions/validators.py#L96-L99
I'm not sure if anyone wants to put in the effort to convert all the validators from list-parsers into proper Iterator parsers... so the other short-term option would be to change the type annotations on the search functions.
Desktop (please complete the following information):
$ conda info
active environment : mapping
active env location : /Users/staniewi/miniconda3/envs/mapping
shell level : 2
user config file : /Users/staniewi/.condarc
populated config files : /Users/staniewi/.condarc
conda version : 23.9.0
conda-build version : not installed
python version : 3.9.18.final.0
virtual packages : __archspec=1=m1
__osx=13.6.2=0
__unix=0=0
base environment : /Users/staniewi/miniconda3 (writable)
conda av data dir : /Users/staniewi/miniconda3/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/osx-arm64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/osx-arm64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /Users/staniewi/miniconda3/pkgs
/Users/staniewi/.conda/pkgs
envs directories : /Users/staniewi/miniconda3/envs
/Users/staniewi/.conda/envs
platform : osx-arm64
user-agent : conda/23.9.0 requests/2.31.0 CPython/3.9.18 Darwin/22.6.0 OSX/13.6.2
UID:GID : 503:20
netrc file : /Users/staniewi/.netrc
offline mode : False
Additional context
Add any other context about the problem here.
Support searching by TEMPORAL_BASELINE_DAYS attribute for S1 ARIA GUNW products
New S1 ARIA GUNW products will soon include a TEMPORAL_BASELINE_DAYS attribute in CMR. @cmarshak is interested in being able to search by this parameter using asf_search.
For more context, see ACCESS-Cloud-Based-InSAR/DockerizedTopsApp#136
This applies to the five CMR collections with ASF_PLATFORM=Sentinel-1 Interferogram (BETA); these CMR collection short names:
- SENTINEL-1_INTERFEROGRAMS
- SENTINEL-1_INTERFEROGRAMS_AMPLITUDE
- SENTINEL-1_INTERFEROGRAMS_COHERENCE
- SENTINEL-1_INTERFEROGRAMS_CONNECTED_COMPONENTS
- SENTINEL-1_INTERFEROGRAMS_UNWRAPPED_PHASE
[Feature]
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
[Bug]Searching the burst stack based on asf_search is much less compared to the ASF site.
Taking 'S1_303069_IW3_20231004T101321_VV_AF9D-BURST' as an example, only 5 images can be searched using python, and 15 images can be searched on the asf website.
No module named 'importlib.metadata ' problem with python 3.6
Hi!
I'm having this problem of no metadata sub-module in importlib, but I do have a importlib_metada module that seems to work fine. I don't know if I'm the only one, but maybe adding something like this could prevent this problem from other users too
try:
from importlib.metadata import PackageNotFoundError, version
except ImportError: # for Python<3.8
from importlib_metadata import PackageNotFoundError, version
I added this to the asf_search/__init__.py , asf_search/search/search.py and asf_search/download/download.py modules and so far it's working well. Could something like that be included in future releases?
[Feature] - nice progress bar during downloading of SLC data
Is your feature request related to a problem? Please describe.
Downloading SLC data takes a long time. Would be nice to track.
Describe the solution you'd like
Something like this - https://gist.github.com/wy193777/0e2a4932e81afc6aa4c8f7a2984f34e2 - would be happy to make a pull request if the feature would be welcome. Understand if this is out of scope.
Describe alternatives you've considered
I think there are other progress bars, but am not familiar with them.
Additional context
There is a lot of output during downloading e.g.
response: 307
Redirect to https://sentinel1.asf.alaska.edu/SLC/SB/S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9.zip
response: 302
Redirect to https://dy4owt9f80bz7.cloudfront.net/...
response: 200
Rather be able to track progress more clearly.
edit: realizing this feature request might be a bit hasty as there might be use cases which this is not desired and annoying - but hopefully there might be a smart way to integrate such a feature.
[Feature] OPERA Datasets searchable by burst id? (yes, we can filter on bursts after we get all the relevant products)
Partly feature request. Also, documenting for OPERA, what is available.
I can find OPERA datasets using the "groupID" in general search function.
I first did this:
import asf_search as asf
resp = asf.granule_search(['OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0'])
resp[0].properties
This is the OPERA-RTC-S1 product.
The output is:
{'beamModeType': None,
'browse': ['https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0_BROWSE.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0_BROWSE.png',
'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0_BROWSE_low-res.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0_BROWSE_low-res.png',
'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0_BROWSE_thumbnail.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0_BROWSE_thumbnail.png'],
'bytes': None,
'centerLat': None,
'centerLon': None,
'faradayRotation': None,
'fileID': 'OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0',
'flightDirection': 'ASCENDING',
'groupID': 'S1A_IWDV_0051_0057_050900_078',
'granuleType': None,
'insarStackId': None,
'md5sum': None,
'offNadirAngle': None,
'orbit': 50900,
'pathNumber': 78,
'platform': 'Sentinel-1A',
'pointingAngle': None,
'polarization': 'VV',
'processingDate': '2023-10-24T09:14:20Z',
'processingLevel': 'RTC',
'sceneName': 'OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0',
'sensor': 'C-SAR',
'startTime': '2023-10-24T00:48:29Z',
'stopTime': '2023-10-24T00:48:32Z',
'url': 'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0_VH.tif',
'fileName': 'OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0_VH.tif',
'frameNumber': None}
The groupID is what I can use to use with intersectsWith.
My feature request:
- Will there be constants for OPERA datasets here: https://github.com/asfadmin/Discovery-asf_search/tree/master/asf_search/constants
- Are there plans to have the OPERA burst id in the metadata? Would be useful to pull by burst id (though equivalently it could be done with geographic point and track). Techinically, could use the filename to populate those that have been already ingested e.g.
T078-165486-IW2.
downloads not working when run from inside AWS
I am using the asf_search library from inside a kubernetes cluster on AWS. I am getting some 400 errors at the final redirected URL due to two sets of authentication information being provided.
I am using the new v3.0.0 and therefore the ASFSession class.
This error occurs for slightly different reasons with both auth_with_token() and auth_with_creds()
The error is:
<Code>InvalidArgument</Code>
<Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message>This is because the previous redirect carries the authentication query parameter and the session object still carries the token in the header (for auth_with_token). For auth_with_creds(), there is still a conflict with authentication methods.
If using auth_with_token(), I can make it work by using session.headers.clear() before the last URL is called. I haven't worked out what the conflict is with auth_with_creds() yet.
This all works perfectly well when not run on an AWS EC2 node because the final redirect is actually via cloudfront and it seems to handle it.
When run from inside AWS, the final redirect is something like:
https://asf-ngap2w-p-s1-grd-7d1b4348.s3.us-west-2.amazonaws.com/S1B_IW_GRDH_1SDV_20210101T095552_20210101T095617_024959_02F879_2BEA.zip?A-userid=....
When run from a machine outside AWS, the final URL is something like:
https://dy4owt9f80bz7.cloudfront.net/s3-06b3213905ebfa9144871d6c7f9306f0/asf-ngap2w-p-s1-grd-7d1b4348.s3.us-west-2.amazonaws.com/S1B_IW_GRDH_1SDV_20210101T095552_20210101T095617_024959_02F879_2BEA.zip?...
I haven't fully investigated, but that is what I think is happening.
Super useful package though!
[Feature] All HyP3 API URLs can end with '/'
Is your feature request related to a problem? Please describe.
When changing the HyP3 API URL in Preferences > On Demand Presets to an API other than hyp3-api, if there is a / at the end of the URL (ex: https://hyp3-test.asf.alaska.edu/ instead of https://hyp3-test.asf.alaska.edu), the job will get an error on submission.
Note that when the API is hyp3-api, the '/' at the end of URL will successfully submit a job (i.e. https://hyp3-api.asf.alaska.edu/ and https://hyp3-api.asf.alaska.edu both successfully submit jobs).
Describe the solution you'd like
All HyP3 API URLs will be recognized even with the trailing '/' (ex: https://hyp3-test.asf.alaska.edu/ is a valid URL).
Describe alternatives you've considered
One could simply add the correct URL without the trailing '/'.
Additional context
While this isn't necessarily a breaking bug, it did cause some confusion on my end and might warrant a minor fix. Also, the actual number of users who will come across this issue is minimal. Feel free to ignore this issue, too!
[Feature] Document `processingLevel='RTC'`
Is your feature request related to a problem? Please describe.
The Search API Keywords page does not document processingLevel='RTC' which is used by OPERA RTC-S1.
Describe the solution you'd like
Update the keywords page.
Describe alternatives you've considered
None.
[Bug] Date parsing error after installing v5.0.1 via conda-forge
I receive a regex._regex_core.error: bad escape \d at position 7 exception when searching using the start parameter to asf_search.search after installing asf_search v5.0.1 from conda forge:
(base) $ conda create --name foo asf_search=5.0.1
...
(base) $ conda activate foo
(foo) $ python
Python 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:04:59) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asf_search
>>> asf_search.search(start='2022-07-20')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/asf_search/search/search.py", line 94, in search
kw_opts = ASFSearchOptions(**kwargs)
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/asf_search/ASFSearchOptions/ASFSearchOptions.py", line 21, in __init__
self.__setattr__(key, value)
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/asf_search/ASFSearchOptions/ASFSearchOptions.py", line 39, in __setattr__
super().__setattr__(key, validate(key, value))
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/asf_search/ASFSearchOptions/validator_map.py", line 19, in validate
return validator_map[key](value)
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/asf_search/ASFSearchOptions/validators.py", line 52, in parse_date
d = dateparser.parse(str(value))
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/dateparser/conf.py", line 89, in wrapper
return f(*args, **kwargs)
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/dateparser/__init__.py", line 54, in parse
data = parser.get_date_data(date_string, date_formats)
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/dateparser/date.py", line 421, in get_date_data
parsed_date = _DateLocaleParser.parse(
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/dateparser/date.py", line 178, in parse
return instance._parse()
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/dateparser/date.py", line 182, in _parse
date_data = self._parsers[parser_name]()
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/dateparser/date.py", line 196, in _try_freshness_parser
return freshness_date_parser.get_date_data(self._get_translated_date(), self._settings)
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/dateparser/date.py", line 234, in _get_translated_date
self._translated_date = self.locale.translate(
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/dateparser/languages/locale.py", line 131, in translate
relative_translations = self._get_relative_translations(settings=settings)
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/dateparser/languages/locale.py", line 158, in _get_relative_translations
self._generate_relative_translations(normalize=True))
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/dateparser/languages/locale.py", line 172, in _generate_relative_translations
pattern = DIGIT_GROUP_PATTERN.sub(r'?P<n>\d+', pattern)
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/regex/regex.py", line 702, in _compile_replacement_helper
is_group, items = _compile_replacement(source, pattern, is_unicode)
File "/home/asjohnston/miniconda3/envs/foo/lib/python3.10/site-packages/regex/_regex_core.py", line 1737, in _compile_replacement
raise error("bad escape \\%s" % ch, source.string, source.pos)
regex._regex_core.error: bad escape \d at position 7
The same asf_search.search call succeeds when installing v5.0.1 via pip:
(base) $ conda create --name bar pip
...
(base) $ conda activate bar
(bar) $ pip install asf_search==5.0.1
...
(bar) $ python
Python 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:06:46) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asf_search
>>> results = asf_search.search(start='2022-07-20')
/home/asjohnston/miniconda3/envs/bar/lib/python3.10/site-packages/dateparser/date_parser.py:35: PytzUsageWarning: The localize method is no longer necessary, as this time zone supports the fold attribute (PEP 495). For more details on migrating to a PEP 495-compliant implementation, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html
date_obj = stz.localize(date_obj)
>>> len(results)
2041
[Bug] ASFSearchResults.searchComplete incorrectly set to False for queries returning less than maxResults
Describe the bug
I believe the searchComplete attribute is incorrectly being set to False for searches that complete successfully but return less than maxResults. This causes ASFSearchResults.raise_if_incomplete() to raise an exception for successful searches.
To Reproduce
This query only returns 3 results, as expected, but seachComplete is False when maxResults>3
>>> asf_search.__version__
'6.5.0'
>>> params = {'beamMode': ['IW'],
... 'end': '2023-06-01T16:59:59Z',
... 'frame': [61],
... 'intersectsWith': 'POLYGON((93.4258 18.5769,94.3544 18.5769,94.3544 '
... '19.4733,93.6797 20.3947,92.8497 20.3947,92.8497 '
... '19.8044,93.4258 18.5769))',
... 'platform': 'S1',
... 'polarization': ['VV+VH'],
... 'processingLevel': 'SLC',
... 'relativeOrbit': [143],
... 'start': '2023-04-30T17:00:00Z'}
>>> results = asf_search.search(**params)
>>> len(results)
3
>>> results.searchComplete
False
>>> results = asf_search.search(**params, maxResults=4)
>>> len(results)
3
>>> results.searchComplete
False
>>> results = asf_search.search(**params, maxResults=3)
>>> len(results)
3
>>> results.searchComplete
True
[Bug] No module named 'asf_search.CMR' after recent version to 3.1.0
Describe the bug
After the recent update to version 3.1.0, I couldn't import asf. When importing I get the following error message (see attached). bug3.1.0.txt . It was working however with version 3.0.0.
To Reproduce
Steps to reproduce the behavior:
- import
asf_search as asf
Desktop (please complete the following information):
- OS: Ubuntu 20.04
[Bug/Feature] Frame search for new `ARIA_S1_GUNW` collection
This is an extension of #198.
Want to be able to search using temporal_baseline_days and frame in this ARIA collection. If I use a >=7.0.9, I am able to get the new collections of ARIA-S1-GUNWs:
import asf_search as asf
opts = asf.ASFSearchOptions(
shortName='ARIA_S1_GUNW',
maxResults=10,
)
scenes = asf.search(opts=opts)
Here is metadata for a sample product:
{'centerLat': None,
'centerLon': None,
'stopTime': '2024-03-16T23:36:26Z',
'fileID': 'S1-GUNW-A-R-091-tops-20240316_20240304-233600-00074W_00037S-PP-4b33-v3_0_1',
'flightDirection': 'ascending',
'pathNumber': 91,
'processingLevel': None,
'url': 'https://grfn.asf.alaska.edu/door/download/S1-GUNW-A-R-091-tops-20240316_20240304-233600-00074W_00037S-PP-4b33-v3_0_1.nc',
'startTime': '2024-03-16T23:35:34Z',
'sceneName': None,
'browse': ['https://grfn-public-prod.asf.alaska.edu/S1-GUNW-A-R-091-tops-20240316_20240304-233600-00074W_00037S-PP-4b33-v3_0_1.png'],
'platform': 'Sentinel-1A',
'bytes': None,
'md5sum': None,
'frameNumber': 14211,
'granuleType': None,
'orbit': [53013, 52838],
'polarization': 'VV',
'processingDate': '2024-03-26T03:18:06Z',
'sensor': None,
'groupID': None,
'pgeVersion': None,
'perpendicularBaseline': 70.8606,
'inputGranules': ['S1A_IW_SLC__1SDV_20240316T233534_20240316T233601_053013_066B21_474E',
'S1A_IW_SLC__1SDV_20240316T233559_20240316T233626_053013_066B21_F192',
'S1A_IW_SLC__1SDV_20240304T233534_20240304T233601_052838_06650A_A731',
'S1A_IW_SLC__1SDV_20240304T233559_20240304T233626_052838_06650A_CD37'],
'ariaVersion': '3.0.1',
'fileName': 'S1-GUNW-A-R-091-tops-20240316_20240304-233600-00074W_00037S-PP-4b33-v3_0_1.nc',
'beamModeType': 'IW',
's3Urls': [],
'additionalUrls': []}
So, I know the above product exists. However, when I update the search to include:
import asf_search as asf
opts = asf.ASFSearchOptions(
shortName='ARIA_S1_GUNW',
maxResults=10,
frame=14211
)
scenes = asf.search(opts=opts)
I get no results.
However, I am able to successfully query using temporal baseline:
opts = asf.ASFSearchOptions(
shortName='ARIA_S1_GUNW',
maxResults=10,
#frame=14211,
temporalBaselineDays=48
)
scenes = asf.search(opts=opts)
Beam mode does not work with geo-search?
Awesome tool!
Is this the expected behavior?
import asf_search as asf
from shapely.geometry import box
from dateparser import parse
import pandas as pd
# bay area
bounds: list = [-121.5, 34.95, -120.2, 36.25]
start_str: str = '2021-06-02'
end_str: str = '2021-06-09'
box_geo = box(*bounds)
start = parse(start_str)
end = parse(end_str)
results = asf.geo_search(platform=[asf.PLATFORM.SENTINEL1],
intersectsWith=box_geo.to_wkt(),
start=start,
end=end,
maxResults=100,
beamMode=[asf.BEAMMODE.IW]
)
df = pd.DataFrame([r.properties for r in results])
df.beamModeType.unique()
The output is array(['IW', 'slc'], dtype=object).
[Bug] 1. Querying with one large polygon vs several smaller polygons; 2. scenes fall outside of query WKT
Describe the bug
Suppose one wishes to either (1) break a large rectangular polygon into smaller rectangular polygons, or (2) consolidate several smaller and contiguous rectangular polygon into one large rectangular polygon. Keeping other query parameters constant, as long as the large rectangular polygon and the union of the smaller rectangular polygons cover the exact same area, one should expect the query to return identical set of scenes. This is not the case.
Moreover, one would expect the scenes returned with a given WKT polygon should fall inside of the given WKT polygon. Some scenes do not.
To Reproduce
Steps to reproduce the behavior:
- Run the attached
test_asf_polygon.txtscript. (I cannot attach.pyfiles so I edited the extension.)
Expected behavior
- The number of scenes returned from using one large polygon vs several smaller polygons covering the same area should be the same; they are not.
- All returned scene footprints should fall inside of the WKT; some do not.
Desktop (please complete the following information):
- OS: macOS 12.4
asf_search: 3.2.2- Python: 3.9.10
[Bug] Importing Error
Describe the bug
I tried to install asf-search on linux OpenSuSE 15.4, it was installed perfectly, but while I tried to import it is giving error. I have attached the error msg below. Please let me know how to correct it. My python version is 3.6.15
Desktop (please complete the following information):
- OS : OpenSuSE Leap15.4
It's very less for burst!!!
[Bug/Question] VH tifs only returned for OPERA-RTC-S1 - but polarizations are all VV in metadata. How do we get h5 products?
I am trying to recreate this search: https://search.asf.alaska.edu/#/?polygon=POINT(-99.8796%2016.8421)&zoom=9.133¢er=-99.712,16.370&resultsLoaded=true&dataset=OPERA-S1&granule=OPERA_L2_RTC-S1_T078-165486-IW2_20231024T004829Z_20231024T091416Z_S1A_30_v1.0&productTypes=RTC
resp = asf.geo_search(intersectsWith=Point(-99.8796, 16.8421).wkt,
processingLevel='RTC',
platform=asf.constants.SENTINEL1)
len(resp)
Gives 4 results.
resp = asf.geo_search(intersectsWith=Point(-99.8796, 16.8421).wkt,
processingLevel='RTC',
platform=asf.constants.SENTINEL1,
polarization=['VV', 'VH'])
len(resp)
Gives 8.
But the latter is only giving me VH. For example:
resp[-1].properties
gives
{'beamModeType': None,
'browse': ['https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0_BROWSE.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0_BROWSE.png',
'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0_BROWSE_low-res.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0_BROWSE_low-res.png',
'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0_BROWSE_thumbnail.png',
's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0_BROWSE_thumbnail.png'],
'bytes': None,
'centerLat': None,
'centerLon': None,
'faradayRotation': None,
'fileID': 'OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0',
'flightDirection': 'DESCENDING',
'groupID': 'S1A_IWDV_0533_0538_050688_041',
'granuleType': None,
'insarStackId': None,
'md5sum': None,
'offNadirAngle': None,
'orbit': 50688,
'pathNumber': 41,
'platform': 'Sentinel-1A',
'pointingAngle': None,
'polarization': 'VV',
'processingDate': '2023-10-10T05:32:17Z',
'processingLevel': 'RTC',
'sceneName': 'OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0',
'sensor': 'C-SAR',
'startTime': '2023-10-09T12:35:27Z',
'stopTime': '2023-10-09T12:35:30Z',
'url': 'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0_VH.tif',
'fileName': 'OPERA_L2_RTC-S1_T041-086893-IW1_20231009T123527Z_20231010T052433Z_S1A_30_v1.0_VH.tif',
'frameNumber': None}
Note in the above the url is for VH, but the polarization is for VV.
Also, how do I get the h5 files in that search?
Geographic Search AOI apparently using bounding box
Describe the bug
Geographic search results appear to using an AOI's bounding box instead of the precise AOI.
To Reproduce
Perform a geosearch with an AOI such as
POLYGON((-96.7615 25.5086,-96.6792 25.9766,-96.8375 26.7206,-96.7503 27.3507,-96.257 27.852,-94.9323 28.5583,-94.1864 29.0071,-93.0955 29.0558,-91.4369 28.5094,-90.2569 28.4409,-89.1097 28.5367,-88.2865 28.9,-87.8524 29.7154,-86.383 29.841,-85.2921 29.1433,-84.023 29.2696,-83.4998 28.2842,-83.1102 26.6247,-82.164 25.6353,-83.344 25.0619,-83.2549 24.0393,-81.407 24.0698,-79.7818 24.9812,-79.3143 26.246,-79.4924 27.4872,-79.6927 28.9584,-80.305 30.5338,-79.8375 31.4118,-77.9116 32.9101,-75.4404 34.1167,-74.6166 35.3425,-75.0285 36.7109,-74.0489 37.8183,-73.1211 40.2664,-72.1158 40.4863,-69.8849 40.5174,-69.245 40.7795,-69.5186 41.3921,-69.8396 42.4594,-69.5893 43.1045,-68.6395 43.3593,-67.4761 43.7878,-66.6899 44.1796,-66.4909 45.0479,-67.8794 44.9678,-69.3344 44.499,-71.1672 44.1076,-72.3644 43.183,-73.3496 42.1078,-74.5195 41.5095,-74.9975 40.9606,-77.4819 40.2428,-78.6797 39.2377,-80.0712 37.9501,-82.665 36.6663,-85.5147 34.9694,-88.2309 34.9876,-90.3603 34.9941,-94.8766 33.812,-98.6298 32.7573,-101.028 29.3073,-99.5497 26.8268,-98.2941 25.6052,-96.8348 25.5118,-96.7615 25.5086))
AOI:
Display the results on your map of choice:
Expected behavior
Results should match the actual AOI:
Additional context
Introduced with 5.0.0 or 5.0.1
Can't install latest `asf_search` on ARM Mac via conda due to `remotezip`
Describe the bug
It looks like the conda package for remotezip, despite being a pure python package, is not getting built as a noarch conda package: https://anaconda.org/conda-forge/remotezip
This means that on my M1 Mac, I get this error during the asf_search install:
$ mamba install "asf_search>=6.7"
...
Could not solve for environment specs
Encountered problems while solving:
- nothing provides remotezip >=0.10.0 needed by asf_search-6.7.0-pyhd8ed1ab_0
It seems like somebody could add remotezip to the list the conda has for migrating packages to osx-arm, but I can't tell from the remotezip recipe it wouldn't be built once as a noarch.
Desktop (please complete the following information):
$ mamba info
active environment : base
active env location : /Users/staniewi/miniconda3/envs/mapping
shell level : 2
user config file : /Users/staniewi/.condarc
populated config files : /Users/staniewi/.condarc
conda version : 4.13.0
conda-build version : 3.23.3
python version : 3.10.10.final.0
virtual packages : __osx=13.6.2=0
__unix=0=0
__archspec=1=arm64
base environment : /Users/staniewi/miniconda3/envs/mapping (writable)
conda av data dir : /Users/staniewi/miniconda3/envs/mapping/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/osx-arm64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/osx-arm64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /Users/staniewi/miniconda3/envs/mapping/pkgs
/Users/staniewi/.conda/pkgs
envs directories : /Users/staniewi/miniconda3/envs/mapping/envs
/Users/staniewi/.conda/envs
platform : osx-arm64
user-agent : conda/4.13.0 requests/2.28.2 CPython/3.10.10 Darwin/22.6.0 OSX/13.6.2
UID:GID : 503:20
netrc file : /Users/staniewi/.netrc
offline mode : False
Additional context
Note that I was able to install the asf_search via pip into my conda environment, so I don't have any urgent problems.
Is there a way to get the entire catalog of GUNW of metadata?
Is your feature request related to a problem? Please describe.
I want to get all the metadata for the entire catalog of GUNWs and make a pretty picture. I always get a subset (~150k products when there is 400k+). Do I have split up the globe?
Describe the solution you'd like
results = asf.search(processingLevel=[asf.PRODUCT_TYPE.GUNW_STD], maxResults=1_000_000) gives me all GUNW metadata
Describe alternatives you've considered
Not much.
Additional context
Make pretty pictures.
[Bug] Permission denied in newer versions
Download functionality of asf_search seems to have broken in v3.1.2 and later.
I suspect that it's no longer reading credentials in $HOME/.netrc as:
- the URLs can be downloaded manually after logging into EarthData
- I can start an ASFSession with my credentials and products will download.
MWE
import os
import asf_search as asf
## this fails in versions >= 3.1.2 with the error below script
dct_kw = {'platform': 'Sentinel-1', 'processingLevel': 'GUNW_STD', 'relativeOrbit': [124], 'lookDirection': None, 'intersectsWith': None}
scenes = asf.geo_search(**dct_kw)
scene1 = scenes[0]
scene1.download(os.getcwd())
## this works in all versions (obviously with real id/pass)
# session = asf.ASFSession()
# session.auth_with_creds('id', 'pass')
# scene1.download(os.getcwd(), session=session)
asf_search.exceptions.ASFAuthenticationError: HTTP 401: HTTP Basic: Access denied.
[Bug] Importing asf_search causes TypeError
Describe the bug
When you try to import asf_search, you get TypeError
To Reproduce
import asf_search as asf
Desktop (please complete the following information):
- Anaconda 3
- Python 3.8
- Pip 23.2.1
- asf-search 6.6.2
Error Context
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[2], line 1
----> 1 import asf_search as asf
File ~\Anaconda3\envs\satcrawler\lib\site-packages\asf_search\__init__.py:29
27 from .constants import *
28 from .health import *
---> 29 from .search import *
30 from .download import *
31 from .CMR import *
File ~\Anaconda3\envs\satcrawler\lib\site-packages\asf_search\search\__init__.py:1
----> 1 from .search import search
2 from .granule_search import granule_search
3 from .product_search import product_search
File ~\Anaconda3\envs\satcrawler\lib\site-packages\asf_search\search\search.py:7
5 from asf_search import ASFSearchResults
6 from asf_search.ASFSearchOptions import ASFSearchOptions
----> 7 from asf_search.search.search_generator import search_generator
9 def search(
10 absoluteOrbit: Union[int, Tuple[int, int], Iterable[Union[int, Tuple[int, int]]]] = None,
11 asfFrame: Union[int, Tuple[int, int], Iterable[Union[int, Tuple[int, int]]]] = None,
(...)
44 opts: ASFSearchOptions = None,
45 ) -> ASFSearchResults:
46 """
47 Performs a generic search using the ASF SearchAPI. Accepts a number of search parameters, and/or an ASFSearchOptions object. If an ASFSearchOptions object is provided as well as other specific parameters, the two sets of options will be merged, preferring the specific keyword arguments.
48
(...)
82 :return: ASFSearchResults(list) of search results
83 """
File ~\Anaconda3\envs\satcrawler\lib\site-packages\asf_search\search\search_generator.py:134
129 raise CMRIncompleteError(f"CMR returned page of incomplete results. Expected {min(INTERNAL.CMR_PAGE_SIZE, hits - sub_query_count)} results, got {len(items)}")
131 return items, hits, response.headers.get('CMR-Search-After', None)
--> 134 def process_page(items: list[ASFProduct], max_results: int, subquery_max_results: int, total: int, subquery_count: int, opts: ASFSearchOptions):
135 if max_results is None:
136 last_page = ASFSearchResults(items[:min(subquery_max_results - subquery_count, len(items))], opts=opts)
TypeError: 'type' object is not subscriptable
[Bug] 4.0.0 install dependency/import error
We're seeing some issues with importing the latest 4.0.0 release of asf_search in some contexts, such as Google Colab.
Specifically, during pip install asf_search, you may encounter the following error:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
Or, if you are able to work past this error, when importing asf_search you may encounter the following error:
ModuleNotFoundError: No module named 'importlib.metadata'
We're working to resolve that problem as quickly as possible and get an update out, but if you run into an install dependency error or importlib.metadata error as above, for now just use the previous version, 3.2.2.
Returning Wrong Frame using Stack from Prodcut
Hello Everyone,
I am trying to search and create stack using the stack_from_product but, I am getting wrong frames in the stack.
I used the following to search my data from the frame number 118
search_results = asf.search(
platform=asf.SENTINEL1,
processingLevel=asf.SLC,
beamMode=asf.IW,
flightDirection=asf.ASCENDING,
relativeOrbit=173, #path
frame=118,
#maxResults=20,
)
The results is like the following: we can see both the frame number and path is correct.
{'beamModeType': 'IW',
'browse': [],
'bytes': '4667648688',
'centerLat': '37.3913',
'centerLon': '71.0762',
'faradayRotation': None,
'fileID': 'S1A_IW_SLC__1SDV_20151117T131342_20151117T131409_008645_00C48E_D516-SLC',
'fileName': 'S1A_IW_SLC__1SDV_20151117T131342_20151117T131409_008645_00C48E_D516.zip',
'flightDirection': 'ASCENDING',
'frameNumber': '118',
'granuleType': 'SENTINEL_1A_FRAME',
'groupID': 'S1A_IWDV_0118_0123_008645_173',
'insarStackId': None,
'md5sum': 'fb1541f7040d93a66a1ec54cb3f72b63',
'offNadirAngle': None,
'orbit': '8645',
'pathNumber': '173',
'platform': 'Sentinel-1A',
.....
**After running the stack_from_product as following:
baseline_results = asf.baseline_search.stack_from_product(reference=search_results[-1])
I get in the stack other frames also such is 116, 117, 118, 121... I have only requested 118 as above in the search**
"properties": {
"beamModeType": "IW",
"browse": [],
"bytes": "2492173707",
"centerLat": "36.8931",
"centerLon": "71.1898",
"faradayRotation": null,
"fileID": "S1A_IW_SLC__1SSV_20141029T131328_20141029T131355_003045_0037AD_275D-SLC",
"fileName": "S1A_IW_SLC__1SSV_20141029T131328_20141029T131355_003045_0037AD_275D.zip",
"flightDirection": "ASCENDING",
"frameNumber": "116",
"granuleType": "SENTINEL_1A_FRAME",
..........................
"temporalBaseline": -385,
..............
Can anyone help me or explain why is that happening?
Best,
Najib
[Bug] HTTPError: 401 Client Error using EDL Token authentication
Describe the bug
The readme suggests using an EDL token for authentication is possible, but I'm running into an 'Unauthorized' error.
To Reproduce
import asf_search as asf
# MYEDLTOKEN from https://urs.earthdata.nasa.gov/users/scottyhq/user_tokens
session = asf.ASFSession().auth_with_token(MYEDLTOKEN)
# NOTE that the EDL token is stored in the `session.headers` as {'Authorization': 'Bearer XXXXXXXXXXX...'}
results = asf.geo_search(platform=[asf.PLATFORM.SENTINEL1],
beamMode=asf.BEAMMODE.IW,
processingLevel=asf.PRODUCT_TYPE.SLC,
intersectsWith='POINT (146.347 -37.486)',
relativeOrbit=45,
start='2021-09-01',
maxResults=1)
results.download('./', session=session)Desktop (please complete the following information):
- chrome browser running jupyterlab 3
- asf-search-3.0.4 from pip
Additional context
traceback below:
---------------------------------------------------------------------------
HTTPError Traceback (most recent call last)
/tmp/ipykernel_366/3425839706.py in <module>
11 maxResults=1)
12
---> 13 results.download('./', session=session)
/srv/conda/envs/notebook/lib/python3.9/site-packages/asf_search/ASFSearchResults.py in download(self, path, session, processes)
30 if processes == 1:
31 for product in self:
---> 32 product.download(path=path, session=session)
33 else:
34 pool = Pool(processes=processes)
/srv/conda/envs/notebook/lib/python3.9/site-packages/asf_search/ASFProduct.py in download(self, path, filename, session)
36 filename = self.properties['fileName']
37
---> 38 download_url(url=self.properties['url'], path=path, filename=filename, session=session)
39
40 def stack(self) -> UserList:
/srv/conda/envs/notebook/lib/python3.9/site-packages/asf_search/download/download.py in download_url(url, path, filename, session)
73 response = session.get(url, stream=True, hooks={'response': strip_auth_if_aws})
74
---> 75 response.raise_for_status()
76 with open(os.path.join(path, filename), 'wb') as f:
77 for chunk in response.iter_content(chunk_size=8192):
/srv/conda/envs/notebook/lib/python3.9/site-packages/requests/models.py in raise_for_status(self)
951
952 if http_error_msg:
--> 953 raise HTTPError(http_error_msg, response=self)
954
955 def close(self):
HTTPError: 401 Client Error: Unauthorized for url: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=BO_n7nTIlMljdvU6kRRB3g&response_type=code&redirect_uri=https://sentinel1.asf.alaska.edu/login&state=%2FSLC%2FSA%2FS1A_IW_SLC__1SDV_20211019T192511_20211019T192539_040192_04C2C2_EF09.zip&app_type=401[Bug] Typo in logger call
Describe the bug
Typo in logger warning generation, causes code to error out when over-riding options in Search.
To Reproduce
- Create ASFSearchOptions
- Override an already declared option using
merge_args - Trigger warning and code errors out due to typo
Expected behavior
Warning is generated and code continues.
This sort of typo can be suppressed by pylint and other PyCQA tooling.
[Bug] Differences between search.py in pip install version and github master branch version
Describe the bug
Line 143 in search/search.py is commented in the master branch, while it is uncommented in the search.py installed by pip.
To Reproduce
Steps to reproduce the behavior:
- python3 -m pip install asf_search
- vim /path/to/asf_search/search/search.py
- see line 143
Expected behavior
The behaviors should be consistent.
[Bug] Incorrect behavior when using `ThreadPoolExecutor` to download multiple files
Describe the bug
In order to download files while specifying desired file names, you must use the download_url function. However, when using download_url in concert with concurrent.future's ThreadPoolExecutor, the path to which each dataset is downloaded becomes mangled. Depending on the random order in which products are ready, the products are downloaded to a random one of the specified filenames.
To Reproduce
from concurrent.futures import ThreadPoolExecutor
from itertools import repeat
from pathlib import Path
import asf_search
urls = [
'https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20240313T140832_20240313T140859_052964_06694F_90B4/IW1/VV/7.tiff',
'https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20240301T140832_20240301T140859_052789_06635B_791A/IW1/VV/7.tiff',
'https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20240313T140832_20240313T140859_052964_06694F_90B4/IW1/VV/7.xml',
'https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20240301T140832_20240301T140859_052789_06635B_791A/IW1/VV/7.xml',
]
paths = [
Path('./burst_20240313.tif'),
Path('./burst_20240301.tif'),
Path('./burst_20240313.xml'),
Path('./burst_20240301.xml'),
]
session = asf_search.ASFSession()
with ThreadPoolExecutor() as executor:
executor.map(
asf_search.download_url,
urls,
[x.parent for x in paths],
[x.name for x in paths],
repeat(session, len(urls)),
)Expected behavior
The above should produce the same as:
from pathlib import Path
import asf_search
urls = [
'https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20240313T140832_20240313T140859_052964_06694F_90B4/IW1/VV/7.tiff',
'https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20240301T140832_20240301T140859_052789_06635B_791A/IW1/VV/7.tiff',
'https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20240313T140832_20240313T140859_052964_06694F_90B4/IW1/VV/7.xml',
'https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20240301T140832_20240301T140859_052789_06635B_791A/IW1/VV/7.xml',
]
paths = [
Path('./burst_20240313.tif'),
Path('./burst_20240301.tif'),
Path('./burst_20240313.xml'),
Path('./burst_20240301.xml'),
]
session = asf_search.ASFSession()
for url, path in zip(urls, paths):
asf_search.download_url(url, path.parent, path.name, session)[Feature] Harmonize parameters between search() and search_count()
Is your feature request related to a problem? Please describe.
It is often useful to be able to check how many records are available (via search_count() before requesting records (via search()). However, several parameters available in search() are not available in search_count().
Describe the solution you'd like
Have the same list of parameters for search() and search_count().
In particular, the Search API Keywords says the dataset keywords is preferred, but search_count() does not accept that keyword.
Describe alternatives you've considered
Request a list records via search() then calculate the lengths of the list, but that is pretty inefficient.
Other information
asf_search.__version__ == 6.7.2
The spatial baseline is NoData !!!
Since June 2023, the spatial baseline of sentinel image data obtained using Discovery-asf_search has been Nodata.
[Feature] Dateline Wrapping
This is thanks to @mgovorcin working on ISCE2 Hyp3 Plugin for ACCESS.
Is your feature request related to a problem? Please describe.
import asf_search as asf
results = asf.granule_search([
'S1B_IW_SLC__1SDV_20201215T054451_20201215T054521_024709_02F04A_6033',
'S1B_IW_SLC__1SDV_20211222T054456_20211222T054526_030134_039920_9391'])
print(results[0].geojson()['geometry'])
# {'coordinates': [[[-177.710968, 52.575905], [-177.271973, 50.790607], [179.223694, 50.39941], [178.642044, 52.181805], [-177.710968, 52.575905]]], 'type': 'Polygon'}
Describe the solution you'd like
I would like to see either a multipolygon or one that does not span the whole globe. Preferably the latter so I could easily map it. Obviously - this may be not synergistic for the great number of services ASF provides via this great library... but here's to hoping!
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.