johntruckenbrodt / spatialist Goto Github PK
View Code? Open in Web Editor NEWA Python module for spatial data handling
License: MIT License
A Python module for spatial data handling
License: MIT License
Since Sep 26th, files packaged with Copernicus Data Service Ecosystem do not include implicit directories in the archive namelist.
These are valid zipfiles, but this leads to issues with packages like PyroSAR that use this to indicate if an archive is.
Example namelists
Old SAFE archive with directories in namelist:
['S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE-report-20231005T214402.pdf',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/calibration-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/calibration-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/noise-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/noise-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/rfi/',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/rfi/rfi-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/rfi/rfi-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/manifest.safe',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/measurement/',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/measurement/s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.tiff',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/measurement/s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.tiff',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/icons/',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/icons/logo.png',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/map-overlay.kml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/product-preview.html',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/quick-look.png',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-calibration.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-measurement.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-noise.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-product.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-quicklook.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-rfi.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-map-overlay.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-object-types.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-product-preview.xsd']
SAFE archive without implicit directories in namelist:
['S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE-report-20231005T214402.pdf',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/calibration-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/calibration-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/noise-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/noise-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/rfi/rfi-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/rfi/rfi-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/manifest.safe',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/measurement/s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.tiff',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/measurement/s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.tiff',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/icons/logo.png',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/map-overlay.kml',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/product-preview.html',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/quick-look.png',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/thumbnail.png',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-calibration.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-measurement.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-noise.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-product.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-quicklook.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-rfi.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-map-overlay.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-object-types.xsd',
'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-product-preview.xsd']
Hi John!
I encountered a problem with spatialist when I wanted to use the Archive-function from pyroSAR.
RuntimeError: failed to load extension spatialite
...was the error message I got.
From what I understand it's a common problem to get the spatialite extension to work under Windows but fortunately I found a solution that might solve this issue for people who are using Win10. It worked for me at least.
The solution is described here in the sqlitebrowser
wiki, which I assume is a source that is quite trustworthy ๐
Quick guide for other people who might read this:
C:\Users\USERNAME\.spatialist\mod_spatialite\
and delete the file libstdc++_64-6.dll
x86_64-win32-seh
files from here or use this direct linklibstdc++-6.dll
and libgcc_s_seh-1.dll
from the bin
folder into the directory mentioned in step 1libstdc++-6.dll
to libstdc++_64-6.dll
As a follow up to #24, it should be possible to use NaN
as the nodata value when writing rasters.
Currently it fails here:
spatialist/spatialist/raster.py
Lines 1154 to 1155 in 15583e5
Hi,
I am using Rasterviewer for the interactive plotting of a stack of Sentinel-1 data (coherence time series in ENVI format). The problem is that although the stack is sorted by date of acquisition, in the rasterviewer sorting is done according to Sentinel-1 A and Sentinel-1 B. Meaning that in the slider first the sorted S1A images are listed then comes S1B images.
Since I am dealing with coherence images(combination of two image), my bandnames in the header looks like this:
S1A_20201502T160245_S1B_20201506T160326
I edited the header file and took only the first date, i.e. 20201502, it seems that it has no impact on sorting.
Could you tell how to fix this issue?
thank you in advance.
I am trying to stack a collection of processed geotiffs together. The Geotiffs are processed using pyroSAR and the CRS is set to EPSG:4326. However, when stacking these images together I run into the following GDAL warp error in the stacking function.
Traceback (most recent call last):
File "utils/process_s1.py", line 154, in <module>
make_stack(args.outdir, roi, mosaic="daily")
File "utils/process_s1.py", line 110, in make_stack
shapefile=roi, sortfun=seconds, separate=True, overwrite=False)
File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/raster.py", line 1268, in stack
multicore(gdalwarp, cores=cores, multiargs={'src': srcfiles, 'dst': dstfiles}, options=options_warp)
File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/ancillary.py", line 357, in multicore
raise (item.re_raise())
File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/ancillary.py", line 396, in re_raise
reraise(self.ee, None, self.tb)
File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/ancillary.py", line 393, in reraise
raise tp.with_traceback(tb)
File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/ancillary.py", line 335, in wrapper
return function(**kwargs)
File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/auxil.py", line 133, in gdalwarp
raise RuntimeError('{}:\n src: {}\n dst: {}\n options: {}'.format(str(e), src, dst, options))
RuntimeError: Cannot find coordinate operations from `EPSG:4326' to `EPSG:4326':
src: /vsimem/S1A__IW___A_20190821T231125_VV_tnr_bnr_Orb_Cal_TC.vrt
dst: /processed/stack/S1A__IW___A_20190821T231125_VV_tnr_bnr_Orb_Cal_TC.tif
options: {'options': ['-q', '-co', 'COMPRESS=DEFLATE', '-co', 'PREDICTOR=2', '-of', 'GTiff', '-te', '-79.0411376953125', '25.7998911820883343', '-76.8878173828125', '27.029770731463536', '-tr', '10', '10', '-r', 'bilinear', '-srcnodata', '0', '-dstnodata', '0', '-multi', '-ovr', 'AUTO'], 'format': 'GTiff', 'outputBounds': (-79.0411376953125, 25.799891182088334, -76.8878173828125, 27.029770731463536), 'multithread': True, 'dstNodata': 0, 'xRes': 10, 'yRes': 10, 'resampleAlg': 'bilinear', 'srcNodata': 0}
(called function 'gdalwarp' with args {'src': '/vsimem/S1A__IW___A_20190821T231125_VV_tnr_bnr_Orb_Cal_TC.vrt', 'dst': '/processed/stack/S1A__IW___A_20190821T231125_VV_tnr_bnr_Orb_Cal_TC.tif', 'options': {'options': ['-q', '-co', 'COMPRESS=DEFLATE', '-co', 'PREDICTOR=2'], 'format': 'GTiff', 'outputBounds': (-79.0411376953125, 25.799891182088334, -76.8878173828125, 27.029770731463536), 'multithread': True, 'dstNodata': 0, 'xRes': 10, 'yRes': 10, 'resampleAlg': 'bilinear', 'srcNodata': 0}})
The Vector object only supports ESRI Shapefiles, although GDAL supports many more formats. It would be good to allow for GeoJSON too as this is a common format in many applications.
Another possibility would be to be able to initialize Vector from Fiona or Shapely objects which are commonly used by GeoPandas etc.
If I am using a Vector object for the shapefile keyword the stacking fails at line 1124 with OSError: File exits: "filename" even if i use overwrite=True.
Without the shapefile keyword it works as expected.
When I tried to import the package pyroSAR
, the package spatialist
caused this error:
Traceback (most recent call last):
File "/home/xxx/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-8-225f6e7efdf2>", line 1, in <module>
conn.enable_load_extension(True)
AttributeError: 'sqlite3.Connection' object has no attribute 'enable_load_extension'
Within conda the package pysqlite2
is not installed. Command 'conda install pysqlite' does not work. After trying to install it with pip this message appeared:
xxx@yyy:~$ pip install pysqlite2
Collecting pysqlite2
Could not find a version that satisfies the requirement pysqlite2 (from versions: )
No matching distribution found for pysqlite2
After a very long search, I came across a pysqlite package in Anaconda Cloud:
conda install -c kbchoi pysqlite
pyroSAR
to directly issue an error just because you can't load package spatialist
?Traceback (most recent call last):
...
ras.write(np_name_tmp, format=driver, array=out.astype('float32'), nodata=0,
File "*\Miniconda3\envs\COPA\lib\site-packages\spatialist\raster.py", line 1121, in write
outDataset = driver.Create(outname, self.cols, self.rows, self.bands, dtype, options)
File "*\Miniconda3\envs\COPA\lib\site-packages\osgeo\gdal.py", line 1952, in Create
return _gdal.Driver_Create(self, *args, **kwargs)
RuntimeError: GDALDriver::Create() ... no create method implemented for this format.
I've been trying to use SNAP in containers lately. The PyroSAR interface is quite handy. However, there is no need to install the entire Jupyter stack with all its dependencies. It really bloats the container.
I saw the following issue in the PyroSAR repository: johntruckenbrodt/pyroSAR#140
What needs to be done to make this split of dependencies happen?
Is the following modification of setup.py
all that is needed?
install_requires=[
"progressbar2",
- "jupyter",
- "IPython",
- "ipywidgets",
- "matplotlib",
"pathos>=0.2",
"numpy",
"tblib",
"pyyaml",
"dill",
],
+ extras_requires={
+ "visualization": ["IPython", "ipywidgets", "matplotlib", "jupyter"],
+ "testing": ["pytest"],
+ },
I am new to Python packages, and I'm using Python Poetry to handle my own dependencies. Because of this, I'm not very sure of how setup.py
works.
With the above modification, I am able to install PyroSAR with Python Poetry with this in my pyproject.toml
[tool.poetry.dependencies]
python = "^3.8"
spatialist = {path = "../spatialist"}
pyroSAR = "^0.12.1"
where the local path is just this git repository with the modified setup.py
.
Dealing with geospatial data it is quite common to use S3 or GCS buckets. There are Python wrappers to make these systems appear more like standard filesystems (e.g. s3fs). I'd propose that the finder method accepts a "filesystem" argument such that an s3fs filesystem object can be passed to it for searching a bucket for imagery.
S3FS wraps most of the standard os functions, such as walk, glob etc.
I have two S1 granules which are part of a larger group of images that I am trying to stack. The aggregation filter is set to mosaic all images on the same day together. However, when the two granules are mosaicked there is a section where data is missing.
You can see the boundary of the granules here by the thin white line
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.