GithubHelp home page GithubHelp logo

spatialist's People

Contributors

felixcremer avatar ibaris avatar johntruckenbrodt avatar maawoo avatar tyler-c2s avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

ibaris grseb9s felixcremer ningxue123456 sumesh1 jena-earth-observation-school maawoo mmarionncern tyler-c2s

spatialist's Issues

finder fails for CDSE SAFE ZIP files without implicit directoruies

Since Sep 26th, files packaged with Copernicus Data Service Ecosystem do not include implicit directories in the archive namelist.

These are valid zipfiles, but this leads to issues with packages like PyroSAR that use this to indicate if an archive is.

Example namelists

Old SAFE archive with directories in namelist:

['S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE-report-20231005T214402.pdf',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/calibration-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/calibration-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/noise-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/noise-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/rfi/',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/rfi/rfi-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/rfi/rfi-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/manifest.safe',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/measurement/',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/measurement/s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.tiff',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/measurement/s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.tiff',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/icons/',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/icons/logo.png',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/map-overlay.kml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/product-preview.html',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/quick-look.png',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-calibration.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-measurement.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-noise.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-product.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-quicklook.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-rfi.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-map-overlay.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-object-types.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-product-preview.xsd']

SAFE archive without implicit directories in namelist:

['S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE-report-20231005T214402.pdf',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/calibration-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/calibration-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/noise-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/calibration/noise-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/rfi/rfi-s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/rfi/rfi-s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/annotation/s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.xml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/manifest.safe',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/measurement/s1a-iw-grd-vh-20231005t190129-20231005t190154-050634-06199a-002.tiff',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/measurement/s1a-iw-grd-vv-20231005t190129-20231005t190154-050634-06199a-001.tiff',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/icons/logo.png',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/map-overlay.kml',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/product-preview.html',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/quick-look.png',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/preview/thumbnail.png',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-calibration.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-measurement.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-noise.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-product.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-quicklook.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-level-1-rfi.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-map-overlay.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-object-types.xsd',
 'S1A_IW_GRDH_1SDV_20231005T190129_20231005T190154_050634_06199A_4144.SAFE/support/s1-product-preview.xsd']

Problem with mod_spatialite when using Win10 (...and a solution)

Hi John!
I encountered a problem with spatialist when I wanted to use the Archive-function from pyroSAR.
RuntimeError: failed to load extension spatialite ...was the error message I got.
From what I understand it's a common problem to get the spatialite extension to work under Windows but fortunately I found a solution that might solve this issue for people who are using Win10. It worked for me at least.
The solution is described here in the sqlitebrowser wiki, which I assume is a source that is quite trustworthy 😉

Quick guide for other people who might read this:

  1. Go to C:\Users\USERNAME\.spatialist\mod_spatialite\ and delete the file libstdc++_64-6.dll
  2. Download one of the x86_64-win32-seh files from here or use this direct link
  3. Extract the zip-file somewhere and copy the files libstdc++-6.dll and libgcc_s_seh-1.dll from the bin folder into the directory mentioned in step 1
  4. Rename libstdc++-6.dll to libstdc++_64-6.dll

issue of sorting stacked images in Rasterviewer

Hi,

I am using Rasterviewer for the interactive plotting of a stack of Sentinel-1 data (coherence time series in ENVI format). The problem is that although the stack is sorted by date of acquisition, in the rasterviewer sorting is done according to Sentinel-1 A and Sentinel-1 B. Meaning that in the slider first the sorted S1A images are listed then comes S1B images.
Since I am dealing with coherence images(combination of two image), my bandnames in the header looks like this:
S1A_20201502T160245_S1B_20201506T160326

I edited the header file and took only the first date, i.e. 20201502, it seems that it has no impact on sorting.
Could you tell how to fix this issue?
thank you in advance.

GDAL warp error in stack when using same CRS and SRS

I am trying to stack a collection of processed geotiffs together. The Geotiffs are processed using pyroSAR and the CRS is set to EPSG:4326. However, when stacking these images together I run into the following GDAL warp error in the stacking function.

Traceback (most recent call last):
  File "utils/process_s1.py", line 154, in <module>
    make_stack(args.outdir, roi, mosaic="daily")
  File "utils/process_s1.py", line 110, in make_stack
    shapefile=roi, sortfun=seconds, separate=True, overwrite=False)
  File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/raster.py", line 1268, in stack
    multicore(gdalwarp, cores=cores, multiargs={'src': srcfiles, 'dst': dstfiles}, options=options_warp)
  File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/ancillary.py", line 357, in multicore
    raise (item.re_raise())
  File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/ancillary.py", line 396, in re_raise
    reraise(self.ee, None, self.tb)
  File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/ancillary.py", line 393, in reraise
    raise tp.with_traceback(tb)
  File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/ancillary.py", line 335, in wrapper
    return function(**kwargs)
  File "/home/lloyd/anaconda3/envs/gdal/lib/python3.7/site-packages/spatialist/auxil.py", line 133, in gdalwarp
    raise RuntimeError('{}:\n  src: {}\n  dst: {}\n  options: {}'.format(str(e), src, dst, options))
RuntimeError: Cannot find coordinate operations from `EPSG:4326' to `EPSG:4326':
  src: /vsimem/S1A__IW___A_20190821T231125_VV_tnr_bnr_Orb_Cal_TC.vrt
  dst: /processed/stack/S1A__IW___A_20190821T231125_VV_tnr_bnr_Orb_Cal_TC.tif
  options: {'options': ['-q', '-co', 'COMPRESS=DEFLATE', '-co', 'PREDICTOR=2', '-of', 'GTiff', '-te', '-79.0411376953125', '25.7998911820883343', '-76.8878173828125', '27.029770731463536', '-tr', '10', '10', '-r', 'bilinear', '-srcnodata', '0', '-dstnodata', '0', '-multi', '-ovr', 'AUTO'], 'format': 'GTiff', 'outputBounds': (-79.0411376953125, 25.799891182088334, -76.8878173828125, 27.029770731463536), 'multithread': True, 'dstNodata': 0, 'xRes': 10, 'yRes': 10, 'resampleAlg': 'bilinear', 'srcNodata': 0}
(called function 'gdalwarp' with args {'src': '/vsimem/S1A__IW___A_20190821T231125_VV_tnr_bnr_Orb_Cal_TC.vrt', 'dst': '/processed/stack/S1A__IW___A_20190821T231125_VV_tnr_bnr_Orb_Cal_TC.tif', 'options': {'options': ['-q', '-co', 'COMPRESS=DEFLATE', '-co', 'PREDICTOR=2'], 'format': 'GTiff', 'outputBounds': (-79.0411376953125, 25.799891182088334, -76.8878173828125, 27.029770731463536), 'multithread': True, 'dstNodata': 0, 'xRes': 10, 'yRes': 10, 'resampleAlg': 'bilinear', 'srcNodata': 0}})

Allow additional filetypes for Vector

The Vector object only supports ESRI Shapefiles, although GDAL supports many more formats. It would be good to allow for GeoJSON too as this is a common format in many applications.

Another possibility would be to be able to initialize Vector from Fiona or Shapely objects which are commonly used by GeoPandas etc.

Unable to load extension (sqlite3.Connection)

When I tried to import the package pyroSAR, the package spatialist caused this error:

Traceback (most recent call last):
  File "/home/xxx/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-8-225f6e7efdf2>", line 1, in <module>
    conn.enable_load_extension(True)
AttributeError: 'sqlite3.Connection' object has no attribute 'enable_load_extension'

Within conda the package pysqlite2 is not installed. Command 'conda install pysqlite' does not work. After trying to install it with pip this message appeared:

xxx@yyy:~$ pip install pysqlite2
Collecting pysqlite2
  Could not find a version that satisfies the requirement pysqlite2 (from versions: )
No matching distribution found for pysqlite2

After a very long search, I came across a pysqlite package in Anaconda Cloud:

conda install -c kbchoi pysqlite

My question is this:

  • Is it necessary for package pyroSAR to directly issue an error just because you can't load package spatialist ?
  • Is there an alternative to how to work around this problem?
  • Is it possible that this is a Linux Ubuntu 18.10 problem? With Ubuntu 18.04 I unfortunately didn't try it.
  • Could it be that I have installed conda in my home directory and not in usr/bin?

enable writing COGs with Raster.write 

Traceback (most recent call last):
  ...
    ras.write(np_name_tmp, format=driver, array=out.astype('float32'), nodata=0,
  File "*\Miniconda3\envs\COPA\lib\site-packages\spatialist\raster.py", line 1121, in write
    outDataset = driver.Create(outname, self.cols, self.rows, self.bands, dtype, options)
  File "*\Miniconda3\envs\COPA\lib\site-packages\osgeo\gdal.py", line 1952, in Create
    return _gdal.Driver_Create(self, *args, **kwargs)
RuntimeError: GDALDriver::Create() ... no create method implemented for this format.

Optional dependencies

I've been trying to use SNAP in containers lately. The PyroSAR interface is quite handy. However, there is no need to install the entire Jupyter stack with all its dependencies. It really bloats the container.

I saw the following issue in the PyroSAR repository: johntruckenbrodt/pyroSAR#140

What needs to be done to make this split of dependencies happen?

Is the following modification of setup.py all that is needed?

    install_requires=[
        "progressbar2",
-       "jupyter",
-       "IPython",
-       "ipywidgets",
-       "matplotlib",
        "pathos>=0.2",
        "numpy",
        "tblib",
        "pyyaml",
        "dill",
    ],
+   extras_requires={
+       "visualization": ["IPython", "ipywidgets", "matplotlib", "jupyter"],
+       "testing": ["pytest"],
+   },

I am new to Python packages, and I'm using Python Poetry to handle my own dependencies. Because of this, I'm not very sure of how setup.py works.

With the above modification, I am able to install PyroSAR with Python Poetry with this in my pyproject.toml

[tool.poetry.dependencies]
python = "^3.8"
spatialist = {path = "../spatialist"}
pyroSAR = "^0.12.1"

where the local path is just this git repository with the modified setup.py.

Make more generic finder methods

Dealing with geospatial data it is quite common to use S3 or GCS buckets. There are Python wrappers to make these systems appear more like standard filesystems (e.g. s3fs). I'd propose that the finder method accepts a "filesystem" argument such that an s3fs filesystem object can be passed to it for searching a bucket for imagery.

S3FS wraps most of the standard os functions, such as walk, glob etc.

stack not mosaicing correctly

I have two S1 granules which are part of a larger group of images that I am trying to stack. The aggregation filter is set to mosaic all images on the same day together. However, when the two granules are mosaicked there is a section where data is missing.

You can see the boundary of the granules here by the thin white line
granuals

However, the mosaic is missing a wedge of data
mosaic

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.