GithubHelp home page GithubHelp logo

tuw-geo / smap_io Goto Github PK

View Code? Open in Web Editor NEW
13.0 6.0 9.0 22.01 MB

Readers and converters for Soil Moisture Active Passive (SMAP) data.

License: MIT License

Python 100.00%
python remote-sensing smap soil-moisture earth-science

smap_io's Introduction

smap_io

image

image

image

image

SMAP (Soil Moisture Active Passive) data readers.

Works great in combination with pytesmo.

Citation

image

If you use the software in a publication then please cite it using the Zenodo DOI. Be aware that this badge links to the latest package version.

Please select your specific version at https://doi.org/10.5281/zenodo.596391 to get the DOI of that version. You should normally always use the DOI for the specific version of your record in citations. This is to ensure that other researchers can access the exact research artefact you used for reproducibility.

You can find additional information regarding DOI versioning at http://help.zenodo.org/#versioning

Installation

Setup of a complete environment with conda can be performed using the following commands:

$ conda create -q -n smap_io -c conda-forge numpy h5py pyproj netcdf4 pyresample pandas
$ source activate smap_io
$ pip install smap_io

You can also install all needed (conda and pip) dependencies at once using the following commands after cloning this repository. This is recommended for developers of the package.

$ git clone https://github.com/TUW-GEO/smap_io.git --recursive
$ cd smap_io
$ conda create -n smap_io python=3.6 # or any supported python version
$ source activate smap_io
$ conda update -f environment.yml
$ python setup.py develop

Supported Products

  • SPL3SMP: SMAP L3 Radiometer Global Daily 36 km EASE-Grid Soil Moisture

Additional products will we added when need arises, feel free to open an issue to add a new data product or even better a pull request.

Contribute

We are happy if you want to contribute. Please raise an issue explaining what is missing or if you find a bug. We will also gladly accept pull requests against our master branch for new features or bug fixes.

Guidelines

If you want to contribute please follow these steps:

  • Fork the smap_io repository to your account
  • make a new feature branch from the smap_io master branch
  • Add your feature
  • please include tests for your contributions in one of the test directories We use py.test so a simple function called test_my_feature is enough
  • submit a pull request to our master branch

smap_io's People

Contributors

cpaulik avatar mazweg avatar sebhahn avatar wpreimes avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

fzaussin fpl lmoesinger wpreimes adivicco47 rameezu scho-github

smap_io's Issues

Installation fails due to conflicting netcdf4 version

Hi, users are unable to run smap-io due to dependency conflict with netcdf4 package. As shown in the following full dependency graph of smap-io, pygeogrids requires netcdf4 *，while pynetcf requires netcdf4 <=1.2.8.

According to pip’s “first found wins” installation strategy, netcdf4 1.5.3 is the actually installed version. However, netcdf4 1.5.3 does not satisfy <=1.2.8.

Dependency tree-----------

smap-io - 0.3
| +- datedown(install version:0.3 version range:*)
| +- ease-grid(install version:0.1 version range:*)
| | +- numpy(install version:1.18.0rc1 version range:*)
| | +- pyproj(install version:2.4.2.post1 version range:*)
| +- h5py(install version:2.10.0 version range:*)
| +- pygeobase(install version:0.3.18 version range:*)
| | +- mock(install version:3.0.5 version range:*)
| | +- numpy(install version:1.18.0rc1 version range:*)
| | +- numpydoc(install version:0.9.1 version range:*)
| | | +- jinja2(install version:2.10.3 version range:>=2.3)
| | | | +- markupsafe(install version:1.1.1 version range:>=0.23)
| | | +- sphinx(install version:1.6.6 version range:>=1.6.5)
| | +- pygeogrids(install version:0.2.6 version range:>=0.1.7)
| | | +- ipython(install version:7.10.2 version range:*)
| | | +- mock(install version:3.0.5 version range:*)
| | | +- netcdf4(install version:1.5.3 version range:*)
| | | +- numpy(install version:1.18.0rc1 version range:*)
| | | +- numpydoc(install version:0.9.1 version range:*)
| | | | +- jinja2(install version:2.10.3 version range:>=2.3)
| | | | +- sphinx(install version:1.6.6 version range:>=1.6.5)
| | | +- pykdtree(install version:1.3.1 version range:*)
| | | | +- numpy(install version:1.18.0rc1 version range:*)
| | | +- pyproj(install version:2.4.2.post1 version range:*)
| | | +- scipy(install version:1.4.1 version range:*)
| | | +- sphinx(install version:1.5.6 version range:==1.5.6)
| | | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | | +- sphinx(install version:1.6.6 version range:*)
| | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | +- sphinx(install version:1.6.6 version range:*)
| +- pygeogrids(install version:0.2.6 version range:*)
| | +- ipython(install version:7.10.2 version range:*)
| | +- mock(install version:3.0.5 version range:*)
| | +- netcdf4(install version:1.5.3 version range:*)
| | +- numpy(install version:1.18.0rc1 version range:*)
| | +- numpydoc(install version:0.9.1 version range:*)
| | | +- jinja2(install version:2.10.3 version range:>=2.3)
| | | | +- markupsafe(install version:1.1.1 version range:>=0.23)
| | | +- sphinx(install version:1.6.6 version range:>=1.6.5)
| | +- pykdtree(install version:1.3.1 version range:*)
| | | +- numpy(install version:1.18.0rc1 version range:*)
| | +- pyproj(install version:2.4.2.post1 version range:*)
| | +- scipy(install version:1.4.1 version range:*)
| | +- sphinx(install version:1.5.6 version range:==1.5.6)
| | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | +- sphinx(install version:1.6.6 version range:*)
| +- pynetcf(install version:0.1.18 version range:==0.1.18)
| | +- mock(install version:3.0.5 version range:*)
| | +- netcdf4(install version:1.2.8 version range:<=1.2.8)
| | +- numpy(install version:1.18.0rc1 version range:*)
| | +- numpydoc(install version:0.9.1 version range:*)
| | | +- jinja2(install version:2.10.3 version range:>=2.3)
| | | | +- markupsafe(install version:1.1.1 version range:>=0.23)
| | | +- sphinx(install version:1.6.6 version range:>=1.6.5)
| | +- pygeobase(install version:0.3.18 version range:*)
| | | +- mock(install version:3.0.5 version range:*)
| | | +- numpy(install version:1.18.0rc1 version range:*)
| | | +- numpydoc(install version:0.9.1 version range:*)
| | | | +- jinja2(install version:2.10.3 version range:>=2.3)
| | | | +- sphinx(install version:1.6.6 version range:>=1.6.5)
| | | +- pygeogrids(install version:0.2.6 version range:>=0.1.7)
| | | | +- ipython(install version:7.10.2 version range:*)
| | | | +- mock(install version:3.0.5 version range:*)
| | | | +- netcdf4(install version:1.5.3 version range:*)
| | | | +- numpy(install version:1.18.0rc1 version range:*)
| | | | +- numpydoc(install version:0.9.1 version range:*)
| | | | +- pykdtree(install version:1.3.1 version range:*)
| | | | +- pyproj(install version:2.4.2.post1 version range:*)
| | | | +- scipy(install version:1.4.1 version range:*)
| | | | +- sphinx(install version:1.5.6 version range:==1.5.6)
| | | | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | | +- sphinx(install version:1.6.6 version range:*)
| | +- pygeobase(install version:0.3.18 version range:>=0.3.15)
| | | +- mock(install version:3.0.5 version range:*)
| | | +- numpy(install version:1.18.0rc1 version range:*)
| | | +- numpydoc(install version:0.9.1 version range:*)
| | | | +- jinja2(install version:2.10.3 version range:>=2.3)
| | | | +- sphinx(install version:1.6.6 version range:>=1.6.5)
| | | +- pygeogrids(install version:0.2.6 version range:>=0.1.7)
| | | | +- ipython(install version:7.10.2 version range:*)
| | | | +- mock(install version:3.0.5 version range:*)
| | | | +- netcdf4(install version:1.5.3 version range:*)
| | | | +- numpy(install version:1.18.0rc1 version range:*)
| | | | +- numpydoc(install version:0.9.1 version range:*)
| | | | +- pykdtree(install version:1.3.1 version range:*)
| | | | +- pyproj(install version:2.4.2.post1 version range:*)
| | | | +- scipy(install version:1.4.1 version range:*)
| | | | +- sphinx(install version:1.5.6 version range:==1.5.6)
| | | | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | | +- sphinx(install version:1.6.6 version range:*)
| | +- pygeogrids(install version:0.2.6 version range:*)
| | | +- ipython(install version:7.10.2 version range:*)
| | | +- mock(install version:3.0.5 version range:*)
| | | +- netcdf4(install version:1.5.3 version range:*)
| | | +- numpy(install version:1.18.0rc1 version range:*)
| | | +- numpydoc(install version:0.9.1 version range:*)
| | | | +- jinja2(install version:2.10.3 version range:>=2.3)
| | | | +- sphinx(install version:1.6.6 version range:>=1.6.5)
| | | +- pykdtree(install version:1.3.1 version range:*)
| | | | +- numpy(install version:1.18.0rc1 version range:*)
| | | +- pyproj(install version:2.4.2.post1 version range:*)
| | | +- scipy(install version:1.4.1 version range:*)
| | | +- sphinx(install version:1.5.6 version range:==1.5.6)
| | | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | | +- sphinx(install version:1.6.6 version range:*)
| | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | +- sphinx(install version:1.6.6 version range:*)
| +- repurpose(install version:0.7.1 version range:*)
| | +- more-itertools(install version:8.0.2 version range:*)
| | +- numpy(install version:1.18.0rc1 version range:*)
| | +- pygeogrids(install version:0.2.6 version range:>=0.1.3)
| | | +- ipython(install version:7.10.2 version range:*)
| | | +- mock(install version:3.0.5 version range:*)
| | | +- netcdf4(install version:1.5.3 version range:*)
| | | +- numpy(install version:1.18.0rc1 version range:*)
| | | +- numpydoc(install version:0.9.1 version range:*)
| | | | +- jinja2(install version:2.10.3 version range:>=2.3)
| | | | +- sphinx(install version:1.6.6 version range:>=1.6.5)
| | | +- pykdtree(install version:1.3.1 version range:*)
| | | | +- numpy(install version:1.18.0rc1 version range:*)
| | | +- pyproj(install version:2.4.2.post1 version range:*)
| | | +- scipy(install version:1.4.1 version range:*)
| | | +- sphinx(install version:1.5.6 version range:==1.5.6)
| | | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | | +- sphinx(install version:1.6.6 version range:*)
| | +- pynetcf(install version:0.1.18 version range:*)
| | | +- mock(install version:3.0.5 version range:*)
| | | +- netcdf4(install version:1.2.8 version range:<=1.2.8)
| | | +- numpy(install version:1.18.0rc1 version range:*)
| | | +- numpydoc(install version:0.9.1 version range:*)
| | | | +- jinja2(install version:2.10.3 version range:>=2.3)
| | | | +- sphinx(install version:1.6.6 version range:>=1.6.5)
| | | +- pygeobase(install version:0.3.18 version range:*)
| | | | +- mock(install version:3.0.5 version range:*)
| | | | +- numpy(install version:1.18.0rc1 version range:*)
| | | | +- numpydoc(install version:0.9.1 version range:*)
| | | | +- pygeogrids(install version:0.2.6 version range:>=0.1.7)
| | | | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | +- pygeobase(install version:0.3.18 version range:>=0.3.15)
| | | | +- mock(install version:3.0.5 version range:*)
| | | | +- numpy(install version:1.18.0rc1 version range:*)
| | | | +- numpydoc(install version:0.9.1 version range:*)
| | | | +- pygeogrids(install version:0.2.6 version range:>=0.1.7)
| | | | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | +- pygeogrids(install version:0.2.6 version range:*)
| | | | +- ipython(install version:7.10.2 version range:*)
| | | | +- mock(install version:3.0.5 version range:*)
| | | | +- netcdf4(install version:1.5.3 version range:*)
| | | | +- numpy(install version:1.18.0rc1 version range:*)
| | | | +- numpydoc(install version:0.9.1 version range:*)
| | | | +- pykdtree(install version:1.3.1 version range:*)
| | | | +- pyproj(install version:2.4.2.post1 version range:*)
| | | | +- scipy(install version:1.4.1 version range:*)
| | | | +- sphinx(install version:1.5.6 version range:==1.5.6)
| | | | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | +- sphinx-rtd-theme(install version:0.4.3 version range:*)
| | | | +- sphinx(install version:1.6.6 version range:*)
| | +- pyresample(install version:1.13.2 version range:*)
| | | +- configobj(install version:5.0.6 version range:*)
| | | | +- six(install version:1.13.0 version range:*)
| | | +- pykdtree(install version:1.3.1 version range:>=1.3.1)
| | | | +- numpy(install version:1.18.0rc1 version range:*)
| | | +- pyproj(install version:2.4.2.post1 version range:>=1.9.5.1)
| | | +- pyyaml(install version:5.2 version range:*)
| | | +- setuptools(install version:42.0.2 version range:>=3.2)
| | | +- six(install version:1.13.0 version range:*)
| | +- sphinx(install version:1.6.6 version range:==1.6.6)
| +- sphinx(install version:1.6.6 version range:==1.6.6)
| +- trollsift(install version:0.2.1 version range:==0.2.1)
| | +- six(install version:1.13.0 version range:*)

Thanks for your help.
Best,
Neolith

download with datedown

Hey! As is said I looked at the download script from @cpaulik and I couldn't make it work. Because the v3 data is not there anymore or it was moved/renamed...
Now I tried to do the download for the version 5 data and it also failed when using datedown. The problem is that the "Minor CRID Version Number" part of the file names changes at some point in time (after 2017-09-05) and also the "Product Counter" changes (so the names not only depend on the dt). So I tried to use the --recursive option in datedown/wget, but I only download the index.html then, unless I add -e robots=off (robots=off is also suggested in the official instructions).
But the robots option is not support in datedown (i've added it now and the download works fine). So what would you suggest? Do the download without datedown, update datedown?, hard code the change in the file names (I would not do that), or maybe you have some different solution?

Original test data and "Soil_Moisture_Retrieval_Data" and NetCDF: Bad chunk sizes

Trying to run test_reshuffle I get the following error:

OSError: Unable to create file (unable to open file: name = '/fs/cetus/fs2/mrb/armn/armngan/tmp/try_pytesmo/test_data/smap_io/tests/test_data/spl3smp/2015.04.01/SMAP_L3_SM_P_20150401_R13080_001.h5', errno = 17, error message = 'File exists', flags = 15, o_flags = c2)

Then trying to play with SMAP L3 data I downloaded from n5eil01u.ecs.nsidc.org I get the following error:

File "/data/ords/armn/armngan/miniconda2/envs/smap_io-environment/lib/python3.5/site-packages/smap_io/interface.py", line 76, in read
latitude = ds['Soil_Moisture_Retrieval_Data']['latitude'][:]
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/data/ords/armn/armngan/miniconda2/envs/smap_io-environment/lib/python3.5/site-packages/h5py/_hl/group.py", line 167, in getitem
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'Soil_Moisture_Retrieval_Data' doesn't exist)"

Finally, adapting to "Soil_Moisture_Retrieval_Data_AM" in interface.py (as it appears to be the new name), i get the following error:

(smap_io-environment) armngan@arxt17:~/cetus/tmp/try_pytesmo/test_data/smap_io/tests$ python test_reshuffle.py
/data/ords/armn/armngan/miniconda2/envs/smap_io-environment/lib/python3.5/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Converting data from 2016-04-01T00:00:00 to 2016-04-02T00:00:00 into folder /tmp/armngan/17803/tmp_qdmkv6m.
Traceback (most recent call last):
File "test_reshuffle.py", line 59, in
test_reshuffle()
File "test_reshuffle.py", line 47, in test_reshuffle
reshuffle.main(args)
File "/data/ords/armn/armngan/miniconda2/envs/smap_io-environment/lib/python3.5/site-packages/smap_io/reshuffle.py", line 137, in main
imgbuffer=args.imgbuffer)
File "/data/ords/armn/armngan/miniconda2/envs/smap_io-environment/lib/python3.5/site-packages/smap_io/reshuffle.py", line 85, in reshuffle
reshuffler.calc()
File "/data/ords/armn/armngan/miniconda2/envs/smap_io-environment/lib/python3.5/site-packages/repurpose/img2ts.py", line 300, in calc
attributes=self.ts_attributes)
File "/data/ords/armn/armngan/miniconda2/envs/smap_io-environment/lib/python3.5/site-packages/pynetcf/time_series.py", line 802, in write_ts_all_loc
dtype=data[key].dtype, chunksizes=chunksizes)
File "/data/ords/armn/armngan/miniconda2/envs/smap_io-environment/lib/python3.5/site-packages/pynetcf/base.py", line 207, in write_var
chunksizes=chunksizes, **kwargs)
File "netCDF4/_netCDF4.pyx", line 2291, in netCDF4._netCDF4.Dataset.createVariable (netCDF4/_netCDF4.c:18426)
File "netCDF4/_netCDF4.pyx", line 3297, in netCDF4._netCDF4.Variable.init (netCDF4/_netCDF4.c:31434)
RuntimeError: NetCDF: Bad chunk sizes.

Am I missing something ??

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.