wmo-cop / pyoscar

Pythonic API to WMO OSCAR

pyoscar

Pythonic API to WMO OSCAR

Overview

pyoscar provides a Pythonic API atop the WMO OSCAR HTTP API.

Installation

Requirements

• Python 3
• virtualenv or Conda

Dependencies

Dependencies are listed in requirements.txt. Dependencies are automatically installed during pyoscar installation.

Installing pyoscar

For users

To install the latest stable version:

pip3 install pyoscar

To keep up to date with stable updates:

pip3 install pyoscar -U

For developers

# setup virtualenv
python3 -m venv pyoscar
cd pyoscar
source bin/activate

# clone codebase and install
git clone https://github.com/wmo-cop/pyoscar.git
cd pyoscar
python3 setup.py build
python3 setup.py install

Running pyoscar via the Command Line

# help
pyoscar --help

# get version
pyoscar --version

# all subcommands support the following options:
# --env (depl or prod, default depl)
# --verbosity (ERROR, WARNING, INFO, DEBUG, default NONE)

# get all station identifiers
pyoscar stations

# get all station identifiers by country
pyoscar stations --country=CAN

# get all station identifiers by program affiliation
pyoscar stations --program=GAW

# get a single station by WIGOS identifier
pyoscar station 0-20000-0-71151

# get a single station by WIGOS identifier in summary mode
pyoscar station 0-20000-0-71151 --summary

# get a single station by WIGOS identifier in WIGOS XML format
pyoscar station 0-20000-0-71151 --format=XML

# get a single station by WIGOS identifier in WIGOS XML format in summary mode
pyoscar station 0-20000-0-71151 --format=XML --summary

# add verbose mode (ERROR, WARNING, INFO, DEBUG)
pyoscar station 0-20000-0-71151 --verbosity=DEBUG

# get contact by country
pyoscar contact -c Canada

# get contact by surname
pyoscar contact -s Karn

# get contact by organization
pyoscar contact -o "Environment Canada"

# upload WMDR XML (to production environment)
pyoscar upload -x /path/to/wmdr.xml -at API_TOKEN -e prod

# upload WMDR XML (to production environment) and save results to file
pyoscar upload -x /path/to/wmdr.xml -at API_TOKEN -e prod -l results.log

# use only GML ids is TRUE by default; use --no-gml-ids to set to FALSE
pyoscar upload -x /path/to/wmdr.xml -at API_TOKEN -e prod -l results.log --no-gml-ids

# harvest all records
pyoscar harvest --env=prod --directory=/path/to/dir

Using the pyoscar API

from pyoscar import OSCARClient

client = OSCARClient()

# get all Canadian stations
stations = client.get_stations(country='CAN')

# get all Canadian stations
stations = client.get_stations(program='GAW')

# get invididual station report
stn_leo = client.get_station_report('0-20000-0-71758')

# get invididual station report in summary mode
stn_leo = client.get_station_report('0-20000-0-71758', summary=True)

# upload WMDR XML

## instantiate client to OSCAR DEPL (default)
client = OSCARClient(api_token='foo')

## ...or to OSCAR production
client = OSCARClient(api_token='foo', env='prod')

with open('some-wmdr-file.xml') as fh:
    data = fh.read()

response = client.upload(data)

Development

Running Tests

# install dev requirements
pip3 install -r requirements-dev.txt

# run tests like this:
cd tests
python3 run_tests.py

# or like this:
python3 setup.py test

# measure code coverage
coverage run --source pyoscar setup.py test
coverage report -m

Releasing

create release (x.y.z is the release version)
vi pyoscar/__init__.py  # update __version__
git commit -am 'update release version x.y.z'
git push origin master
git tag -a x.y.z -m 'tagging release version x.y.z'
git push --tags

# upload to PyPI
rm -fr build dist *.egg-info
python3 setup.py sdist bdist_wheel --universal
twine upload dist/*

# publish release on GitHub (https://github.com/wmo-cop/pyoscar/releases/new)

# bump version back to dev
vi pyoscar/__init__.py  # update __version__
git commit -am 'back to dev'
git push origin master

Code Conventions

• PEP8

Bugs and Issues

All bugs, enhancements and issues are managed on GitHub.

Contact

• Tom Kralidis

pyoscar's Issues

Use OAI-PMH for downloading individual records

I want to download the XML WMDR records for > 100 stations, downloading via the API is slow compared to using the OAI endpoint. Consider using OAI endpoint for individual records:, relevant line here:

pyoscar/pyoscar/__init__.py

Line 118 in a18e630

request = f'{self.api_url}/search/station'

This will also require updating of the xpaths.

Comparison for single record (using curl):

> (.venv) wis2downloader@0d5597e53856:~$ time curl -o /dev/null "https://oscar.wmo.int/surface/rest/api/wmd/download/0-20000-0-08001"
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100 99081    0 99081    0     0  21150      0 --:--:--  0:00:04 --:--:-- 28504
> real    0m4.693s
> user    0m0.035s
> sys     0m0.000s
> (.venv) wis2downloader@0d5597e53856:~$ time curl -o /dev/null "https://oscar.wmo.int/oai/provider?verb=GetRecord&metadataPrefix=wmdr&identifier=0-20000-0-08001"
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100 99752    0 99752    0     0   583k      0 --:--:-- --:--:-- --:--:--  586k
> real    0m0.174s
> user    0m0.027s
> sys     0m0.000s

PyPi version outdated

Hey there, thanks for the effort to bring this powerful tool to Python. Unfortunately the examples given in the README are outdated for the PyPi version.

For example
pyoscar station --identifier 0-20000-0-71151
does not work under the PyPI verison. Can you create a new release for the current version?

Upload xml-files without using gml id for Data URL after latest OSCAR/Surface release 1.8.4

Dear Tom
I have been using pyoscar with great success to upload xml-files to OSCAR/Surface containing instrument metadata of the EUMETNET E-PROFILE network. With the last OSCAR/Surface release 1.8.4. this September, a new feature was implemented leading to the fact that the field "Data URL" under Deployment (Observations/Measurements par) is not filled anymore even if my xml files still contain the respective information. According to Lucia Cappelletti (MeteoSwiss) this is due to the fact that with the new release several Data URLs can now be provided and therefore a respective gml id has been introduced. Nevertheless, it is still possible to submit the xml files without such a corresponding gml id, but in that case, the POST Request during the upload has to be supplemented with the parameter «UseOnlyGmlIds=FALSE». Would it be possible for you to implement this in pyoscar?
In advance, thanks so much for your feedback and best regards
Simone