ncpp / ocgis

OpenClimateGIS is a set of geoprocessing and calculation tools for CF-compliant climate datasets.

For documentation: https://ocgis.readthedocs.io/en/latest/

For additional project information: http://www.earthsystemcog.org/projects/openclimategis/

For questions or to file a bug report, please create a GitHub issue.

Overview

OpenClimateGIS (OCGIS) is a Python package designed for geospatial manipulation, subsetting, computation, and translation of spatiotemporal datasets stored in local NetCDF files or files served through THREDDS data servers. OpenClimateGIS has a straightforward, request-based API that is simple to use yet complex enough to perform a variety of computational tasks. The software is built entirely from open source packages.

OpenClimateGIS supports many file formats and NetCDF metadata conventions:

• Climate & Forecast (CF) Grid
• Unstructured Grid (UGRID)
• SCRIP
• ESMF Unstructured
• GDAL-supported vector formats (ESRI Shapefiles, File Geodatabases, GeoPackage, GeoJSON, etc.)
• Comma-Separated Value (CSV)

GIS Capabilities

• Subsetting (intersects and intersection) of climate datasets by bounding box, Shapely geometries, or shapefiles (city centroid, river reach, a single county or watershed, state boundaries).
• Time and level range subsetting. Also allows for arbitrary label-based slicing.
• Single or multi-dataset requests (concatenation).
• Area-weighted aggregation (spatial averaging) to selection geometries.
• Handles CF-based coordinate systems with full support for coordinate transformations (including the rotated pole coordinate system)
• Geometry wrapping and unwrapping to maintain logically consistent longitudinal domains.
• Polygon, line, and point geometric abstractions.
• ESMPy regridding

Data Conversion

• Access to local NetCDF data or data hosted remotely on a THREDDS (OPeNDAP protocol) data server. Only the piece of data selected by an area-of-interest is transferred from the remote server.
• Stream climate data to multiple formats.
• Extensible converter framework to add custom formats.
• Automatic generation of request metadata.
• Push data to a familiar format to perform analysis or keep the data as NumPy arrays, perform analysis, and dump to a supported format.

Computation

• Extensible computational framework for arbitrary inclusion of NumPy-based calculations.
• Apply computations to entire data arrays or temporal groups.
• Computed data may be streamed to any supported formats.

ocgis's Issues

meta output in converter return

ocg meta file should be written to each folder output location unless overridden by converter class.

disconnect download from geometric operations

This is useful for operations on remote data. A single request for the data would limit the number of hits to a remote server, but this may not be necessary and should only be implemented if the server loads are eventually deemed to high.

time: 2 to 3 days

sphinx style documentation

modify docstring contents to conform to sphinx specification for autodocumentation.

use time range bounds for subsetting

remove interp package.

shift enclosed items to api package.

add object methods for common operations

common operations will be exposed as methods of the OcgOperations object.

change name of operations dictionary

currently, the operations dictionary is called "desc". name should be changed to "ops".

spatial transformations for coordinate systems

spatial coordinates claim to be lat/lon but are often in ranges 0-360, etc. requiring a grid shift to actual WGS84 coordinates. provide options in the interface for requesting a spatial shift.

fabric deployment for linux systems

multivariate calculations

reintroduce multivariate support. requires alteration of calculation routines and iterators.

option for arbitrary user geometry selection

by passing integer ids, users may limit selection geometries.

code profiling

drill into variable dimensional information for initial info

use the dimension attribute to identify dimensions. since not all data will have ways to map centroids to bounds, those will need to be auto-discovered. recommended by charles and luca.

add option to overload all aspects of interface

this is especially important for remapping of coordinate variables to recognizable spatial coordinates.

spatial aggregation for raw=True, agg=True

make a determination for functions if spatial aggregation should occur following a calculation

raise exception when requested time range returns no data

integrate prefix into operations

operational web interface to subset, snippet, inspect, and basic calculations

metadata formatting for nc

identify how to handle metadata extracted from a netcdf. nc metadata should also be provided with all outputs.

handle orphan variables in nc output

there may be variables contained in a netcdf file that are not associated with the target variable (excluding bounds which are searched for automatically). it is likely not a good idea to strip these, however, some may just be junk leftover from a processing output. right now what is written are only the target variable's bounds and dimensions.

sample size for multivariate calculations not working

The sample size calculation for multivariate calculation requests is not working.

time: < 1 day

add url to meta output

may have to better encode URL.

netcdf output for data subset

fix nc output for non-calculation requests

currently, only sparse metadata is applied to netcdf outputs. also, for calculations, the time variables are wonky. for example, it is possible to generate a representative date for a month,year combination, but what in the case of just month? if the combinations were limited, it would be easier to implement. currently, a time group variable and dimension are created and the date parts added as variables (i.e. month, year).

ordering of coordinates in nc output

the need to handle wrapping of longitude causes potentially strange ordering of coordinates in the nc output for longitude values -180 to 0 (180 to 360). verifying coordinate shift through datum testing is also recommended.

netcdf output for calculations

transposed iterators

add support for transposed iterators (variable and calculation names as headers).

add spatial index intersection

currently only provides support for intersects. the spatial index is currently created inside the spatial interface. may need to move it out to connect to clip/intersection operations.

update inspect output to account for bounds on time and level variables

support for point request

only polygon request geometries current supported.

release - update documentation

multivariate calculation iterator for melted formats

iterator only working for keyed output

add argument to allow empty returns for geometries.

especially in the case of point data, some geometries may not overlap with dataset geometries.

provide ability to concatenate data requests across multiple subset options

this requires modifying the collection storage to store geometry and time vectors independently. changes to the iterators are also required. changes to operations may also be required. likely, the operations objects will be part of an operations collection.

fix temporal resolution in inspect output

element-wise calculation grouping

provide option to have calculations performed on each element w/out specifying an arbitrary group like "day" or "month|day".

multivariate calculations not working

add snippet option to every url.

provide the option for snippets on each calculation.

standardize handling of projections and geometry objects

zipped output

add option to converters to provide zipped output in addition to folder location.

remove request_url keyword argument from OcgOperations

this is confusing from the Python API perspective. it should be a private parameter used only by the software itself.

add option to union geometry

it may be the case that users with to union a provided geometry before making an operation like intersects.

add support for equivalent variable names from different datasets

with current variable storage, it is possible to overwrite a variable if the variable originates from different datasets (i.e. uris) but have the same name. storage should be able to handle this.

validation and geometry management for nc requests

for netcdf request, geometries must be unioned and only a subset of operations are available (i.e. no aggregation).

add inspect option for non-variable requests

new interface requires a variable designation. there should be an inspect option when no variable is provided.

sample size in iterator for multivariate output

the sample size is not included with multivariate calculation output iterators.

fix 'id' and 'ugid' naming conflict

shpidx geometry attribute name should be gid??

in the keyed output, the name of the geometry attribute appears to be invalid.

raise exception when requested level range returns no data

merge slug and ocg parameters into single class system

prototype object developed.