odm2 / yoda-tools

Tools to validate and manage YODA files

ODM2

The next version of the Observations Data Model.

For more information about the ODM2 development project, visit the wiki.

Have a look at the ODM2 paper in Environmental Modelling & Software. It's open access!

Horsburgh, J. S., Aufdenkampe, A. K., Mayorga, E., Lehnert, K. A., Hsu, L., Song, L., Spackman Jones, A., Damiano, S. G., Tarboton, D. G., Valentine, D., Zaslavsky, I., Whitenack, T. (2016). Observations Data Model 2: A community information model for spatially discrete Earth observations, Environmental Modelling & Software, 79, 55-74, http://dx.doi.org/10.1016/j.envsoft.2016.01.010

If you are interested in learning more about how ODM2 supports different use cases, have a look at our recent paper in the Data Science Journal.

Hsu, L., Mayorga, E., Horsburgh, J. S., Carter, M. R., Lehnert, K. A., Brantley, S. L. (2017), Enhancing Interoperability and Capabilities of Earth Science Data using the Observations Data Model 2 (ODM2), Data Science Journal, 16(4), 1-16, http://dx.doi.org/10.5334/dsj-2017-004.

Getting Started with ODM2

SQL scripts for generating blank ODM2 databases can be found at the following locations:

• Microsoft SQL Server
• MySQL
• PostgreSQL
• SQLite

View Documentation of ODM2 Concepts

For more information on ODM2 concepts, examples, best practices, the ODM2 software ecosystem, etc., visit the Documentation page on the wiki.

View Diagrams and Documentation of the ODM2 Schema

Schema diagrams for the current version of the ODM2 schema are at:

• Entity Relationship Diagrams

Data Use Cases

The following data use cases are available. We have focused on designing ODM2 to support these data use cases. Available code and documentation show how these data use cases were mapped to the ODM2.

• Little Bear River - Hydrologic time series and water quality samples from an ODM 1.1.1 database. Implements an ODM2 database in Microsoft SQL Server.
• PRISM-XAN - Water quality depth profiles and samples from Puget Sound. Implements an ODM2 database in PostgreSQL.

Our Goal with ODM2

We are working to develop a community information model to extend interoperability of spatially discrete, feature based earth observations derived from sensors and samples and improve the capture, sharing, and archival of these data. This information model, called ODM2, is being designed from a general perspective, with extensibility for achieving interoperability across multiple disciplines and systems that support publication of earth observations.

Credits

This work was supported by National Science Foundation Grant EAR-1224638. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

ODM2 draws heavily form our prior work with the CUAHSI Hydrologic information system and ODM 1.1.1 (Horsburgh et al., 2008; Horsburgh and Tarboton, 2008), our experiences workin on the Critical Zone Observatory Integrated Data Management System (CZOData), and our experiences with the EarthChem systems (e.g., Lehnert et al., 2007; Lehnert et al., 2009). It also extensively uses concepts from the Open Geospatial Consortium's Observations & Measurements standard (Cox, 2007a; Cox, 2007b; Cox, 2011a; Cox, 2011b; ISO, 2011).

References

See a full list of ODM2 related references

Cox, S.J.D. (2007a). Observations and Measurements - Part 1 - Observation schema, OGC Implementation Specification, OGC 07-022r1. 73 + xi. http://portal.opengeospatial.org/files/22466.

Cox, S.J.D. (2007b). Observations and Measurements – Part 2 - Sampling Features, OGC Implementation Specification, OGC 07-002r3. 36 + ix. http://portal.opengeospatial.org/files/22467.

Cox, S.J.D. (2011a). Geographic Information - Observations and Measurements, OGC Abstract Specification Topic 20 (same as ISO 19156:2011), OGC 10-004r3. 54. http://dx.doi.org/10.13140/2.1.1142.3042.

Cox, S.J.D. (2011b). Observations and Measurements - XML Implementation, OGC Implementation Standard, OGC 10-025r1. 66 + x. http://portal.opengeospatial.org/files/41510 (accessed September 16, 2014).

Horsburgh, J.S., D.G. Tarboton, D.R. Maidment, and I. Zaslavsky (2008). A relational model for environmental and water resources data, Water Resources Research, 44, W05406, http://dx.doi.org/10.1029/2007WR006392.

Horsburgh, J.S., D.G. Tarboton (2008). CUAHSI Community Observations Data Model (ODM) Version 1.1.1 Design Specifications, CUAHSI Open Source Software Tools, http://www.codeplex.com/Download?ProjectName=HydroServer&DownloadId=349176.

ISO 19156:2011 - Geographic information -- Observations and Measurements, International Standard (2011), International Organization for Standardization, Geneva. http://dx.doi.org/10.13140/2.1.1142.3042.

Lehnert, K.A., Walker, D., Vinay, S., Djapic, B., Ash, J., Falk, B. (2007). Community-Based Development of Standards for Geochemical and Geochronological Data, Eos Trans. AGU, 88(52), Fall Meet. Suppl., Abstract IN52A-09.

Lehnert, K.A., Walker, D., Block, K.A., Ash, J.M., Chan, C. (2009). EarthChem: Next developments to meet new demands, American Geophysical Union, Fall Meeting 2009, Abstract #V12C-01.

yoda-tools's Issues

See CZO Yoda Data for a majority of issues

https://github.com/CZOData/YODA-File

ODM2 Output

Add the ability to save ODM2 api Objects into an ODM2 database. It must include validation so we are not adding duplicate data into the database

Add ability to run data loading tool from the command line

Need to add this option so data loading workflows can be automated via scripting.

Consider using a CSV-focused data-packaging format

YODA and the Excel format are both unfriendly to large datasets, Excel because of row limits, and Yoda because it appears to require loading the entire dataset into memory before being able to generate the YODA file. Supporting a CSV-focused packaging approach such as http://frictionlessdata.io/data-packages/ could make for a large-data-friendly ODM2 data format.

Yoda Output

create on odm2 yoda template from ODM2 api objects

Excel Parser not creating a Dataset object

The code for parsing excel templates, does not create a Dataset objects. And I'm actually not entirely sure where all of the information for the DataSet object should come from.

Excel template Output

Add the ability to save ODM2 API objects into an excel template

Track the packaging progress

@sreeder and @emiliom this issue is to help me track what I need to do here. Please let me know what you think.

• fix Travis-CI (#21)
• add AppVeyor testing (#22)
• add a setup.py (#24)
• how to manage versioning (@sreeder are you OK with versioneer or do you want to handle that differently?) (#24)
• entrypoint to call the GUI
• conda recipe and package for the ODM2 channel
• source (wheels?) distribution for PyPI
• "desktop app" metadata like icons, etc (pending static resources)

Investigate the tests failures

Before I start changing the code we should try to get Travis-CI to work so I don't introduce bugs or breakages to the code.

See https://travis-ci.org/ODM2/YODA-Tools/builds/231737622

@sreeder do you have an idea of what is going on with Travis-CI? The last green build was https://travis-ci.org/ODM2/YODA-Tools/builds/206801750

Yaml input

add ability to parse a .yaml file into ODM2 api objects

Excel Input

add ability to read an excel Yoda template into ODM2 api objects

xlrd not installed as dependency to conda package

I see that xlrd is on requirements.txt, but it doesn't get installed as a dependency when the yotdatools conda package is installed. Hmm.

@ocefpaf, any idea what's going on?

Merge repo with YODA-File

We discussed this during our BiG-CZ calls in fall 2017, but never created an issue at the time.

The motivation is that the YODA-File repo contains the Excel templates that YODA-Tools are designed to parse. It has been previously been quite confusing to keep these in sync, and the documentation for it all is also spread out between two repos.

Connected to ODM2/YODA-File#63

JSON Schemas

Create new Json Schemas

Differences in YODA files generated by YODA Tools versus the Excel Template

I have been comparing (as best I can) the files generated by @cdesyoun 's code and those that were generated directly from the Excel template. Here's some observed differences that could be fixed:

Testing using the YODA_TimeSeries_Example1_Template_0.3.1-alpha file:

• In the YODA block, "YODAPy" should be enclosed in the quotation marks within the CreationTool element

• Not sure this really matters, but the elements appear out of order in the files generated by YODA-Tools

• Citations block is missing

• AuthorLists block is missing

• DataSetCitations block is missing

• PrimaryPhone is missing from both the YODA file AND the Excel Template

• The YODA File contains 4 spatial reference systems in the SpatialReferences block even though only one is actually used in the file

• The ActionDescription in the Actions block should not be NULL. We need a description for the Action. The Excel template used "Generic observation action generated by Excel Template for TimeSeries"

• In the CoumnDefinitions block, do the "Label" and "ODM2Field" element values need to be in quotations?

• In the Data block you switch to using single quotes. I think you should consistently use double quotes.

• In the Data block, if you add a space before the date/time and one after the last value it's a lot easier to split the table into columns in an Excel spreadsheet. This is unnecessary syntax-wise, but would make Anthony happy. So, it would look like this:

  [ "2010-06-03 11:15:00", -5, 1.0208, 1.022, 1.0232 ]

Other Observations that I think are better in the YODA Tools generated files:

• In the SamplingFeature block, Elevation_m is enclosed in quotations in the YODA file generated from the Excel template and not in the one generated by YODA Tools. I'm assuming not enclosing it is the correct form for numeric data.
• In the YODA-Tools version there is only one Action, whereas in the version generated by the Excel template there are three Actions. I believe it is more correct to use only 1 action for this file. This was a limitation of the Excel Template.
• In the YODA-Tools version there is only one FeatureAction. This corresponds to the single Action, which is the desired behavior.
• In the YODA-Tools version there is only one ActionBy record. This is consistent with the single Action record, but illustrates a deficiency in the Excel template in that only a single "Data Collector" can be affiliated with a data column. Not sure we need to do anything about this.