GithubHelp home page GithubHelp logo

odm2 / cz-manager Goto Github PK

View Code? Open in Web Editor NEW
9.0 9.0 12.0 46.08 MB

CZ Manager (formerly ODM2 Admin) is an application for site level data management of environmental observations using Observation Data Model 2 (ODM2) for documentation and a detailed walkthrough see:

Home Page: http://odm2.github.io/CZ-Manager

License: MIT License

Python 35.91% HTML 9.97% CSS 2.29% JavaScript 40.42% Shell 0.08% PLpgSQL 10.57% Dockerfile 0.05% SCSS 0.71%

cz-manager's Introduction

ODM2

The next version of the Observations Data Model.

For more information about the ODM2 development project, visit the wiki.

Have a look at the ODM2 paper in Environmental Modelling & Software. It's open access!

Horsburgh, J. S., Aufdenkampe, A. K., Mayorga, E., Lehnert, K. A., Hsu, L., Song, L., Spackman Jones, A., Damiano, S. G., Tarboton, D. G., Valentine, D., Zaslavsky, I., Whitenack, T. (2016). Observations Data Model 2: A community information model for spatially discrete Earth observations, Environmental Modelling & Software, 79, 55-74, http://dx.doi.org/10.1016/j.envsoft.2016.01.010

If you are interested in learning more about how ODM2 supports different use cases, have a look at our recent paper in the Data Science Journal.

Hsu, L., Mayorga, E., Horsburgh, J. S., Carter, M. R., Lehnert, K. A., Brantley, S. L. (2017), Enhancing Interoperability and Capabilities of Earth Science Data using the Observations Data Model 2 (ODM2), Data Science Journal, 16(4), 1-16, http://dx.doi.org/10.5334/dsj-2017-004.

Getting Started with ODM2

SQL scripts for generating blank ODM2 databases can be found at the following locations:

View Documentation of ODM2 Concepts

For more information on ODM2 concepts, examples, best practices, the ODM2 software ecosystem, etc., visit the Documentation page on the wiki.

View Diagrams and Documentation of the ODM2 Schema

Schema diagrams for the current version of the ODM2 schema are at:

Data Use Cases

The following data use cases are available. We have focused on designing ODM2 to support these data use cases. Available code and documentation show how these data use cases were mapped to the ODM2.

  • Little Bear River - Hydrologic time series and water quality samples from an ODM 1.1.1 database. Implements an ODM2 database in Microsoft SQL Server.
  • PRISM-XAN - Water quality depth profiles and samples from Puget Sound. Implements an ODM2 database in PostgreSQL.

Our Goal with ODM2

We are working to develop a community information model to extend interoperability of spatially discrete, feature based earth observations derived from sensors and samples and improve the capture, sharing, and archival of these data. This information model, called ODM2, is being designed from a general perspective, with extensibility for achieving interoperability across multiple disciplines and systems that support publication of earth observations.

ODM2 Schematic

Credits

This work was supported by National Science Foundation Grant EAR-1224638. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

ODM2 draws heavily form our prior work with the CUAHSI Hydrologic information system and ODM 1.1.1 (Horsburgh et al., 2008; Horsburgh and Tarboton, 2008), our experiences workin on the Critical Zone Observatory Integrated Data Management System (CZOData), and our experiences with the EarthChem systems (e.g., Lehnert et al., 2007; Lehnert et al., 2009). It also extensively uses concepts from the Open Geospatial Consortium's Observations & Measurements standard (Cox, 2007a; Cox, 2007b; Cox, 2011a; Cox, 2011b; ISO, 2011).

References

See a full list of ODM2 related references

Cox, S.J.D. (2007a). Observations and Measurements - Part 1 - Observation schema, OGC Implementation Specification, OGC 07-022r1. 73 + xi. http://portal.opengeospatial.org/files/22466.

Cox, S.J.D. (2007b). Observations and Measurements – Part 2 - Sampling Features, OGC Implementation Specification, OGC 07-002r3. 36 + ix. http://portal.opengeospatial.org/files/22467.

Cox, S.J.D. (2011a). Geographic Information - Observations and Measurements, OGC Abstract Specification Topic 20 (same as ISO 19156:2011), OGC 10-004r3. 54. http://dx.doi.org/10.13140/2.1.1142.3042.

Cox, S.J.D. (2011b). Observations and Measurements - XML Implementation, OGC Implementation Standard, OGC 10-025r1. 66 + x. http://portal.opengeospatial.org/files/41510 (accessed September 16, 2014).

Horsburgh, J.S., D.G. Tarboton, D.R. Maidment, and I. Zaslavsky (2008). A relational model for environmental and water resources data, Water Resources Research, 44, W05406, http://dx.doi.org/10.1029/2007WR006392.

Horsburgh, J.S., D.G. Tarboton (2008). CUAHSI Community Observations Data Model (ODM) Version 1.1.1 Design Specifications, CUAHSI Open Source Software Tools, http://www.codeplex.com/Download?ProjectName=HydroServer&DownloadId=349176.

ISO 19156:2011 - Geographic information -- Observations and Measurements, International Standard (2011), International Organization for Standardization, Geneva. http://dx.doi.org/10.13140/2.1.1142.3042.

Lehnert, K.A., Walker, D., Vinay, S., Djapic, B., Ash, J., Falk, B. (2007). Community-Based Development of Standards for Geochemical and Geochronological Data, Eos Trans. AGU, 88(52), Fall Meet. Suppl., Abstract IN52A-09.

Lehnert, K.A., Walker, D., Block, K.A., Ash, J.M., Chan, C. (2009). EarthChem: Next developments to meet new demands, American Geophysical Union, Fall Meeting 2009, Abstract #V12C-01.

cz-manager's People

Contributors

lsetiawan avatar miguelcleon avatar ocefpaf avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cz-manager's Issues

error saving sampling features

I tried to save a feature as a polygon to see what would happen it seems that GEOS may not function with polygon's in Django. In any case after I tried to save a sampling feature as a polygon. I got errors, then I tried to change the feature back to a point and I couldn't get the sampling feature to save again as a point. I ended up just mannually changing it back to a point in SQL.

Docker file/image

I'm opening this issue just to track progress on creating a Docker file/image (I don't know what's the difference!), as discussed in PR #96. To reiterate what was discussed there, it looks like work on conda packaging ODM2 Admin (previously discussed off and on in issue #11; see also PR #102) will stop, in favor of a Docker solution.

Pinging @lsetiawan since it looked he was going to take some lead on this, with input/help from @ocefpaf. Don probably won't have much time this week to advance this, though.

Please chime in if I've misstated something.

Add a duplicate_dataloggerfile_event in forms.py

To make it easier to add multiple copies of data logger files with the same format add a copy event like duplicate_Dataloggerfilecolumns_event. However in this case we want to copy the Dataloggerfilecolumns and the dataloggerfile

map enhancements

@lsetiawan @emiliom

I have two improvements to maps I'd like to make.

  1. change the way time series are listed so that related time series results are viewed as a continuous time series instead of two separate listings. I think this should be based on two results having a related results record of type continues.
  2. We already can filter the map by datasets. I'd like to add URL variables so links can be shared with the map already filtered.

Add setting for sampling features to exclude from mapping

There is concern about loading a map with thousands of sampling features. This maybe too slow. To alleviate that we should add a MAP_CONFIG setting in the settings files to specify sampling feature types to exclude from the map.

Packaging

@miguelcleon I am unfamiliar with packages that use Djando so I don't really know where to start. Usually the boilerplate is to create a setup.py that installs the module, but I believe there are more pieces here that needs to be taken care of. Is that assessment correct?

xref: ODM2/conda-recipes-ODM2#27

include example ODM2 database

@ocefpaf suggested including a database in the github repository to make it easier to setup ODM2 Admin perhaps we could use the CZIMEA one? I was thinking about Jeff Hoursbough’s little bear river one but it is MSSQL so it won’t work for this. Maybe you want to make an altered copy of the CZIMEA database to include? What do you guys think @emiliom @lsetiawan ? I could try to make a really stripped down version of the LCZO one but it seems like that maybe a lot of work.

Use ODM2REST for mapping interface?

I think once ODM2REST is released, ODM2-Admin should use ODM2REST for it's mapping interface backend? This will allow practical, production testing for ODM2REST and enhance the mapping capability. What do you think @miguelcleon?

Misspellings on models.py cause errors

@miguelcleon I noticed that you have a couple misspellings in models.py which lead to errors when trying to retrieve values from the ODM2 database. There are instances that you have misspelled identifier as identifer I fixed this in my code, but now ODM2Admin is messed up when run with your database in few areas such as citations. Thanks

cc: @emiliom

Django ORM Srcipts broken in order to pass pep8

These files have been broken in order to pass pep8/travis:

https://github.com/miguelcleon/ODM2-Admin/blob/master/soilsIngestionExample/ImportMStoneData.py

https://github.com/miguelcleon/ODM2-Admin/blob/master/ODM2CZOData/NewProcessinglevelForTimeSeries.py

They need this at the start:

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "templatesAndSettings.settings") 

before other django imports, @ocefpaf if we add #noqa to these lines will that work?

Data entry via ODM2-Admin, or to populate ODM2 DB used by ODM2-Admin

@miguelcleon, I forget what mechanism you have developed to load data (and metadata) into the ODM2 database behind ODM2-Admin. I know you preserve sensor log files, so I assume you have a log-file loader for a set of pre-defined sensors. Besides that, does ODM2-Admin have the ability to do manual, form-drive data entry? If so, is it being used heavily? Or do you use your own Excel (or csv) templates as intermediates between miscellaneous data spreadsheets and ODM2? Thanks!

children of field areas not appearing in map

http://dev-odm2admin.cuahsi.org/LCZO/mapdata.html

I’m not certain why my field areas aren’t showing children. I thought maybe it was because their children didn’t have IGSNs, not all of the relationships between field areas and excavations are in the database (yet). So I used field area because they represent a catena over which 9 soil pits were dug.

COOX-3 should has one child with an IGSN at the moment and 3 without. This is at about lat 18.3100, lon -65.8185

Don, I added a ‘Is part of’ relationship between COOX-3 (samping feature id=659) and COOX-3-P55 Ridge (sampling feature id=405)

405 is part of 659

Enhancing the management and exposure of persistent universal identifiers

Hi @miguelcleon and @emiliom,

I am currently, enhancing ODM2Admin by displaying IGSN in the Sampling Features change list, and allow one to enter IGSN when creating a Sampling Feature. I have also added a feature to display ORCID for a Person and their affiliation. Additionally, one can enter a person's ORCID within the person add form.

I improved the UI a bit for listing editable item types in addition to those enhancements.

You can follow my changes here: link

Here are some screenshots if you're interested: link

Thanks.

Pre-release

@miguelcleon Can we have some sort of pre-release for me to test against when working on the Dockerfile?

docker file improvements

@lsetiawan from issue #123
The next round of improvements are to allow user to

  1. use their own settings file
  2. use uwsgi to run as production
  3. separating the database from the application

I was thinking to use the model like docker-erddap, where user attach volume within their host machine to specify where the settings files are, and we have some sort of template where we can let user to download and use... What do you think @ocefpaf ?

Map loading slowly

@lsetiawan the map view is now loading much more slowly then before. We should add a loading indicator so the user knows that the map is still loading the sampling features. Can something be done to make it load more quickly again?

Bug when updating Django-ajax-selects from 1.4.3 to version 1.5.2

@lsetiawan I found one of potentially two problems from the Conda packaging @ocefpaf has been working on. The problem appears to be related to the read only group. The bug only appears after upgrading Django-ajax-selects to version 1.5.2 (the servers are using 1.4.3).

NoReverseMatch at /admin/odm2admin/variables/1/change/
Reverse for 'odm2admin_cvvariabletype_add' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: []

\odm2admin\readonlyadmin.py in get_form
13.        form = super(ReadOnlyAdmin, self).get_form(request, obj, **kwargs)

Can you try to reproduce this?

GeoDjango

Consider adding GeoDjango API for PostGIS data.

Error opening form for method

I've loaded a backup of my EarthChem ODM2 database into the Postgres running in the ODM2-Admin docker container. To do that I did a pg_dump (to a directory, with -E for SQL_ASCII encoding and -O), and put the dump in a directory attached to my docker container when I run it. I'm not genereating all the ODM2 tables in my EC database, just the ones that are used or I think we'll need.

I dropped the ODM2 schema from the container postgres db, and then used pg_restore to bring in the backup. No errors reported. I have to configure my container to map container port 5432 to a different port (I'm using 5437) to connect to the db using pgAdmin on my host machine. (there's some other tweaks necessary, but that's a differnent topic) All the tables appear to be OK.

I can connect to ODM2-Admin with browser on my host machine (e.g. http://127.0.0.1:8010/odm2admin/samplingfeatures/) but for some entities (action, method) when I click on a link to drill into the entity I get an error with a heading like "TypeError at /odm2admin/methods/22/change/
'NoneType' object has no attribute 'getitem' ".

Any ideas what might be causing this? I'm attaching and example of the whole error report (ends up being several pages): TypeError-at_odm2admin_methods_22_change_.pdf

issues using docker container

@lsetiawan
I'm back and trying to run the docker container. It seems to work except the django tables are not in the odm2admindb.backup database. Either a migration needs to be ran on the backup database so that the django tables exist or the migrate command for manage.py needs to be ran I think in startup.sh

I was trying to do this on my own branch but docker can't seem to find my repo, maybe because I added -b miguelworking to point to the branch in the dockerfile git clone command.
I also tried to build the image from the local Dockerfile but it is failing with the below, not sure why.

Step 1/18 : FROM miniconda:latest
repository miniconda not found: does not exist or no pull access

Enforce a linter for pep8 rules

@miguelcleon do you want to enforce a linter for pep8 rules here? I usually do that in my code and I even add it as part of the continous integration tests.

(See https://github.com/ocefpaf/ODM2-Admin/blob/Travis-CI/.travis.yml#L28 for example.)

I already started to re-organizing the code in some places but the amount of work required to bring ODM2-Admin to compliance is quite a lot. Before taking this path I want to if there is an interest from the authors to continue.

Pros:

  • cleaner code;
  • lower the entry barrier for new contributors;
  • higher maintainability.

Cons:

  • too much work;
  • authors are a closed group who do not care about pep8.

The commit ocefpaf@4fafdeb has ~20% of the job done and it is already unmanageable (hard to review without tests). That is the main reason I am raising this issue.

Comments & questions on Luquillo CZO database used by ODM2-Admin

@miguelcleon, I'm dumping a long list of my comments and questions in this single issue. I had already typed this in a local markdown document, so it's easiest for me to just dump it all in one place. We can split it off later. I'm pinging @lsetiawan (Don), so he's in the discussion.

When trying to restore the database backup file (using PgAdmin) on my laptop (Ubuntu 14.04, Postgresql 9.3, PostGIS 2.1), the database was installed with a couple of severe gaps. The problem boiled down to PostGIS failing to install, which in turn prevented the samplingfeatures table from installing.

PostGIS installation

  • To get around the PostGIS installation failure I first created a blank database using a template database that already has my standard PostGIS installation. Then I restored (using PgAdmin) from your backup file into that database. The CREATE EXTENSION PostGIS command in your backup probably checks if the extension exists already, and if it does, it doesn't try to install it.
  • Don and Miguel were able to install the dabase from Miguel's DB backup. In Don's case, PostGIS 2.2 was installed; in Miguel's case, 2.1.x
  • Unnecessary, postgis-related extensions are installed: topology and tiger extensions are in the backup database. They are not needed at all. I think they pollute the database instance and add to the complexity of installations.
  • ME: think about incorporating my PostGIS / samplingfeatures "additional" commands (index, dim constrain, etc)

database structure

  • A custom schema has been created, odm2extra. It holds 3 custom tables: Measurementresultvaluefile, featureactionsNames and processdataloggerfile. Note that the names of two of these are defined to be case-sensitive, which is inconsistent with general ODM2 PostgreSQL conventions
  • The custom table Measurementresultvaluefile is present in two schemas, odm2 and odm2extra. That doesn't seem right.
  • A custom function was created in the odm2 schema: MeasurementResultValsToResultsCountvalue
  • Django user and authorization management tables
    • A number of tables have been added for this purpose. Per Don, they're probably all created by Django. The tables are: auth_group, auth_group_permissions, auth_permission, auth_user, auth_user_groups, auth_user_user_permissions, django_admin_log, django_content_type, django_migrations, django_session
    • Ideally they should be placed in a separate schema, not under the default odm2 schema. I think it'd be best to have a dedicate schema such as odm2admin_auth or django_auth. But they could also go under odm2extra

table usage (records, implicit conventions, issues, etc)

samplingfeatures and related Sampling Features entities

  • Problems with the geospatial encoding (featuregeometry) for many of the records:
    • No lat-lon: 136 records have (0,0) for the lat-lon encoding
    • (Update: this has been fixed) Swapped lat-lon: 2 records have lat-lon values that are obviously wrong and simply swapped
    • query to select these records:
      sql select st_astext(featuregeometry), * from odm2.samplingfeatures where st_x(featuregeometry) > -1 order by samplingfeatureid
  • Some sampling features seem to be generic classifiers rather than actual features with geospatial information; eg, "Tabonuco forest"
  • Many feature relationships are used (relatedfeatures). It'll be helpful to learn more about what these are. An interesting example is from the relationship of samplingfeatureid 768, 769, 656.
  • sites and specimens are not used (no records)

result types

  • based on record counts in <resulttype>results tables, only two result types are being used: measurement and profile. But a count of results.resulttypecv entries yields two different result types: "Measurement" (12,192) and "Time series coverage" (199).
  • <resulttype>results table record counts: measurementresult (205) and profileresult (12,113). No records in timeseriesresult
  • These inconsistencies should create problems, in principle

actions

  • actions are probably "coarse" and aggregated, b/c there are only 29 actions records. Here's their distribution by actiontypecv:
actiontypecv count
Data retrieval 1
Equipment maintenance 1
Estimation 1
Instrument calibration 1
Observation 12
Specimen analysis 13
  • relatedactions is not used

citations

  • In citations table (has 159 records), the vast majority of entries include the authors and year in the title column; that column is intended to be limited to the title per se. But note that authorlists is also used extensively (has 278 records), so the author information may already be properly captured, and authors in titles could be removed.

CZO Sampling Feature in map

@miguelcleon I've been developing the web map to meet the needs of CZIMEA. I have implemented filter by Sampling Feature Type, but I notice that you have a type that is not in the standard library, but rather from an externalidentifier CZO. I've been cleaning up and rearranging the code, but wasn't sure how I can put this in.. What was your thought in adding CZO Sampling feature type?

.gitignore changes to better handle jupyter notebooks, etc

In my notes from our call last Friday I wrote that I was going to make tweaks to your .gitignore file. For now, the simplest way for me to do this (instead of a PR) is to list them here and have you apply them. Sorry ...

Remove:
*.ipynb

I would remove .gitignore from the list, so that changes to it are propagated. But there are probably different schools of thought on that.

Add:
.ipynb_checkpoints

You might as well also add these two, that are generally good to have:
*~
.DS_Store

NoReverseMatch on forms with ajax

I'm getting NoReverseMatch errors on change form with ajax.

For Example:
NoReverseMatch at /admin/odm2admin/units/2/change/

Reverse for 'odm2admin_cvunitstype_add' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: []

Connect ODM2 admin in Docker Container to postgres db on host

I've got the Docker container with ODM2 Admin running, and have been able to restore a backup of my ODM2 database (EarthChem db, ETL to canonical ODM2). I need to be able to connect the ODM2 Admin tools to a database on the host before I start actually working on data, otherwise I either need to commit the container, or lose anything I've changed in the container db. There needs to be a way to configure the database connection somewhere and I can't see how to do it.

use UTF-8 character encoding

When I connect to db using pgAdmin get message:
"The database odm2_db is created to store data using the SQL_ASCII encoding. This encoding is defined for 7 bit characters only; the meaning of characters with the 8th bit set (non-ASCII characters 127-255) is not defined. Consequently, it is not possible for the server to convert the data to other encodings. "

The Earthchem data definitely has some characters that are outside of the 7 bit ascii space. Shouldn't the db use UTF_8. I think this explains why I couldn't use a text pg_dump to restore my EarthChem Db to the Docker container odm2_db-- had to use pg_restore and directory with explicit character encoding parameter.

Docker Documentation

@miguelcleon I think we need better documentation on how to use ODM2-Admin with Docker. I have created an ODM2 Organization in Docker Hub, now the repository is odm2/odm2admin.

  • Installation
  • Connect with Database
  • Loading Data

conda environment generated from requirements.txt is not completely functional

When attempting to run ODM2 Admin with a conda environment generated from requirement.txt some errors are encountered:

  1. inline forms were not rendering, I'm guessing this has to do with this import not working properly https://github.com/miguelcleon/ODM2-Admin/blob/master/odm2admin/forms.py#L2

  2. ajax lookup fields sometimes weren't working. I haven't been able to figure out why some worked and others did not.

@ocefpaf probably there is some problem with django-ajax-selects?

@lsetiawan maybe you could see if you can replicate the problems?

change folder names

change file folder names from:
ODM2testapp to ODM2CZOData
and
ODM2testsite to templatesAndSettings

Rename ODM2CZOData folder to something more generic, eg, odm2admin?

@miguelcleon, is having a base folder name ODM2CZOData necessary or even helpful at this point? It doesn't seem like it.

As we prepare to make a last push (hopefully) on conda packaging, plus improvements to the setup/configurations that @lsetiawan has made recently (see his commits on his branch since 12/20), this is a good time to think about this. It seems to me to like ODM2CZOData is a holdover that now just adds confusion. Assuming I'm right, I'd suggest renaming to odm2admin.

@ocefpaf, any thoughts on good practices for that "base" code folder?

Travis failing due to links

@ocefpaf @lsetiawan @emiliom

I've been adding sphinx documentation for the meeting next week, Travis is failing though because of links that require you to be logged in to a sandbox copy of ODM2 Admin like this link below

https://dev-odm2admin.cuahsi.org/Sandbox/odm2admin/results/

Here is a travis job that failed.

https://travis-ci.org/ODM2/ODM2-Admin/jobs/299922357

The error:

HTTPSConnectionPool(host='dev-odm2admin.cuahsi.org', port=443): Max retries exceeded with url: /Sandbox/ (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))

It actually says the certificate failed to be verified, but in any case I just want to suppress the error for now. Is there a way to suppress it?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.