The data-publication from inbo

Update README

Other than the migration of files and issues (#12), here are some other things we should do to make this repository accessible:

Decide on a name (keep it short)
Update README
Mention that one can watch this repo
Add CONTRIBUTING guidelines
Rename /occurrences to guidelines/occurrences/ or use guidelines and rename containing files.
Decide on LICENCE

Add preferred citation for all dataset repositories

I think it would be a good idea to add a preferred citation to our dataset repositories on GitHub, so users know how to cite them. Here's what I did for watervogels:

Preferred citation

Want to use this repository in a scholarly publication? You can cite it as:

Brosens D, Desmet P, Devos K (2014) Watervogels - Wintering waterbirds in Flanders, Belgium. https://github.com/LifeWatchINBO/watervogels-occurrences (accessed yyyy-mm-dd)

Do you think this is a good idea?
What about the format?
I've added the main contributors as authors, in alphabetical order. What do you think?

Note: once we agree on this, we could change the LICENSE for those repositories from MIT (which really is for software) to CC0. That would mean everything, including the metadata on the repository also becomes CC0 (while the data paper might be licensed CC BY). An alternative is to specify which parts (data, scripts, metadata) have which license, but that is more complicated.

Points north of Madagascar

In the gull tracking dataset on GBIF, one or more points are positioned above Madagascar (50,98 / 0,00): http://www.gbif.org/dataset/83e20573-f7dd-4852-9159-21566e1e691e. Those points are included in the download as well (I noticed it in QGIS)

These might be outliers that should have been filtered out. I'll try to find the ID of those coordinates.

Migrate sas-occurrences repository to this repository

sas-occurrences

Too many records on GBIF

The bird tracking gulls dataset shows 1.6 million records on GBIF: http://www.gbif.org/dataset/83e20573-f7dd-4852-9159-21566e1e691e, while there are only 1.1. Might have something to do with the changed occurrenceIDs.

Bird tracking gulls: Items that could change in metadata on update

Basic metadata

The dataset contains over 440,000 occurrences, recorded in 2013 by 27 GPS tags mounted on 22 Lesser Black-backed Gulls and 5 Herring Gulls breeding at the Belgian coast.

Taxonomic coverage

The dataset contains tracking data from 22 Lesser Black-Backed Gulls (Larus fuscus) and 5 Herring Gulls (Larus argentatus) breeding at the Belgian coast.

Geographic coverage

The birds breed at the Belgian coast in two colonies: in the port of Zeebrugge and in the city of Ostend. Their foraging range includes the West of Belgium, Northern France, the North Sea, and the English Channel. The Lesser Black-backed Gulls migrate south in winter, hibernating in the South of Spain, Portugal and Northern Africa.

Update coordinates of bounding box

Temporal coverage

Add new formation period (e.g. winter season 2013-2014)

Update end date

Methodology

Most birds were trapped on their nest using a walk-in cage. In 2013 and 2014 respectively 22 and 24 ground-nesting LBBG were caught in the port of Zeebrugge and respectively 5 and 8 HG on the roof of the Vismijn in Ostend. Additionally, in 2014 one ground nesting HG was caught in the port of Zeebrugge and 3 HG were caught with a small canon net when feeding on the Visserskaai in Ostend. We took biometrics of all captured gulls (bill length, bill depth, tarsus length, wing length, and body mass) and a feather sample to determine the sex. The UvA-BiTS GPS trackers were attached to the back of the gull using a harness of Teflon tape.

Project data

This bird tracking network was funded for LifeWatch by the Hercules Foundation (http://www.herculesstichting.be/in_English/), with additional contributions from the Terrestrial Ecology Unit (TEREC) at the University of Ghent.

Note: if more species or breeding locations are added, a more substantial review of the metadata is in order

Template checklist for DwC occurrence fields (full list)

legend: mandatory, recommended/useful, optional, ~~don't use~~

id

Root

Occurrence

Event

Location

Identification

Taxon

Use standard delimiter for recordedBy

Current format:

Emilie Gelaude;Seth Martens;Yves Jacobs

Standard format (notice space):

Emilie Gelaude|Seth Martens|Yves Jacobs

Format the localities dataset as a data package

If we really care about the localities as a dataset that could be used, I propose to standardize it as a data package. Advantages:

Metadata about origin
Definitions of fields
Machine readable

We can start with this creator and look at this example.

Note: data packages in a subfolder of a GitHub repository can currently not be read by the tools out there. Should we create a new repository?

Number of records mismatch with GBIF

The latest publication (March 6) has 3,638,726 records, while GBIF indicates 3,638,729 (3 more). This might be due to an old problem where GBIF records where not deleted, see: http://dev.gbif.org/issues/browse/POR-2343

Update broedvogel-atlas-occurrences standardization

Changes to apply:

Record

Populate occurrenceID
Map license instead of rights
Update accessRights.
Use dataset DOI in datasetID (see #21)
Decide on wether to use informationWithheld

Occurrence

Verify recordedBy (e.g. encoding) and identifiedBy
Add additional observers to recordedBy?
Update individualCount: x 2 because these are pairs

Event

Correct eventDate: 22 records with 1969-03-01 (km/punt onderzoek)
Verify eventDate: Some records from 2003 (km/punt onderzoek)
Update samplingProtocol (was surveys on both 5km x 5km and 1km x 1km scales and a total of 645 squares; Bird Census News 2004:1/2:35-47): now Bird Census News 2004 1/2 p.36 or loose observations
Update samplingEffort(was uncomplete 5X5 UTM survey; observation hours=29): now {"observationHours":29}

Locality

Taxon

Correct taxonRank for non-species
Add scientificNameAuthorship for Ringtaling and Kleine barmsijs
Correct vernacularName for Soepeend' and Soepgans' (has extra quote)

Production

Apply changes in production

Mint DOIs for our datasets

All our datasets have DOIs assigned by GBIF, which link to their dataset page on GBIF. We propose the DOIs as the link for our datasets: in our resource citations and usage norms.

With the release of IPT 2.2, we can now mint our own DOIs, which would then link to the resource page on our IPT. That is arguably a better representation of the source of our datasets.

Here's what needs to happen:

Register with DataCite (maybe already done for PURE)
Repurpose the GBIF DOIs for all our datasets

Template checklist for DwC occurrence fields (short list)

legend: mandatory, recommended/useful, optional

id

Root

Occurrence

Event

Location

Identification

identifiedBy
dateIdentified

Taxon

Update visherintroductie SQL

Note: Data verified for version 14.4 on 2015-10-01. Metadata has been completely verified, updated, and republished to comply with our new guidelines regarding usage norms, recourse citation, etc.

Changes to apply for occurrence core:

Changes to apply for measurement extension:

Update measurementType to use lowercase: #32
Remove modified

Questions:

Decide on waterBody + verbatimLocality: #30
There is no recordedBy, is this on purpose? See also #26
Can we add a samplingEffort? #29
Can we add more taxon levels, e.g. class etc.

Add samplingEffort

Does Tom know what sampling effort was used to catch the fish?

Add orcids for researchers associated with datasets

People with Orcid

Dimitri Brosens 0000-0002-0846-9116
Peter Desmet 0000-0002-8442-8025
Tim Adriaens 0000-0001-7268-4200
Willem Bouten 0000-0002-5250-8872

People without Orcid

Anny Anselin
Bert Van Der Krieken
Dirk Maes
Eric Stienen
Filiep T'Jollyn
Gerlinde Van Thuyne
Francisco Hernandez
Frederic Piesschaert
Gilles San Martin
Glenn Vermeersch
Hendrik Devriese
Hugo Verreycken
Jan Breine
Jan Gabriëls
Jan Stevens
Jorg Lambrechts
Koen Devos
Koen Lock
Kris Decleer
Luc Lens
Marc Herremans
Tom De Boeck
Tom Van Den Neucker

Install IPT 2.2.1

Read release notes

Test IPT

Backup data directory
Replace .war
Follow installation instructions
Verify installation is working

Production IPT

Backup data directory
Replace .war
Follow installation instructions
Verify installation is working
Choose license for each dataset (and some other tasks, see #12)
Republish each dataset

Standardize samplingProtocol

From:

Electro Fishing
unknown

To:

electrofishing
(leave blank)

Remove field sex

Is currently set to undetermined for all record. Decided to leave blank in these cases.

Not all records in bounding box for VIS inland

A bunch of occurrence records (around one lake) fall outside the bounding box defined in the metadata: http://www.gbif.org/dataset/823dc56e-f987-495c-98bf-43318719e30f Are these correct? Should we expand the bounding box?

Use DOI for datasetID

We currently populate datasetID with something like: http://dataset.inbo.be/vis-inland-occurrences. This links to the resource on our IPT. This however, is not the dataset key used in the GBIF portal and through the GBIF API, which is 823dc56e-f987-495c-98bf-43318719e30f. Should we change to this GBIF UUID?

+

Single key used everywhere

-

The UUID is a URL, but Googling it gets you the page
The page the UUID links to does not mention the IPT page, but users can download the original Darwin Core Archive from there.
We can only get the UUID once we publish, so we can only populate it at second publication.

Update watervogels-occurrences standardization

Changes to apply:

Papers to update with new order of sections

See issue #2

Remove collectionCode and catalogNumber (and use occurrenceID)

Procedure

Verify that dataset has occurrenceIDs
Verify that occurrenceIDs are harvested by GBIF (this guarantees a link between occurrenceID and triplet)
Bookmark an occurrence record page and take a screenshot
Remove collectionCode and catalogNumber from source
Republish
Wait 24h and very that records have remained the same (use bookmark and screenshot)
Do a search to verify all records no longer have a collectionCode, e.g. http://www.gbif.org/occurrence/search?DATASET_KEY=823dc56e-f987-495c-98bf-43318719e30f&COLLECTION_CODE=INBO

Datasets (check if resolved or logged elsewhere)

Standardize verbatimSRS and verbatimCoordinateSystem

From:

verbatimCoordinateSystem: Lambert 72
verbatimSRS: BD72

To:

verbatimCoordinateSystem: Belgian Lambert 72
verbatimSRS: Belgian Datum 1972

verbatimLocality and waterBody

It seems verbatimLocality is concatenated in the SQL:

Zeeploop - stroomopwaarts hoekpaal wei coniferenhaag

Just like for VIS, maybe we could provide waterBody as well and have verbatimLocality concatenated with ,(see https://github.com/LifeWatchINBO/vis-inland-occurrences/issues/4)

waterBody: Zeeploop
verbatimLocality: Zeeploop, stroomopwaarts hoekpaal wei coniferenhaag

Update SQL for bird-tracking-wmh

Note: Data verified for unpublished version on 2015-10-06. Metadata has been not yet been updated (waiting for gull data paper).

Changes to apply:

Use DOI as datasetID (once issued)

Questions:

Is the datasetName OK?
What do we choose as organismID? Ideally it should be a code that is used internationally and across projects. For gull we chose ring_code. https://github.com/LifeWatchINBO/bird-tracking-wmh-occurrences/issues/6 Candidates:
```
```
device_info_serial: 586
bird_name:          Mia
color_ring_code:    -
ring_code:          H173481
```
```
Do all the fields in bird_tracking_devices make sense in the context of wmh?
Do outlier criteria make sense for wmh?

Personnel

There is some confusion on who to include where for personnel, especially the difference between principal investigator and point of contact.

Principal investigators:
Resource contact, resource creator, point of contact:
Metadata provider:
Content providers:
Developer:
Processors: Dimitri Brosens, Peter Desmet

Provide guidelines on georeferencing

There is some good documentation in this issue.

Also, review datasets to include the proper fields.

Update vis-estuarine-occurrences SQL

Note: Data verified for version 9.2 on 2015-10-07. Metadata has been completely verified, updated, and republished to comply with our new guidelines regarding usage norms, recourse citation, etc.

Changes to apply:

Questions:

Should we add datapaper citation to bibliographicCitation? It might unnecessarily increase the size. Note: the data paper DOI is mentioned in the resource citation.

Source website appears twice on GBIF

Since we started using the IPT 2.2 and republished a dataset, the source website appears twice on GBIF:

This is already the case for:

Bird tracking gulls: http://www.gbif.org/dataset/83e20573-f7dd-4852-9159-21566e1e691e
Florabank: http://www.gbif.org/dataset/271c444f-f8d8-4986-b748-e7367755c0c1

Order of sections in data paper

The IPT .rtf output, the VASCAN datapaper, our datapapers, and regular papers all have a different order for the sections. I think we should settle for an order that we think is the most logical and use that from now on.

Here's the current order we use, based on the VASCAN paper:

Title
    Authors
    Affiliations
    Corresponding author
    Review dates
    Citation
    Resource citation
Abstract
Keywords
Data published through
Project details
    Project title
    Personnel
    Funding
    Study area description (default in IPT = short, while in paper this is long)
    Design description
    Purpose
    Additional information (default = extra data, such as measurements)
Taxonomic coverage
    *No title: description of taxonomic coverage*
    Taxonomic ranks
    Common names
Spatial coverage
    *No title: description of spatial coverage*
    Bounding box for covered area
Temporal coverage
Sampling methods
    Study extent description (default in IPT = longer, while in paper this is short)
    Sampling description
    Quality control description
    Method step description
Dataset
   *No title: Description of dataset*
   *No title: Norms*
   *No title: List with characteristics*
   Suggestion citation for the latest version of the dataset
External datasets (we don't use this)
Acknowlegdements
References

Verify broedvogels dataset

Data

Review data
Report data issues
Resolve data issues

Metadata

Republish

Republish

Invasive species datasets: title

@DimEvil, here's my suggestion for the titles of the invasive datasets: Grouping name - Vernacular name (scientific name) in Flanders, Belgium. At first I thought of using IAS as a grouping name, but a Google search on this doesn't reveal much. I think Invasive species is more informative.

Invasive species - Ruddy duck (Oxyura jamaicensis) in Flanders, Belgium invasive-duck-occurrences
Invasive species - American bullfrog (Lithobates catesbeiana) in Flanders, Belgium invasive-bullfrog-occurrences
Invasive species - Chinese mitten crab (Eriocheir sinensis) in Flanders, Belgium invasive-crab-occurrences
Invasive species - Muntjac (Muntiacus reevesi) in Flanders, Belgium invasive-muntjac-occurrences
Invasive species - Raccoon (Procyon lotor) in Flanders, Belgium invasive-raccoon-occurrences
Invasive species - Summering geese (Anserinae) in Flanders, Belgium invasive-geese-occurrences

FYI, the original format was Invasive Oxyura jamaicensis - Ruddy duck occurrences in Flanders.

Is this OK for you?

Gull data paper

Before submitting

After review

Update text

After acceptance

After DOI is known

Add paper DOI to metadata.md (2 places)
Add paper DOI to IPT (2 places)
Add paper reference to metadata.md (as first reference)
Add paper reference to IPT (as first reference)
Add paper reference to README
Republish dataset
Update dataset on VODP (also citation)

Incorrect bounding box for visfauna Yser

All occurrence records seem to fall outside the bounding box defined in the metadata: http://www.gbif.org/dataset/0ac24b3c-feb9-48d5-bf02-da4a103f024e. One of the two is incorrect, probably the metadata. This Yser dataset has a correct bounding box: best just copy that one?

Coordinates for 1km?

Do we have coordinates for the kilometerhokken. If not, I need to update some location fields.

Update vis-inland-occurrences SQL

Note: Data verified for version 9.2 on 2015-10-07. Metadata has been completely verified, updated, and republished to comply with our new guidelines regarding usage norms, recourse citation, etc.

Changes to apply to the data:

Questions:

Should we add datapaper citation to bibliographicCitation? It might unnecessarily increase the size. Note: the data paper DOI is mentioned in the resource citation.

measurementType lowercase

In the extension, please use:

weight
length

for measurementType.

DOI for VIS datasets

The two VIS datasets currently do not have a DOI issued by GBIF. One option - just like for Florabank - would be to give it the DOI of the datapaper (http://doi.org/10.3897/zookeys.475.8556). The difference would be that the two datasets would then share the same data paper DOI.

@DimEvil, thoughts? If OK, I'll contact GBIF.

Add coordinates for 4 grids

These grids are missing coordinates:

31UFS8674   90  km hok
31UFS88B    1   atlas hok
31UFT2107   4   km hok
31UFT49B    52  atlas hok

Publish inactive areas

Suggestion by @DimEvil:

Now we publish only the active areas! What about publishing the non active areas as an historical dataset. We should be able to georeference most of the localities... (UTM5). It's about 31000 records.

Use concatenation instead of hashes for occurrenceIDs

Suggestion by @timrobertson100: something like 891-20150301112532 (device info serial and date time) has the advantage that it is recognizable and sortable, which a hash is not.

Update metadata for bird-tracking-wmh

This issue groups metadata issues of the old repository and includes some new ones.

Changes to apply:

Update the metadata for this dataset based on the data paper for the gulls, since a lot of changes also apply to this dataset.
Complete sections of the metadata that are specific to the WMH dataset
Create a CartoDB visualization and add to abstract and geospatial coverage
Elements to update with new data: Basic metadata, taxonomic coverage (number of individuals), temporal coverage and potentially methodology and project information

Questions:

Is the title OK? Is it applicable to future data?

Set language to English

In data, use en

Update license and norms for all datasets

See this tutorial. Steps:

Add usage norms in the metadata document
Add usage norms in additional information in the IPT
Add accessRights to data mapping and populate with: http://www.inbo.be/en/norms-for-data-use
Add license to data mapping and populate with: http://creativecommons.org/publicdomain/zero/1.0/
Remove rights in the data mapping.

Remove the dataset labels for which:

This has been done in the metadata AND
This has been done in the data (or an issue is logged to do so, e.g. #57)

Add default resource citation for all datasets

IPT is now providing an automatic citation, which we will use from now on. Don't forget to add the DOI as the Resource citation identifier. The result should be this:

Vermeersch G, Anselin A, Devos K, Herremans M, Stevens J, Gabriëls J, Van Der Krieken B, Brosens D, Desmet P (2014): Broedvogels - Atlas of the breeding birds in Flanders 2000-2002. v1.5. Research Institute for Nature and Forest (INBO). Dataset/Occurrence. https://doi.org/10.15468/sccg5a

Notes:

In the metadata.md, we copy this citation as the resource citation, but drop the version number.
We can turn it off the automatic citation for datasets with a data paper, to include data paper information.
The authors of the citation = the resource creators = the authors on the Markdown metadata. We no longer populate personnel or associated parties, except if we want to highlight organizations.
The DOI currently links to GBIF, but we can update that to the IPT page in the future.

Example for a resource citation with data paper:

Breine J, Verreycken H, De Boeck T, Brosens D, Desmet P (2013): VIS - Fishes in estuarine waters in Flanders, Belgium. Research Institute for Nature and Forest (INBO). Dataset/Occurrence. https://doi.org/10.15468/estwpt Data paper: https://doi.org/10.3897/zookeys.475.8556

Install IPT 2.1

Install IPT 2.1 on:

Development
Testing
Production

Bugs and issues to verify by @DimEvil and @peterdesmet

All datasets should have a globally unique occurrenceID
Remove collectionCode
Coordinates bug fixed?

eventDate misinterpreted by GBIF

Note: this issue was first reported on July 25, 2014

I noticed a massive drop in the temporal precision of our data, that can only be contributed to a change in this (big) dataset.

We indeed started to use ISO 8601 date ranges about a year ago and it seems that GBIF has problems interpreting those: 2004-05-01/2004-05-31 is interpreted as having no date, while it should be interpreted as 2004-05 with a precision up to a month. See this record for an example.

Issue recorded for GBIF at http://dev.gbif.org/issues/browse/POR-2339

Not all records in bounding box for trekvis

Some occurrence records fall out the bounding box defined in the metadata: http://www.gbif.org/dataset/b2d0f29e-4614-4001-93c8-f651878a86d2

inbo / data-publication Goto Github PK

data-publication's People

Contributors

Stargazers

Watchers

Forkers

data-publication's Issues

Preferred citation

sas-occurrences

Basic metadata

Taxonomic coverage

Geographic coverage

Temporal coverage

Methodology

Project data

Root

Occurrence

Event

Location

Identification

Taxon

Record

Occurrence

Event

Locality

Taxon

Production

Root

Occurrence

Event

Location

Identification

Taxon

People with Orcid

People without Orcid

Test IPT

Production IPT

+

-

Procedure

Datasets (check if resolved or logged elsewhere)

Data

Metadata

Republish

Before submitting

After review

After acceptance

After DOI is known

Recommend Projects

Recommend Topics

Recommend Org

Jobs