The goal of dataspice is to make it easier for researchers to create basic, lightweight and concise metadata files for their datasets. These basic files can then be used to:
- make useful information available during analysis.
- create a helpful dataset README webpage.
- produce more complex metadata formats to aid dataset discovery.
Metadata fields are based on schema.org and other metadata standards.
A fully worked example can be found here and a live preview of the output here. An example of how Google sees this can be found here.
You can install the development version from GitHub with:
# install.packages("devtools")
devtools::install_github("ropenscilabs/dataspice")
create_spice()
write_spice()
build_site()
create_spice()
creates template metadata spreadsheets in a folder (by default created in thedata
folder)
The template files are:
- attributes.csv - explains each of the variables in the dataset
- biblio.csv - for spatial and temporal coverage, dataset name, keywords, etc.
- access.csv - for files and file types
- creators.csv - for data authors
The user needs to fill in the details of the 4 template files. These csv files can be directly modified, or they can be edited using some helper functions and/or a shiny app.
-
prep_attributes()
populates thefileName
andvariableName
columns of theattributes.csv
file using the header row of the data files. -
prep_access()
populates thefileName
,name
andfileFormat
columns of theaccess.csv
file from the files in the folder containing the data.
To see an example of how prep_attributes()
works, load the data files
that ship with the package:
data_files <- list.files(system.file("example-dataset/",
package = "dataspice"),
pattern = ".csv",
full.names = TRUE)
This function assumes that the metadata templates are in a folder called
metadata
within a data
folder.
attributes_path <- here::here("data", "metadata",
"attributes.csv")
Using purrr::map()
, this function can be applied over multiple files
to populate the header names
data_files %>% purrr::map(~prep_attributes(.x, attributes_path),
attributes_path = attributes_path)
The output of prep_attributes()
has the first two columns filled out:
fileName |
variableName |
description |
unitText |
---|---|---|---|
BroodTables.csv |
Stock.ID |
NA |
NA |
BroodTables.csv |
Species |
NA |
NA |
BroodTables.csv |
Stock |
NA |
NA |
BroodTables.csv |
Ocean.Region |
NA |
NA |
BroodTables.csv |
Region |
NA |
NA |
BroodTables.csv |
Sub.Region |
NA |
NA |
Each of the metadata templates can be edited interactively using a shiny app
edit_attributes()
opens a shiny app that can be used to editattributes.csv
. The shiny app displays the currentattributes
table and lets the user fill in an informative description and units (e.g. meters, hectares, etc.) for each variable.edit_access()
: opens an editable version ofaccess.csv
edit_creators()
: opens an editable version ofcreators.csv
edit_biblio()
: opens an editable version ofbiblio.csv
Remember to click on Save when finished editing.
The first few rows of the completed metadata tables in this example will look like this:
access.csv
has one row for each file
fileName | name | contentUrl | fileFormat |
---|---|---|---|
StockInfo.csv | StockInfo.csv | NA | CSV |
BroodTables.csv | BroodTables.csv | NA | CSV |
SourceInfo.csv | SourceInfo.csv | NA | CSV |
attributes.csv
has one row for each variable in each
file
fileName | variableName | description | unitText |
---|---|---|---|
BroodTables.csv | Stock.ID | Unique stock identifier | NA |
BroodTables.csv | Species | species of stock | NA |
BroodTables.csv | Stock | Stock name, generally river where stock is found | NA |
BroodTables.csv | Ocean.Region | Ocean region | NA |
BroodTables.csv | Region | Region of stock | NA |
BroodTables.csv | Sub.Region | Sub.Region of stock | NA |
biblio.csv
is one row containing descriptors including spatial and
temporal
coverage
title | description | datePublished | citation | keywords | license | funder | geographicDescription | northBoundCoord | eastBoundCoord | southBoundCoord | westBoundCoord | wktString | startDate | endDate |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Compiled annual statewide Alaskan salmon escapement counts, 1921-2017 | The number of mature salmon migrating from the marine environment to freshwater streams is defined as escapement. Escapement data are the enumeration of these migrating fish as they pass upstream, … | 2018-02-12 08:00:00 | NA | salmon, alaska, escapement | NA | NA | NA | 78 | -131 | 47 | -171 | NA | 1921-01-01 08:00:00 | 2017-01-01 08:00:00 |
creators.csv
has one row for each of the dataset
authors
id | givenName | familyName | affiliation | |
---|---|---|---|---|
NA | Jeanette | Clark | National Center for Ecological Analysis and Synthesis | [email protected] |
NA | Rich | Brenner | Alaska Department of Fish and Game | richard.brenner.alaska.gov |
write_spice()
generates a json-ld file (“linked data”) to aid in dataset discovery, creation of more extensive metadata (e.g. EML), and creating a website.
Here’s a view of the dataspice.json
file of the example data:
build_site()
generates an index.html file in the repositorydocs
folder, to create a website that shows a simple view of the dataset with the metadata and an interactive map. For example, this repository results in this website
A few existing tools & data standards to help users in specific domains:
- Darwin Core
- Ecological Metadata
Language
(EML) (&
EML
) - ISO 19115 - Geographic Information Metadata
- ISO 19139 - Geographic Info Metadata XML schema
- Minimum Information for Biological and Biomedical Investigations (MIBBI)
…And others indexed in Fairsharing.org & the RDA metadata directory.
This package was developed at rOpenSci’s 2018 unconf by (in alphabetical order):