r-geoflow / geoflow

R engine to orchestrate and run (meta)data workflows

geoflow

R engine to orchestrate and run (meta)data workflows

geoflow provides an engine to facilitate the orchestration and execution of metadata-driven data management workflows, in compliance with FAIR (Findable, Accessible, Interoperable and Reusable) data management principles. By means of a pivot metadata model, relying on the DublinCore standard, a unique source of metadata can be used to operate multiple and inter-connected data management actions. Users can also customise their own workflows by creating specific actions but the library comes with a set of native actions that have been identified as key steps most data managers, in particular actions oriented to the publication on the web of metadata and data resources to provide standard discovery and access services.

At first, default actions of the library were meant to focus on providing turn-key actions for geospatial (meta)data:

• by creating manage geospatial (meta)data complying with ISO/TC211 and OGC geographic information standards (eg 19115/19119/19110/19139) and related best practices (eg. INSPIRE); and
• by facilitating extraction, reading and publishing of standard geospatial (meta)data within widely used software that compound a Spatial Data Infrastructure (SDI), including spatial databases (eg. 'PostGIS'), metadata catalogues (eg. 'GeoNetwork', CSW servers), data servers (eg. GeoServer).

The library was then extended to actions for other domains:

• biodiversity (meta)data standard management including handling of EML metadata, and their management with DataOne servers,
• in situ sensors, remote sensing and model outputs (meta)data standard management by handling part of CF conventions, 'NetCDF' data format and OPeNDAP access protocol, and their management with Thredds servers,
• generic / domain agnostic (meta)data standard managers (Dublin Core, DataCite), to facilitate the publication of data within (meta)data repositories such as Zenodo or DataVerse.

The execution of several actions will then allow to cross-reference (meta)data resources in each action performed, offering a way to bind resources between each other (eg. reference 'Zenodo' DOIs in 'GeoNetwork'/'Geoserver' metadata, or vice versa reference 'Geonetwork'/Geoserver' links in 'Zenodo' or EML metadata). The use of standardized configuration files (JSON format) allows fully reproducible workflows to facilitate the work of data and information managers.

Please check the online documentation for more details! (documentation in preparation)

For questions about using or contributing to geoflow, you can ask them in the discussions panel: https://github.com/r-geoflow/geoflow/discussions

Sponsors

Many thanks to the following organizations that have provided fundings for strenghtening the geoflow package:

The following projects have contributed to strenghten geoflow:

• Blue-Cloud Blue-Cloud has received funding from the European Union's Horizon programme call BG-07-2019-2020, topic: [A] 2019 - Blue Cloud services, Grant Agreement No.862409.
• CCSAFE
• G2OI project, cofinanced by the European Union, the Reunion region, and the French Republic.

Sponsoring

For geoflow sponsoring/funding new developments, enhancements, support requests, please contact me by e-mail

Citation

We thank in advance people that use geoflow for citing it in their work / publication(s). For this, please use the citation provided at this link

geoflow's Issues

[action][geometa-create-iso-19115] missiong condition to add DOI as meta identifier

Add action's options description and helper to get it

Enrich DOI CSV output table with Zenodo deposit action

The output CSV will contain:

• Identifier: entity identifier
• Status: simple status handling the Zenodo status draft or published
• DOI_for_allversions: generic / top level DOI (named "conceptdoi" by Zenodo).
• DOI_for_version: record version-specific DOI

[action][geometa-create-iso-19115] manage date types

[action][geometa-create-iso-19115] - add option to add geometries as bounding polygon

Enable INSPIRE option for geonapi/ows4R metadata publication

At that time it is enabled only on the geometa action. INSPIRE is tested on XML files produced locally (and eventually uploaded with actions like Zenodo), but it is not tested on the XML pushed on geonapi/ows4R. These 2 packages may need enhancements to take into account INSPIRE geometa option.

• ows4R action - eblondel/ows4R#28
• geonapi action - eblondel/geonapi#7

New handler for contacts - LDAP

Error when handling relation titles (citations) that include colon and/or comma symbols (often with co-authoring)

Hi Emmanuel,
Geoflow cannot publish more than one relation (http type).
Many thanks,

wilfried

Need sanitize_date function

Keep spatial coverage if not a bbox to enrich geometa with bounding polygons

The spatial coverage may be a simple bbox wkt but it could be also a more complex spatial coverage (eg. simplified polygons, simplified lines) less reductor than a bounding box. At now geoflow reduces the WKT into a bounding box, which results in a loss of meta information. Some logic should be added to keep such more detailed spatial coverage. In geometa action it will result in setting bounding polygons, in addition to the geographic bounding box.

Zenodo / Export modified workflow config

Support for software argument handler

In some software, such as for DB connectors (#30), we need to be able to specify a handler function for a given argument.

For example, in the case of DBI, the main handler is DBI::dbConnect, but the first argument drv requires an instance of driver, that can be instantiated using DBI::dbDriver(<drivername>). The software argument definition could be extended with such handler, here in that case DBI::dbDriver.

The software function getHandlerInstance should then be able to evaluate such argument handler.

Set-up R markdown to manage master documentation

Zenodo / Add support for keywords (free text) and subjects

New software/action to publish (meta)data on Dataverse

Useful links:

• https://datacite.org/
• https://dataverse.org/
• http://guides.dataverse.org/en/latest/api/sword.html
• R package:
  • CRAN: https://cran.r-project.org/package=dataverse
  • Github: https://github.com/iqss/dataverse-client-r
  • Github: https://github.com/eblondel/atom4R

Tasks:

• add atom4R as embedded geoflow software
• add atom4R dataverse oriented action
  • manage metadata
  • support for files upload
  • support for publication

Temp directory used to set data should be deleted after each workflow job

[action][geometa-create-iso-19110] enrich listed values with registers

Zenodo / Export record as Zenodo metadata formats

After publication, export all Zenodo supported format as metadata files for the record considered.

Relations error (ignored with geosapi action is used)

Hi Emmanuel,

Relations (at least http links) are ignored when geosapi (WMS/WFS) is used.
Thanks in advance.

Cheers

Support multiple entity types

By default, a single type will be specified as generic.
For Zenodo action, several types will be managed for:

• upload: zenodoUploadType. If missing, by default, the generic type will be used
• publication: zenodoPublicationType, specified if available and if zenodo upload type is set to publication
• image: zenodoImageType, specified if available and if zenodo upload type is set to image

Add entity/contact database handler

Add action for generating feature catalogue (ISO 19110)

The action should:

• interpret entity data component to see if they are attributes and variables declared. When attributes are declared, the list of unique values will be added to the feature attribute description. If no attributes are listed, all attributes of the source features will be set by default (raw feature catalogue description).
• produce an ISO 19110 feature catalogue ISO 19139 xml object
• add FC scopes for geoflow (uploadType) and OpenFairViewer (query strategy among ogc_filters, ogc_dimensions - not yet supported in geoflow -, or ogc_viewparams) required for OFV feature catalogue handling

In addition:

• action geonapi-publish-iso-19139 should detect the produced FC and publish it
• action ows4R-publish-iso-19139 should detect the produced FC and publish it

Excluded from this task:

• handling of vocabularies/codelists to enrich feature attribute listed values with labels/definitions

Add global option to skip file download

When files do not changes, either to update record metadata or to trigger the final record publication, in particular in Zenodo, we may want to skip the file download. This can be done by adding a global option skipFileDownload.

Set-up stack of Integration tests

R functions standard documentation with royxgen2

[action][geometa-create-iso-19115] Additional owners are ignored

Hi Emmanuel,

Some contacts (owner role for example) are ignored when metadata is published.

Thanks!

wilfried

Mapping geoflow and Zenodo data types

is it possible to modify the line of the action in geoflow_action_zen4R_deposit_record.R where the data type is hard coded since the geoflow_entity class already assign a default value "dataset" in the line 8 ?
Something like
zenodo_metadata$setUploadType(entity$type)
This would enable to either send directly one of the values expected by Zenodo API or to let it empty and get "dataset" as a default value.

[action][geometa-create-iso-19115] add spatial representation from data features

When entity data features are available, a spatial representation will be added. For now bound to vector spatial representation.

Capacity to configure SQL layers, either directly (data sql) or through sql file

The configuration of SQL layers could be done in two ways:

• as SQL file path / URL put to a datasource. In that way geoflow will try to read the SQL file to pick up the SQL query (then set using geoflow_data$setSql()), or
• directly as sql data property in the Data column

Some code will be experimented to read the SQL query, if a Db driver is set-up in the configuration. Reading the SQL query will enable dynamic metadata retrieval, as it is already done for shapefile (uploadType = 'shp')

[action][geometa-create-iso-19110] Need control in case of NA attribute value

Support multiple data sources

To manage multiple sources, the geoflow data component requires a refactoring. From the table viewpoint, the column "Data" will be refactored:

• source: now the same logic as other columns can be applied. For a given file, we will specify the syntax filename@fileurl or filename@filepath. Example: sample.pdf@http://someurl.org/sample.pdf
• sourceName: this property is removed. The source name is now the filename specified in source
• upload: this remains. Until now it was mandatory to specify it, now if not specified, upload will be set to TRUE by default. upload = TRUE means data will be uploaded to the web-tool considered. If set to false, the data files in source will not be uploaded. This is valid for Zenodo or Geoserver
• type: Given the fact it deals with upload task, it is now renamed to uploadType for more consistency. Until now it was mandatory to specify it, now if not specified, the uploadType will be set to 'other'.
• identifier: this is specific to the production of OGC web layers, this will be renamed layername

Example of a Data element for a Zenodo file upload:

source:sample1.pdf@http://www.africau.edu/images/default/sample.pdf,sample2.pdf@https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf;
uploadType:other
upload:true

Example of a Data element for a Geoserver shapefile upload:

source:shapefile1.zip@D://sandbox-geoflow/shapefile1.zip;
uploadType:shp;
upload:true;
layername:layer1

[action][geometa-create-iso-19115] Handling of topic categories as subject

"Description" column of geoflow entities: issues with "abstract:" values management

I noticed that "abstract:..." values are not anymore managed properly. Sometimes the content is cut, sometimes "abstract:" prefix is not removed.
Other issue: even if II use a ";" to close the "abstract:" section, the fist line break ("\n") within the abstract section is interpreted as the end of the section.

Upload of shapefile in geosapi action not working with URL source

[action][geometa-create-iso-19115] add INSPIRE data quality reports with INSPIRE option

New handler for contacts - ORCID

Support cleaning of Zenodo draft deposits by query or list of DOIs or communities

Cleaning should be able on a sublist of deposits, and not only all:

• by query: supported by a Zenodo API ElasticSearch query
• by doi: one or more prereserved DOIs for which we want to clean / delete the depositions
• by community: one or more communities for which we want to clean / delete the depositions

Add options to zen4R action to disable update of metadata/files

2 options:

• update_metadata: to update zenodo record metadata properties. Default is TRUE
• update_files: to update zenodo record files. Default is TRUE

Entity language set in entity table (gsheet, csv, etc) is ignored

Hi,

Metadata language seems to be ignored during metadata creation with geoflow.

Cheers.

Zenodo: allow to upload multiple files separately or as Zip

new data options (to use in Data entity column):

• uploadZip: TRUE/FALSE; Default is FALSE. If TRUE, data source files will be zipped, and the zip will be uploaded in Zenodo action.
• uploadZipOnly: TRUE/FALSE; Default is FALSE. Ignored if uploadZip is FALSE. If TRUE, only the zip file will be kept and upload in Zenodo action.

Need to set a safe line separator for tabular entity/contact handler

For the time-being ';\n' is used, but not enough generic as ';' or '\n' are likely to be used as content.
A default line separator has to be set, and should be modifiable as option for user preferences.

Manage DBI software connector

Defaut DB connectors should be made available as embedded software clients/connectors. For now this targets DBI. To start, imports for RPostgreSQL and RSQLite will be added. Others could be added if needed.

Add DOI to ISO 19115 when DOI is available

Use of metadata identifier in data identification/ citation, according to https://geo-ide.noaa.gov/wiki/index.php?title=DOI_Minting_Procedure#Third.2C_Include_the_DOI_and_citation_text_in_the_ISO_Metadata_Record

Support registers handling

The principe is to let users configure registers. Later geoflow may embedd default common registers, eg country codelists, etc. For now no register included in geoflow, but the possibility to configure registers in the configuration with registers.

[action][geometa-create-iso-19115] add spatial representation type

Different handling of gsheet entities depending on 'readr' availability

Depending on readr availability in R session, gsheet::gsheet2tbl will not return the same output. Indeed as stated in the documentation: If the package readr is available, then it will be used. This can produce slightly different, but normally better, parsings.

To make sure that geoflow entity/contact loading works properly in the same way for all. readr is going to be added.

Support upload of Geotiff coverage

Manage upload of raster (coverage) resources including Geotiff format supported by GeoServer base distribution. other formats will be tackled in different tickets as not directly targeted by use cases now.

• ArcGrid,
• WorldImage,
• ImageMosaic.

See eblondel/geosapi#31.

Additional support required for extensions such as NetCDF.

Spatial Extent troubles

Hi again Emmanuel,

Spatial extent of datasets with native EPSG is not visible in geonetwork (although it's in the xml document).

Cheers,

wilfried

Zenodo action: support update based on concept DOI / DOI

When a DOI is available as identifier of an entity, the Zen4R action should try to get existing Zenodo record based on this DOI, first with a attempt by Concept DOI (assuming the DOI is a concept DOI), next to get record by simple DOI. This will allow to get/update published records (update metadata only) including existing versions of a record that are uniquely identified with a DOI.

Zenodo / Export modified source entities table with DOIs