neicnordic / sda-pipeline

A federated storage for sensitive data, NeIC SDA

Archival notice

⚠️ This repository is no longer maintained. The code has been integrated and it is further developed at: https://github.com/neicnordic/sensitive-data-archive

sda-pipeline

sda-pipeline is part of NeIC Sensitive Data Archive and implements the components required for data submission. It can be used as part of a Federated EGA or as a isolated Sensitive Data Archive. sda-pipeline was built with support for both S3 and POSIX storage.

Deployment

Recommended provisioning method for production is:

• on a kubernetes cluster using the helm chart;

For local development/testing see instructions in dev_utils folder. There is an README file in the dev_utils folder with sections for running the pipeline locally using Docker Compose.

• First run
• Production like run
• Manual execution

Core Components

Component	Role
intercept	The intercept service relays message between the queue provided from the federated service and local queues. (Required only for Federated EGA use case)
ingest	The ingest service accepts messages for files uploaded to the inbox, registers the files in the database with their headers, and stores them header-stripped in the archive storage.
verify	The verify service reads and decrypts ingested files from the archive storage and sends accession requests.
finalize	The finalize command accepts messages with accessionIDs for ingested files and registers them in the database.
mapper	The mapper service registers the mapping of accessionIDs (IDs for files) to datasetIDs.
backup	The backup service accepts messages with accessionIDs for ingested files and copies them to the second/backup storage.

Internal Components

Component	Role
broker	Package containing communication with Message Broker SDA-MQ.
config	Package for managing configuration.
database	Provides functionalities for using the database, as well as high level functions for working with the SDA-DB.
storage	Provides interface for storage areas such as a regular file system (POSIX) or as a S3 object store.

Documentation

sda-pipeline documentation can be found at: https://neicnordic.github.io/sda-pipeline/pkg/sda-pipeline/

NeIC Sensitive Data Archive documentation can be found at: https://neic-sda.readthedocs.io/en/latest/ along with documentation about other components for data access.

Contributing

We happily accepts contributions. Please see our contributing documentation for some tips on getting started.

sda-pipeline's Issues

Send messages on error

Upon failures that aren't likely local/temporary, we need to make sure we send a message to the correct queue for errors queue for communication to the user as well as nacking the message so we do not get our queues filled up with messages that only lead to failures.

[Intercept] move faulty messages to error queue

[Ingest] copy file from inbox to archive

• split the c4gh header
• decrypt and validate the header

Use pipes to connect readers and writers

At the moment we are not able to perform batched S3 uploads since we cannot perform append-like operations using the S3 API. The solution might be to put all writes into a pipe and then let the uploader consume the pipe's reader.

Investigate error when `verify_peer=true` for MQ connection

Fail on startup if S3 backend can not be connected to

Create unit tests

All separate functions should have a corresponding test.

Add contributors and licenses

• include a section with allcontributors.org
• mention licenses with fossa.com

as pointed out in #206

Sync reads whole file in memory

The sync service is currently reading the whole file in memory, causing in to crash when the size gets bigger. We need to allow for specific buffer size instead

Standardize error queue messages

The error messages that pour in to the error queue should be made more uniform. This could be done e.g. by using one of the structs that we already have in place for errors.

Check mapper service Nacks for incorrect requeue

• DB
• MQ

Add `sync/backup` service to test suite

DoD: Ingestion test run with the sync/backup service doing backup of file from archive before finalize receive the message to start it's work. sync/backup listens to accessionIDs and publishes with the routing key so that the message ends up in the queue finalize listens to. finalize can not listen to accessionIDs in this scenario.

Check verify service Nacks for incorrect requeue

• DB
• MQ

We need to provide the proper md5 checksum for accession from verify.

Create dataset id that is not dependent on path

Currently the orchestrator is creating dataset ids based on the path of the file. That needs to change, given that files might be uploaded in the same folder.

Ideas:

• Deploy a service that gets the message from completed and upon user interaction it creates the dataset id
  The dataset id will be given (possibly from the data stewards - DOI)

Inputs:

1. File paths as they are defined in the metadata
2. DOI for the specific dataset
3. Submitter's username

Output:
Message for mapper service

Could even be a Kubernetes job

Note: The dataset id creation is not an automatic process

Docker files for SDA-Pipeline

create integration test

• ingest on posix + posix followed by verification
• ingest on posix + S3 followed by verification
• ingest on S3 + posix followed by verification
• ingest on S3 + S3 followed by verification

Publish the initial message to the files routingkey in the localega exchange to start the ingestion process. This should be able to be done using curl via RabbitMQs API.

{
"user": "test",
"filepath": "dummy_data.c4gh"
}

Remove verified files from inbox

Dod:

• When a file has been verified OK it should be removed from the inbox.

Fail on startup if c4gh keyfile can not be opended

shared config code

Handle database connection issues

Make sure we handle everything correctly if there are issues with the connection.

Encapsulate db logic with structs and interfaces

Db logic should implement interfaces to their main methods.

Messages, s3 and eventual consistency

Since s3 is eventually consistent, it's quite possible that we get messages for a file in inbox and when we try to access. I believe this (or some similar issue) is currently seen in #141.

The fix for that will likely be different, but for ingestion as well as verification, we should probably handle the case that the message arrives before the s3 provider gets its things in order - that is, we should not treat a failure of getting a filesize or reader as fatal but rather see that as that we should retry (up to a certain limit).

I'm not sure how that is best done through the broker (requeue or send a new message) and what can be done (include a counter? Do we have a timestamp we can look at?) but we will probably need to do something.

Check ingest service Nacks for incorrect requeue

• DB
• MQ

Document DB permissions

Might need to refactor the DB schema to simplify stuff in the end.

Add shellsheck and shfmt to actions suite

Run on each push to any branch except master.

allow custom accession id patterns for standalone deployment

current deployment is restricted to ^EGAF[0-9]{11}$'] pattern however to use it in standalone mode one would need to enable a custom pattern.

The aim is to enable custom schema depending on deployment

Fix broken test

8_bad_json.sh and 10_trigger_failures.sh are broken and needs investigating.

Rename `Sync` to `backup`

There is a service called sync in the pipeline. This should more aptly be named backup (since it does backups). This service needs to be renamed, documentation updated, and references updated.

DB TLS configuration

sda-pipeline/internal/config/config.go

Line 247 in 997cfd6

if viper.IsSet("db.sslMode") && viper.GetString("db.sslMode") == "verify-full" {

Right now only verify-full mode is supported as optional. We need to support all SSL modes, or, at least, require.

Refactor db logic

We should use encapsulate db structs in a generic Db struct instead of using the psql struct directly.

Log correlation ids

In INFO mode we should also log the correlation ID

Run all static code tests on PUSH

That means go ver, go lint gofmt and so on

FEGA Accesion IDs problematic if same file uploaded several times

Accession ids from FEGA is based on the checksum of the (probably decrypted) file. So if the same file is uploaded several times there will be two entries in the database with the same accession id.
This makes it difficult to figure out which line belongs to which dataset. The reason this is important for example the filename needs to be correct for the private metadata (that is uploaded to the FEGA node). Also there should not be private metadata in the filenames but this could happen and those should not leak by mistake between datasets.

DoD
We need to be able to better distinguish between files in the database and when we ingest them.
This might involve some db changes to https://github.com/neicnordic/sda-db if we want to modify the db schema

shared storage components

• posix support
• S3 support

Organise storage business

We should be able to get rid of the config.ArchiveType check and keep storage logic contained.

Add ingestion test for standalone deployment

Ensure the messages get nacked and put back in the queue if the work can not be completed.

Fail on startup if posix inbox/archive path does not exists

calculate un-encrypted file size in `verify`

we should calculate un-encrypted file size in verify

Problem is which column to use to store in db. inbox_filesize seem unnecessary at the moment and could be used

[Verify] Implement file verfication

• Decrypt the file using https://github.com/elixir-oslo/crypt4gh
• Don't keep the key unlocked unless it is in use.
• Compute SHA256 sum of the unencrypted file as we decrypt it

Add docker-compose without TLS

Currently the dev_utils has TLS enabled by default, we need a docker-compose for local development to start the services, sda-db and sda-mq without TLS.

validate incomming messages via json schema

sda-pipeline reconnect to MQ

[broker] publish message

publish message using exchange and body as payload.

Check finalize service Nacks for incorrect requeue

• DB
• MQ

Add mapper

• Rework mapper to use our shared configuration
• Include in functionality tests

Document the Go code

(Pontus, X)

With inline comments following go documentation practices.

Extend sda-sync service to also copy header

For big picture we need to sync files between Sweden and Finland. To do this we need to be able to sync the header as well.

Extend the s3sync service so it can get the header from the database and reencrypt it using another public key.

Question about protocol? Should header and body be concatenated or should it be sent separately?

• The simplest solution would be to fetch the header, encrypt it and pass it through the MQ message

• reencrypt with Finish public key

• standalone deployment template

Deployment documentation

Update and/or add documentation on how it's supposed to be setup and configured.

Figure out how to cross-reference documentation we have in the project.