arcalot / arcaflow-plugin-minio

Arcaflow plugin for ephemeral object store using MinIO

Arcaflow MinIO Plugin

The MinIO plugin creates an ephemeral object store that can be used during the operation of a workflow. A single arca-bucket storage bucket is created, and the plugin returns in its output object the required credentials for accessing the bucket.

Accessing the bucket

In order to access the bucket, the MinIO plugin container will need to be launched with internal port 9000 exposed.

The MinIO object store is S3 compatible, so it is possible to use the S3 libraries, the simpler MinIO libraries (recommended), or any other S3 compatible client methods to access the bucket.

The MinIO server is run as insecure so connections are via HTTP without certificates.

Persistent storage

If you would like for your bucket contents to survive after the workflow execution, you will need to mount a persistent filesystem to the container at /mnt/arca-bucket. You will then be able to run a MinIO server separately using this storage location and re-establish access to the bucket outside of the plugin.

Simple ephemeral object store

Do you just need a quick and simple containerized ephemeral object store? You can easily run the plugin directly with a JSON input object like this:

$ echo '{"minio_user":"myuser", "minio_password":"mypassword", "bucket_name":"mybucket", "run_duration":"300"}' | docker run -i --rm -p 9000:9000/tcp -p 9001:9001/tcp [-v /tmp/arca-bucket:/mnt/arca-bucket] quay.io/arcalot/arcaflow-plugin-minio:latest --debug -f -

This will create a MinIO container hosting a mybucket bucket via container port 9000 and accessible via S3 protocols with the user/password combo of myuser/mypassword. You can optionally use the -v parameter of docker run to map a local storage location to the container /mnt/arca-bucket path. Additionally, the MinIO console is made available via container port 9001. The container will exit automatically after the run_duration (in seconds) expires.

Autogenerated Input/Output Documentation by Arcaflow-Docsgen Below

MinIO (minio)

Runs the MinIO server and sets up a bucket

Input

Type:	scope
Root object:	InputParams
Properties	bucket_name (string) Name: bucket name Description: Name for object bucket Required: No Default (JSON encoded): "arca-bucket" Type: string minio_password (string) Name: MinIO password Description: The MinIO server password Required: No Type: string minio_user (string) Name: MinIO username Description: The MinIO server username Required: No Type: string run_duration (int) Name: run duration Description: Time in seconds that the MinIO plugin runs before being forceably stopped Required: Yes Type: int
Objects InputParams (object) Type: object Properties bucket_name (string) Name: bucket name Description: Name for object bucket Required: No Default (JSON encoded): "arca-bucket" Type: string minio_password (string) Name: MinIO password Description: The MinIO server password Required: No Type: string minio_user (string) Name: MinIO username Description: The MinIO server username Required: No Type: string run_duration (int) Name: run duration Description: Time in seconds that the MinIO plugin runs before being forceably stopped Required: Yes Type: int

Outputs

error

Type:	scope
Root object:	ErrorOutput
Properties	error (string) Required: Yes Type: string
Objects ErrorOutput (object) Type: object Properties error (string) Required: Yes Type: string

success

Type:	scope
Root object:	SuccessOutput
Properties	access_key (string) Name: access key Description: The MinIO server access key (user) Required: Yes Type: string secret_key (string) Name: secret key Description: The MinIO server access secret (password) Required: Yes Type: string
Objects SuccessOutput (object) Type: object Properties access_key (string) Name: access key Description: The MinIO server access key (user) Required: Yes Type: string secret_key (string) Name: secret key Description: The MinIO server access secret (password) Required: Yes Type: string