Exported Artifact
The exported artifact from Aqueduct will have two main data items:
Experiments metadata: This is mainly database records and their relations alongside extra information bits about the version of Aqueduct, the user that has created them, etc.
Experiments' files: These are the experiment files uploaded by the users or generated by the plugins.
The exported artifact is going to be a single tar.gz* file that has the following directory structure:
artifact
| metadata.json
|
----experiment_files
|
--------[UUID of experiment]
| | ...
- Python supports most of the main compression methods through different libraries. The trade-off is mainly between compression speed versus compressed size. I think gunzip (tar.gz) has a good balance, which is why it is pretty common, but this is open to discussion. More here: All The Ways to Compress and Archive Files in Python
Exported Experiments metadata Format
Based on the requirements of exporting data into a text document, the usual options are JSON and YAML. JSON is a better solution for data transfer as it can be minimised by removing white spaces and there is more support for parsing JSON rather than YAML, for example, Pydantic is based on JSON. Furthermore, if size was a problem. JSON can be converted to CBOR, which is an IETF standard for binary representation of JSON documents.
The JSON schema is version-less on its own, but it can include metadata to show the version and other information about the Aqueduct instance it was exported from.