Aggregations about fmu-dataio

[perolavsvendsen]

Also potentially challenging wrt to relative_file_path and absolute_file_path, depending on how these are made under the hood.

[perolavsvendsen]

Should perhaps be seen in relations to #136. Very different data, but possibly similar implementation.

[perolavsvendsen]

Refinement 13 jan 2022 @jsorb @jcrivenaes

Requirements

• fmu-dataio needs to be able to produce metadata for aggregated data. E.g. when given an aggregated surface + necessary arguments, return that aggregated surface to disk along with metadata.
• Aggregations can be multi-ensemble, multi-case, etc.
• For now, limit to aggregations across realizations within one iteration within one case.

Assumptions

• A set of realizations exists.
• Compliant metadata already exists.
• Calculation of aggregated data is not responsibility of fmu-dataio
• Fmu-dataio is to be used by other calculators to produce metadata only

Currently (in x installations):

• One realization (0 if present) is used as metadata template
• Aggregation functions modify the realization metadata into aggregation metadata + checks consistency across all realizations.

Proposed input to fmu-dataio:

• List of all metadata dictionaries for all inputs to the aggregation.
• Type of aggregation

Initial tests:

• Given a surface + a list of metadata, fmu-dataio produce aggregated metadata according to schema.
• Create surfaces with known values, create metadata with fmu-dataio, aggregate, check that resulting metadata is according to spec.
• Look for corner cases as we go along.

[perolavsvendsen]

Notes from discussion 2022-02-28:

At least two situations to be covered:

1. Metadata exists and is available for a set of input data objects.
   1 a) fmu-dataio shall use existing metadata as template for new metadata
   1 b) fmu-dataio shall not use existing metadata, build metadata from nothing (through input args as in .export())

2. Metadata does not exist/is not available for a set of input objects.

--> Use existing metadata as input, OR leave it to the aggregation service to handle existing metadata, and provide only key arguments in. From the perspective of fmu-dataio, it will be like generating metadata from nothing.

Is each post-process/pre-process "special" enough to justify separate functionality? Or are "most" generic, with a few special? Or is none special?

Use of existing metadata: Risk of handling metadata logic outside fmu-dataio is that updates to metadata becomes very difficult.

Possibly smart to mitigate conceptual discussion by having context-specific (public) classes in fmu-dataio.

Try to rebuild existing ExportData to handle also aggregations (and other pre/post-processed data). Under the hood call other methods? Start with writing examples.