Output format revamped about eko

[Zaharid]

I went over these messages and would tend to agree that .npy looks like a good match. Reading its description it looks like our use case and it has the bindings one might need to use it directly. I like the idea of a tar with a yaml header plus the grid as npy.

[alecandido]

Incidentally we speeded up quite a lot the IO operations @Zaharid @felixhekhorn.

Most likely since the yaml parser is quite slow (and maybe same thing for the writer), so we get quite a lot from stripping the bulk data from the .yaml.

[alecandido]

The full proposal is then to make the output:

• a .tar archive of a folder containing
  • a .yaml file with metadata
  • a .npy.lz4 rank 5 tensor

[Zaharid]

Out of curiosity, how big are these files typically?

[alecandido]

It really depends on how many Q2 you ask for, and which interpolation grid you use (on input and output, there is a third one used internally that does not affect the output shape, and it's used to increase accuracy even when the input and output are rough; by default they are all the same, but maybe there won't be a default at all).

I tested with current version, and for a single Q2, with the same xgrid in input and output of 49 points, you get:

• 908k compressed
• 11M uncompressed

For the uncompressed is essentially (whatever block size) x input_xgrid x output_xgrid x #Q2, for the compressed one might be less trivial (even more considering that we'll compress the rank 5 tensor, so all Q2 together).