Revamping Matrices: Data Format Changes and Integration of Apache Arrow IPC Binary Format about antarest

[sylvlecl]

Thanks for the description!
As discussed a few days ago, the proposed data model seems very good to me as the data model on the front side.

Then, in order to advance the dev step by step we would:

1. As a first step, re-create this data model from the current backend endpoints, and use data glide from it.
   This will allow us to validate the use of data glide without having to do heavy refactorings on the backend

2. As a second step, improve the way this data model is populated from the backend:

• By using arrow instead of JSON for the matrix content
• By computing the index on the backend side

3. About aggregates computation on the backend side, this would be aded in again another step I think. In particular, it will probably not be possible to have them inside the same arrow structure as the matrix content. So we need to think of how this could be transferred: either another endpoint (but then we should take care of caching the values on the backend side to not reload the file just for computing those values), or finding a way to return in the same body the arrow table and those values.

About matrix updates from front to back

Currently, those updates are specified through operations which have the advantage of not sending the whole matrix to the backend. I thinks in a first step we should keep this mechanism.
Then we should assess if it's OK to send the whole matrix every time as an arrow table or not.
Or maybe having several endpoints for different kinds of operations (updating only a few cells vs. updating a whole subpart of the table).