rreverser / serde-ndim

Serde support for n-dimensional arrays from self-describing formats

serde-ndim

Overview

This crate provides a way to serialize and deserialize arrays of arbitrary dimensionality from self-described formats such as JSON where no out-of-band data is provided about the shape of the resulting array.

This is useful for some data sources (e.g. in astronomical applications), but not the format supported by the built-in Serde integration of popular crates like ndarray or nalgebra.

Consider input like the following:

[
    [
        [1, 2, 3, 4],
        [4, 5, 6, 7]
    ],
    [
        [7, 8, 9, 10],
        [10, 11, 12, 13]
    ],
    [
        [13, 14, 15, 16],
        [16, 17, 18, 19]
    ]
]

This should deserialize into a 3-dimensional array of shape [3, 2, 4]. This crate provides serialize and deserialize functions that can be used via #[serde(with = "serde_ndim")] that do just that.

Deserialization

The tricky bit is that deserialization is built to learn and ensure internal consistency while reading the data:

1. During the first descent, it waits until it reaches a leaf number (1) to determine number of dimensions from recursion depth (3 in example above).
2. It unwinds from the number one step up and reads the sequence [1, 2, 3, 4], learning its length (4). Now it remembers the expected shape as [unknown, unknown, 4] - it hasn't seen the lengths of the upper dimensions, but at least it knows there are 3 dimensions and the last one has length 4.
3. It unwinds a step up, recurses into the next sequence, and reads [4, 5, 6, 7]. This time it knows it's not the first descent to this dimension, so instead of learning it, it validates the new length against the stored one (4 == 4, all good).
4. It reached the end of this sequence of sequences, so now it knows and stores the expected shape as [unknown, 2, 4].
5. By repeating the process, it eventually learns and validates the shape of the whole array as [3, 2, 4].
6. All this time it was collecting raw numbers into a flat Vec<_> traditionally as an optimised storage of multidimensional arrays. Now it just needs to call a function that constructs a multidimensional array from the shape and flat data.

Note: The resulting array will be in the standard column-major layout.

Constructors for ndarray::Array and nalgebra::DMatrix are provided out of the box under the ndarray and nalgebra features respectively, so you can use them like this:

use serde::{Deserialize, Serialize};

#[derive(Deserialize, Serialize)]
struct MyStruct {
    #[serde(with = "serde_ndim")]
    ndarray: ndarray::ArrayD<f32>,
    /* ... */
}

You can also reuse deserialization for custom types by implementing the serde_ndarray::de::MakeNDim trait.

Serialization

Serialization is also provided. Its implementaton is much simpler, so I won't go into details here, feel free to check out the code if you want.

It's also provided for ndarray::Array and nalgebra::DMatrix, but if you want to serialize custom types, you can do so by implementing the serde_ndarray::ser::NDim trait.

serde-ndim's Issues

Absolutely no clue what could be going on here

I've been trying to just run the example code provided and I constantly get this exact error no matter what combination of dependency versions I specify. I've even gone as far as to download and manually build the source of this crate and, as far as I can see, the trait is most certainly implemented for the types I'm trying to use. Any ideas?

#[derive(Debug, PartialEq, Deserialize)]
pub struct PolynomialFeatures {
    //pub params: PolynomialParams,
    #[serde(with = "serde_ndim")]
    ndarray: ArrayD<f32>,
}

error[E0277]: the trait bound `ArrayBase<OwnedRepr<f32>, Dim<IxDynImpl>>: MakeNDim` is not satisfied
   --> src\refractionizer.rs:54:28
    |
54  | #[derive(Debug, PartialEq, Deserialize)]
    |                            ^^^^^^^^^^^ the trait `MakeNDim` is not implemented for `ArrayBase<OwnedRepr<f32>, Dim<IxDynImpl>>`
    |
note: required by a bound in `serde_ndim::deserialize`
   --> C:\Users\Ben\.cargo\registry\src\index.crates.io-6f17d22bba15001f\serde-ndim-1.1.0\src\de.rs:230:8
    |
228 | pub fn deserialize<'de, A, D>(deserializer: D) -> Result<A, D::Error>
    |        ----------- required by a bound in this function
229 | where
230 |     A: MakeNDim,
    |        ^^^^^^^^ required by this bound in `deserialize`
    = note: this error originates in the derive macro `Deserialize` (in Nightly builds, run with -Z macro-backtrace for more info)