ocramz / mnist-idx-conduit

conduit utilities for mnist-idx

mnist-idx-conduit

Conduit decoders for MNIST IDX data files, as described in http://yann.lecun.com/exdb/mnist/ .

The IDX format is a low-level binary representation for storing general vector, image or tensor data.

Here the parsers are specialized to the MNIST dataset, which stores scans of handwritten digits and is a classic benchmark for text classification.

mnist-idx-conduit's Issues

Roundtrip test fails

Serialization (sinkIDX_ and functions using it) seems to be buggy, namely part of the header is not serialized correcly, leading to inconsistent deserialization. One can see it by running the test (stack test).

As a result of the test, a small IDX file is produced in the test/ folder.

The header is mostly correct, but the last field of the tuple shows a single element list whereas there should be two ([5, 2], as declared in ConduitSpec.writeData)

λ> readHeader "test/ConduitSpec_temp.idx"
(IDXMagic {idxType = IDXUnsignedByte, idxNumDims = 2},11,[5])