Convert clojure data -> NDArrayWritable about jutsu.ai

[hswick]

I made attempts at this before and ran into the same complexity. I actually think the best way to do is to write some java code to make a more clojure friendly DataSetIterator

[behrica]

I did something like this in parts
The snippet below takes "features" as a vector of maps,
and "y" as a vector of targets
and it will create an NDArray
and then a DataSet and then a DataSetIterator
which could be used with the existing method "train-net!"

(def data (Nd4j/create (into-array (map double-array (map vals features))))
 (def labels (.transpose (Nd4j/create y)))
(def data-set (DataSet. data labels))

.. split  "data-set" into test and train

(def dataset-iterator (ViewIterator. train-data-set 1000) 10))

What makes it complex, is that the logic of

• epochs
• batches vs. mini-batches
• in-memory vs stream data from disk
• test-train split
• normalization

is all
encapsulated or uses the DataSetIterator interface, so it has lots of implementations, and the user needs to be able to choose and configure them.

To have this functionality fully on the Clojure site, a lot of implementations of DatasetIterator
logic needed to be ported to Clojure:
https://deeplearning4j.org/api/latest/org/nd4j/linalg/dataset/api/iterator/DataSetIterator.html

And this code is very much state-full OO style, so hard to make functional.

But maybe a single function , implementing "one scenario",
like

(defn iterate-data [features (vector of maps)   labels (vector)   test-train-split-percentage num-batches num-epochs)

-> returns 2  data-set iterators (for train and test)

could be useful. And it would not required any change in existing methods, as they can already handle any DataSetIterator