Does TFDF support lazy dataset loading during training? about decision-forests

[achoum]

Hi Willian,

Thanks for the interest :)

Yggdrasil DF supports two types of dataset inputs for training: Training from an in-memory dataset or Training from a set of dataset files (for example a collection of TFRecord files). The first option is efficient for small datasets, while the second one is best for large datasets (e.g. datasets not fitting in memory). Each learning algorithm implements one of the other interfaces.

We have not opensourced any learning algorithm for large datasets in Yggdrasil yet. This should happen soon (mid Q3). Notably, we will open source the Exact Distributed Random Forest algorithm.

Currently, TF-DF uses the in-memory interface of Yggdrasil. At training time, the dataset is streamed and stored in memory (with a few memory optimizations). At the end of the first epoch, the Yggdrasil training starts.

We are currently working on a distributed version of TF-DF compatible with TF Distributions Strategy. Both the data and the computation will be distributed. This will be released at the same time as the distributed learning algorithm in Yggdrasil.

Cheers,
M.

[rjchee]

Hi, sorry for reopening this old issue. Is there currently any solution which allows for a single worker to stream over a dataset instead of requiring the entire dataset to be in memory?

In my case, I have a large dataset that worked reasonably well with a GBDT model from TF1's estimator API, but I'm having difficulty migrating it over to use this library due to the massive memory requirements from storing the dataset. Migrating to use distributed training on a cluster of workers is not an option for me unfortunately.