Multiprocess load batch about semantic-segmentation-tensorflow-eager

[bhack]

If you want to continue in Eager mode check also https://www.tensorflow.org/tutorials/eager/eager_basics#datasets

[bhack]

Generally the dataset API is already highly optimized.

[Shathe]

Yeah, you're right
I am currently using my custom Loader for loading and performing data augmentation but... I think it's about time to move into the data.Dataset API haha
I am gonna open another issue for that

[bhack]

https://www.tensorflow.org/performance/datasets_performance

[Shathe]

Nevertheless... I just saw it andd... Do you know if you can use it when the data does not fit into RAM? I mean, if you cannot load the entire dataset at once... You can still use the Dataset API?

[bhack]

Yes of course

[Shathe]

You know how? I mean, can you provide me a link or a function name or something? I'll try to search on internet and on the guide you linked in previous messages too

[bhack]

Almost all the real datasets don't fit in memory so you will find many many dataset API examples. Just to mention one https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/contrib/eager/python/examples/generative_examples/image_captioning_with_attention.ipynb

[Shathe]

I mean, referring to the size of the dataset

[bhack]

That dataset doesn't fit in memory

[Shathe]

Thanks a lot!