IndexedDataset using string as index for large dataset about torchnet

[lvdmaaten]

This sounds like a sensible approach to me. What is the problem with it?

[Cadene]

Actually, I could not make up my mind between using tds.Hash or torch-hdf5 for the hash table or looking for something else in torchnet.

tds.Hash seems to be perfect for a not so large dataset (mscoco size).

Thanks for your help.

[lvdmaaten]

Well when you're training with SGD, you wouldn't need the tds.Hash() at all. I would expect you only use it when you want to compute results for specific images at test time.

[Cadene]

You are right. However, my dataset is multimodal (Visual Question Answering). Thus I need to extract in a first step the features associated to the mscoco images. Then I train my model with SGD over the VQA annotations datasets. Two annotations can be associated to the same mscoco image.
That is why I need such a two-step pipeline.