how to input my own dataloader to it? about pytorch-widedeep

[javier-cazana]

Hi there @lireagan

thanks for opening the issue and sorry for the delay, I am a bit buys these days but I aim to come back to the package soon

Regarding your request, at the moment you can't. DataLoader are built internally here:

pytorch-widedeep/pytorch_widedeep/models/wide_deep.py

Line 499 in bdf777d

train_loader = DataLoader(

within the main "collector" class WideDeep. The idea is that the user inputs arrays the rest in handled internally. I guess I could add the option of passing directly a loader to the .fit method, but that would require changing some code.

Alternatively, you could tell me what you want to do and maybe you can already do it with the package (?)

[lireagan]

@javier-cazana

Thank you for your response! There are 2 reasons why i want to use customized dataloader:

1. If I only use csv files as input, it will load the whole dataset on cpu memory, and when the dataset is very large(for example, 2M samples), it will result in OOM. So I want to use some dataloaders which support batch-input.
2. in some cases, my datasets are not save locally, they maybe on cloud, so I need the customized dataloaders provided by the cloud company.

[javier-cazana]

Hi @lireagan

ok, yeah, possibly the easiest option would be refactoring the code to allow custom dataloaders, I will think how to do that smoothly.

In the meantime:

1. You could code you own batch generator, generate batches out of np arrays and pass them to the model. Bear in mind that one of the reason why I designed the dataloader as I did is because all components (wide, deep, text, image) have to be loaded in the same order (and you cannot do "random" data augmentation)

2. let's say that your data is on S3. You can still download it locally and then pass it to the model? I mean, eventually is going to happen anyway, at some point during the process the data will be in your RAM, so you could design a process by which: S3 -> RAM (array) -> model (? )

Maybe I don't see all the process (or the whole pic), but let me know if this helps. I will try to add examples for those two solutions and let you know when I am done. I am bit busy these days, but I will do my best!

[lireagan]

Yes, I'm refactoring the fit function and use my batch generator as an alternative.
The question 2 is that, the cloud provide a on-line read function:

reader = common_io.table.TableReader(table_name, selected_cols)
data = reader.read(num_records)

And I want to use this API to construct my own Dataset and generate batch-size inputs.

[jrzaurin]

Ok, I see...

I will then look to add that functionality in the future.

Going to close this issue. I will re-open it if needed

Thanks for the issue @lireagan