What's the difference between sdata & webdataset? about generative-models

[Jonisa2022]

import webdataset as wds
Create a webdataset object
dataset = wds.WebDataset("/path/to/data")
Load the data
data = dataset.load()
Process the data
...

The raw webdataset class is not recommended for use because it is not as efficient as the other classes in the webdataset library.

[amjadkhanniazi]

webdataset format is efficient for training large models and related data is bundled together in this format. where sdata is in simple text format or json format, each line contains a data sample, good for training small or medium models.