How to shuffle dataset after ParallelDatasetIterator ? about torchnet

[colesbury]

Here's what I generally do:

local d = tnt.TableDataset{data = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}}
d = d:shuffle() -- before the iterator()

-- So the we can call 'iterator:exec('manualSeed', seed)'
function d:manualSeed(seed) torch.manualSeed(seed) end

local iterator = tnt.ParallelDatasetIterator{
    nthread = 3,
    init = function() require 'torchnet' end,
    closure = function()
        return d
    end,
    ordered = true
}

for sample in iterator() do
    print(sample)
end

iterator:exec('manualSeed', epochNumber)
iterator:exec('resample')

for sample in iterator() do
    print(sample)
end

The call to manualSeed is there so that each thread shuffles the data in the same way.

[lvdmaaten]

I suppose we could add an optional seed to ResampleDataset (and ShuffleDataset)?

[Cadene]

That is a clever way. However it seems a bit tricky to me. Do you think there could be a way to integrate this functionality into torchnet?

[szagoruyko]

would be useful to have a dataset that reshuffles itself each :size() calls

[lvdmaaten]

I think that's very tricky. Imagine coding up some ranking loss that does hard negative mining during training. In that case, the number of get() calls per epoch will likely be different from size().