Support Infinite PyDataset about keras

[fchollet]

There are two separate issues here: support for infinite cardinality, and support for dynamically sized batches.

For infinite cardinality support, we can use None to indicate it. We can definitely support it, similar to what we do for infinite tf.data.Dataset instances. If you're able, feel free to open a PR!

[LarsKue]

Yes, I felt the two issues are similar enough to group them in one issue. Apologies if this is not how you see it.

As for returning None as the length, I feel this is a little risky, since forgetting a return statement also looks like returning None from the outside. This could be an issue for a dataset that has a more complex length computation, such as one that increases in length as training goes on (e.g. experience replay buffers).

[hertschuh]

Hi @LarsKue ,

Thank you for reporting this.

For item 2., this is already how it works. __getitem__ returns a full batch, and the batch size can be different from batch to batch.

For item 1. I just pushed a change that enables this. You implement the num_batches property and return None to signal an infinite dataset. I understand the concern that if you forget a return clause, you could return None by accident. But weighing this against other options, it seemed relatively less error prone (you could easily do incorrect math with -1).

Note that you can also easily feed a infinite dataset to model.fit() by using a generator.

def generate():
    while True:
        x = ...
        y = ...
        yield x, y

model.fit(generate(), ...)

It's less code than implementing a PyDataset and doesn't require being able to access your data by index. However, it doesn't support multiprocessing / multithreading.