yasufumy / pipelib

:zap:A minimal data pipeline library for Python

Please use lineflow

pipelib's Issues

Implement parallel computation

https://github.com/uqfoundation/multiprocess

Update README

• update README.md
• update examples

Save pipeline functions

• add pipeline saving example
• use cloudpickle

Bug fix in test code

• TextDatasetTestCase doesn't work
• DirDatasetTestCase doesn't work

Add Dataset.unzip

def unzip(self):
    def f(datasets):
        x = datasets.first()
        assert any(isinstance(x, t) for t in (list, tuple))

        dataset, *others = zip(*datasets)
        return (Dataset(dataset),) + tuple(Dataset(other) for other in others)
   return PipelinedDataset(self, f)

But this is eager evaluation

Add a concat method

• add Dataset.concat

class Dataset:
    def concat(self, *others):
        def f(dataset):
            yield from chain(dataset, *others)
    return PipelinedDataset(self, f)

Shorten concatenate method

• rename Dataset.concatenate to Dataset.concat

Add a zip method

class Dataset:
    def zip(self, *others):
        def f(dataset):
            yield from zip(dataset, *others)
        return PipelinedDataset(self, f)

Option for order/unorder

• Dataset.map_parallel
• Dataset.flat_map_parallel
• Dataset.filter_parallel

Update a test case for Dataset.load

add examples

Save pipeline

• save pipeline by cloudpickle

Refactoring Dataset.shuffle

def f(dataset):
    iterator = iter(dataset)
    for chunk in takewhile(bool, (list(islice(iterator, shuffle_size)) for _ in count(0))):
        random.shuffle(chunk)
        yield from chunk

Add docstring

Update README

Make a PyTorch example

• use pipelib to load dataset
• FYI

Refactoring example/training_loop.py

• fix here

Dataset.split

Do we need Dataset.split?

>>> data = Dataset(range(100))
>>> train, test = data.split(ratio=[0.8, 0.2])
>>> len(train.all())
80
>>> len(test.all())
20

Add Dataset.window method

FYI

Fix bug in Dataset.save

class Dataset:
    def save(self, filename):
        evaluated_dataset = list(self)
        with open(filename, 'wb') as f:
            pickle.dump(evaluated_dataset, f)
        return EvaluatedDataset(evaluated_dataset)

class EvaluatedDataset(PipelinedDataset):
    ...

add save/load methods

Move Dataset.save and Dataset.load to pipelib.serializers

Dataset.save and Dataset.load are ambiguous.
What is exactly serialized by these methods?

• update a method name
• move these methods to pipelib.serializers