Memory issues for huge number of samples about pyphs

[afalaize]

Hi!
Yes, originally we used generator to avoid memory overload for large input data and had to switch to arrays to be able to write the entire input data to hdf5 in one shot.
I agree with your proposal to allow both arrays and generators with a buffer/writer solution
Best regards;
Antoine

[FabricioS]

Hi,
I tested a little more the snipplet given above. It seemed to be quite inefficient as it scans the generator at the Python level. Another option is to let numpy scan the generator (in a chunked way, no creating the full array), which is done in the C-side.

Here is a small "benchmark":

import h5py as h5
import time, warnings
import itertools
from_iterable = itertools.chain.from_iterable

# Naive method: Python-side scan of the generator
def set_dataset_from_generator_naive(dataset, generator, chunksize=1024):
    offset, npt = 0, chunksize
    while npt:
        buffer = [el for ind, el in zip(range(chunksize), generator)]
        npt = len(buffer)
        dataset[offset: offset+npt] = buffer
        offset += npt

# Numpy method
def set_dataset_from_generator_numpy(dataset, generator, chunksize=1024):
    nt, nsamples = dataset.shape
    dtype = dataset.dtype

    offset, npt, count = 0, 1, chunksize * nsamples
    flattened_generator = from_iterable(generator)
    
    while True:
        if nt >= chunksize + offset:
            buffer = np.fromiter(flattened_generator, dtype, count)
            dataset[offset: offset+chunksize] = buffer.reshape(chunksize, nsamples)
            offset += chunksize
        else:
            warnings.warn('Last chunk : %d < %d' % (nt - offset, chunksize))
            buffer = np.fromiter(flattened_generator, dtype)
            dataset[offset: offset+chunksize] = buffer.reshape(-1, nsamples)
            break

lnpt = 2**np.arange(14, 26)

timings = np.zeros((len(lnpt), 2)) * np.nan

with h5.File('/tmp/test1.h5', 'w') as fid:
    for inpt, npt in enumerate(lnpt):
        dset = fid.create_dataset('/arr1', shape=(npt, 1), dtype=np.float64)
        tip = time.time()
        set_dataset_from_generator_naive(dset, ((el,) for el in range(npt)), chunksize=1024)
        timings[inpt, 0] = time.time() - tip
        del fid['/arr1']

        dset = fid.create_dataset('/arr2', shape=(npt, 1), dtype=np.float64)
        tip = time.time()
        set_dataset_from_generator_numpy(dset, ((el,) for el in range(npt)), chunksize=1024)
        timings[inpt, 1] = time.time() - tip
        del fid['/arr2']

import matplotlib.pyplot as plt
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.loglog(lnpt, timings, marker='o')
ax.legend(('Naïve', 'Numpy'))
ax.set_xlabel('# values')
ax.set_ylabel('Elapsed time (s)')

Conclusion : an average x2 speed-up with the set_dataset_from_generator_numpy method

[FabricioS]

Last point, I am unsure the scan of the generator is fully done at the C level even in the second method.
In fact, np.fromiter only builds 1D array and the itertools.chain.from_iterable function is used to produce a flattened version of the generator and solve the np.fromiter limitation.

Knowing the shape of the dataset and of the expected inputs, one could produce an even more efficient method.

[afalaize]

Thank you @FabricioS for this benchmark. I implemented the "Numpy method" in the file numerics/simulations/h5data/tools.py called in H5Data. The code is currently in develop branch and PR pass tests.
@FabricioS @WetzelVictor Can you check this solves your case?

[WetzelVictor]

I tested the patch on the same simulation than before: system of dimensionality 4 and 1e6 samples. With this, the program barely used any memory. Previously it used at least 1/3 of it.

No problems on my machine, just warning from hdf5 not being happy about pyphs using generators.

Thank you @afalaize and @FabricioS !

[FabricioS]

Hi, can you paste the warning ? [PS: I had to edit the comment that I first posted through the email API]

[afalaize]

Hi! Good to hear that.
I just forgot to remove the deprecation warning, forget about it: it has no meaning...