Pickling issue with multiprocessing about hyperactive

[SimonBlanke]

I was not able to reproduce this error in Ubuntu 20.04 with the same python and hyperactive version. The fact that it works on my end surprises me, because Hyperactive uses multiprocessing as default. I always thought multiprocessing cannot handle lambda because it uses pickle.

Maybe you could try using joblib for Hyperactive. It is a shot in the dark but maybe this works for your OS.

Here is a snipped to try out:

import numpy as np
from hyperactive import Hyperactive

space = dict(alpha=np.linspace(0, 1), beta=np.linspace(0, 1), start=[-0.95], end=[0.95])
points = np.random.rand(10)

def model(opt):
    rv = lambda x: x
    return np.sum(rv(points))

hyper = Hyperactive(distribution="joblib")
hyper.add_search(
    model,
    space,
    n_iter=50,
    n_jobs=2,
)
hyper.run()

Let me know if this solves your problem.

It is difficult for me to reproduce this error, because I do not have access to MacOS. I hope we find a solution with joblib.

[m-rph]

Joblib works, thanks!

One more thing, n_jobs is independent of the number of runs, is this correct? So if I wanted to run the experiment 50 times, I'd have to call run 50 times and collect the results each time, is this correct? I am asking because running:

hyper.add_search(model, space, n_iter=50, n_jobs=N)

gives me min(N, 12) for N >= -1, and I assume 12 is because my CPU shows 12 cores.

[SimonBlanke]

Very good! It works now, because joblib uses an alternative to pickle.

I am not sure i understand your question.
n_jobs opens n processes during the optimization run. Each of those processes runs n_iter iterations. If n_jobs == -1 then n_jobs is set to the number of CPU cores available.

I hope this answers your question. If you need more clarification or have another question feel free to open another issue.