Random seed being set somewhere hidden inside baybe? about baybe

[AdrianSosic]

Hi @brandon-holt, I'm just experimenting with some ideas. Could you do me a little favor and run

pip install git+https://github.com/emdgroup/baybe.git@fix/random_seed

to see if your problem is solved on that branch?

[AdrianSosic]

Update: I think I might have found an easy solution actually. Can try to implement this in the next days and will keep you updated. Until then, I hope you can use your manual workaround 🙃

[AdrianSosic]

HI @brandon-holt. Yes, currently global random seed is set within the simulation package – not great, I know 🙈 I've had this on my bucket list for ages, but never had the time to change it, unfortunately... So yes, currently a known "feature" of the simulator. (And again, +1 item on the simulation refactoring list 😄). How critical is this for you currently?

[brandon-holt]

@AdrianSosic okay phew glad Im not crazy!! haha 😮‍💨

I'd be more than happy to just do a local workaround on my end for the time being, if you could just point me to the area where this is being set! Would it break anything for baybe if i just comment out where it sets the global seed?

[AdrianSosic]

Sure, there is a call set_random_seed(random_seed) rather at the beginning of simulate_experiment that controls it. The function simply sets all global seeds of all relevant random number generators (built-in random module, numpy and torch).

I have already tried to wrap my head around what would be a better/reasonable alternative, but so far I could only come up with solutions that required some rather invasive changes to function calls to ensure that all modules at the lower levels actually use their own local random generator instances. But perhaps that's simply the way to go. Let me know if you see a nice solution to the problem 👍🏼

[brandon-holt]

Sounds great, thanks!

[brandon-holt]

@AdrianSosic This solved my problem (mostly)! While the results returned from my method are now different every iteration, the results from baybe.simulate_experiment are still similar like 3-4 times out of 5 iterations. When they are the same, they are exactly the same, as it is the same experiments recommended each iteration.

I suppose this might make sense, since the same experiments that maximize information gain would be same given the same dataset, however, I am using a TwoPhaseMetaRecommender with initial_recommender=RandomRecommender(), so shouldnt the first set of experiments be totally random each time?

[AdrianSosic]

Hi @brandon-holt. Yes, so if you use initial_recommender=RandomRecommer() and provide no initial data at the start of the campaign, you should definitely get different results ... 🤔 Do you perhaps have a minimal example for me to reproduce?

[brandon-holt]

@AdrianSosic Okay good to know my understanding is correct.

Sure thing, here is a repro that should show the behavior I'm describing. Currently included are the saved results from the last time I ran it, but if you just open the directory and run repro.ipynb it should overwrite the results and show you the fresh run.

So you should see that for the Bayesian outputs, the results are the same all 5 iterations, whereas for the Control, the outputs are different each time.

[AdrianSosic]

@brandon-holt: Ah, now I see what caused the confusion. In your code, you repeatedly call simulate_experiments inside a loop. However, the way that function is currently designed is that it automatically sets a default seed for the simulation if not specified otherwise. And because you don't explicitly increment the seed in your loop, it always sets the same seed in all of your runs.

Context: the function was designed for the purpose of running a single campaign only. For running several campaigns, we currently have the simulate_scenarios wrapper, whose job it is exactly to take care of setting seeds etc... But now that I see how you intended to use the function (and other people probably will, too), I think it would be reasonable to replace the default seed with None and just use the current system seed in that case.

I can adjust this in the branch that I shared and will ping you once it's there. That should solve your problem 👍🏼

[AdrianSosic]

@brandon-holt: if you do a fresh install now using the command shared above, you should get the desired result

[brandon-holt]

@AdrianSosic
Ahhh gotcha, didn't realize that but thank you for the fix!
Just tested it and it does indeed work now. 🙏