[Bug]: Large sample time increase in ax-platform >= version 0.3.5 about ax

[dme65]

I verified the suspicion above by running your code snippet with the two different acquisition functions:

• With qNoisyExpectedImprovement (the default before #1792) the average sample time is 0.65s.
• With qLogNoisyExpectedImprovement (the default after #1792) the average sample time is 2.06s.

[dme65]

My guess is that this slowdown is coming from #1792, which improved the default acquisition functions in Ax. This is not a bug, but rather a significant improvement.

Before this change, the acquisition functions were much harder to optimize (because of the gradients being numerically zero) so .gen() would sometimes terminate very quickly. After #1792, we are doing a much better job optimizing the acquisition function. While this will result in better optimization performance as Ax generates better candidates, it will also take longer since we will take many more gradient steps when optimizing the acquisition function.

cc @SebastianAment

[esantorella]

Thanks for reporting this. A slowdown definitely wasn't expected behavior. We'll look into this.

[mgrange1998]

Hi, also taking a look through the code changes between 0.3.4 and 0.3.5 here 0.3.4...0.3.5

[mgrange1998]

Haven't identified the source of the regression, but the long running method in the .get_next_trial() call is GenerationStrategy::_gen_multiple, which contains the long running call GenerationNode::gen

I ran the code example on my own, and noticed that as "ax_client.get_next_trial()" was called repeatedly, the run time for _gen_multiple would increase over time. Over 50 calls, its run time increased from 0.03 to 15 seconds.

So far, I haven't been able to find the change in GenerationNode that accounts for the increase between versions 0.3.4 and 0.3.5.

[GilesStrong]

Ah, I see! That does seem like a worthwhile trade-off then.
Many thanks everyone for the quick response and diagnosis. I'll close this issue now.