What should my evaluation function return for unknown SEM greater than 0? about ax

[bletham]

All of the models currently built-in to Ax do assume known observation noise SEM. To infer the noise level, you'll have to plug-in a separate Botorch model. You can do this by following the directions here: https://botorch.org/tutorials/custom_botorch_model_in_ax

In that tutorial, instead of defining SimpleCustomGP you could replace that with

from botorch.models.gp_regression import SingleTaskGP

and then everywhere in the tutorial where it uses SimpleCustomGP, just put in SingleTaskGP instead.

If you follow all of the steps in that tutorial, you will be using a model that infers the noise level. You can then return whatever you want as the SEM in the evaluation function (0 for instance), and it will be ignored by the model.

(The only alternative to this that I can think of would be to do a few function evaluations at a single point to try and estimate the noise level directly and then use that as SEM).

[72496385]

Ah, thank you.

[bletham]

This is something that is in-progress, so I'll leave this issue open until it is resolved.

[lena-kashtelyan]

There is now a fix for this on the latest stable version (0.1.5); if you just do not pass SEM, it will no longer be assumed to be 0. @72496385, which API were you using for your optimization? Let me know if you'd like me to show an example of what it would look like to just not provide SEM to it.

[lena-kashtelyan]

Closing this issue, feel free to reopen if help is needed with how to not provide the SEM!

[tqfjo]

Thanks a bunch for this feature.

feel free to reopen if help is needed with how to not provide the SEM!

Just a note that it took me a while to figure out that I needed to explicitly pass None as the SEM. (I first tried passing the evaluation result alone, then the evaluation result alone in a tuple.)