Efficient sampling of measurment bound functions: BatchExecutor? about adaptive

[akhmerov]

This is an excellent question, but also extremely tough!

First of all, I am not sure what a good algorithm or even a clear problem setting might be. Depending on how costly it is to vary the parameters, and the dimensionality of the parameter space, the optimal strategy may vary dramatically.

From the API standpoint, I believe that planning the queries should be the responsibility of the Learner and not the Executor, so that it can plan ahead. A possible implementation would keep track of the last supplied point, and reweigh the losses based on distance from that. A batched strategy with pruning is another similar option.

[terrorfisch]

I agree that a general optimial solution is very hard to find if you want to include a "manipulation cost" into the learner's strategy. Especially, if the dimensionality is high.

An easy and clean way to solve this problem might be to keep the original strategy but use the "free" function evaluations that are avaiable on the path between the requested evaluation points. As far as I understand one can already use Learner.tell for non-requested points. We decided to not go this path because we wanted the Runners convinience functionalities like plotting.

I agree that the proper place is not in the Executor but maybe this can be best integrated in the Runner. At the and it is a known property of the function "evaluation method". I think the relevant properties of function evaluation are:

1. Can the function be evaluated concurrently?
2. Does the evaluation cost depend on the point itself?
3. If the function cannot be evaluated concurrently: does the function evaluation cost depend on the previously evaluated point?