Comments (8)
@aagarwal1999, yep, there is –– you will need to make a custom generation strategy. You can get one by resetting it right after your experiment:
ax_client = AxClient()
ax_client.create_experiment(...)
ax_client.generation_strategy = choose_generation_strategy(
search_space=ax_client.experiment.search_space,
random_seed=239, # Whatever random seed you want.
)
You can import choose_generation_strategy
from here:
Ax/ax/service/utils/dispatch.py
Line 18 in 7d34dd3
Then, you will need to set the torch seed before each call to ax_client.get_next_trial
, like so: torch.manual_seed(1000)
. More on the torch seed in this issue: #95.
Let me know if that fully solves the issue.
from ax.
I am getting an error stating:
TypeError: choose_generation_strategy() got an unexpected keyword argument 'random_seed'
I suspect this is due to the fact that I am not using the latest version of Ax. However, when I updated to the latest version, I was getting an import error when I try and import AxClient
File "/mnt/nfs/home/ankit/.conda/envs/dstorch-ankit/lib/python3.6/site-packages/ax/models/torch/botorch_defaults.py", line 21, in <module> from botorch.optim.optimize import optimize_acqf ImportError: cannot import name 'optimize_acqf'
from ax.
@aagarwal1999, you're right that the argument was unexpected because you had a later version; the BoTorch defaults import error, however, is news to me. Did you try to update BoTorch also?
from ax.
That fixed it! Thanks
from ax.
@aagarwal1999, also, I didn't know this when I responded to your question, but setting the random seed this say in the long run is non-ideal as it sets the seed for torch globally, which may impact model performance. I will have a fix for this on master by tomorrow –– will add a random_seed
top-level argument to AxClient
instantiation, which should make the seed safer to use.
from ax.
Hi, has the fix been implemented for the AxClient seed?
from ax.
Hi, Ankit! Not yet, but a partial fix will be on master soon (possibly today; I will update this issue like I said I would). Partial, because there isn't a way to fully make the Bayesian optimization part of the process reproducible (that is safe; I learned that setting the torch.manual_seed
is not safe for the model performance). When we fix the torch seed, it's fixed globally, which can have drastic negative effects on the performance of the BoTorch-backed models, which we use for Bayesian optimization (that's what we are doing through the ad-hoc solution you are using now).
Now, the partial fix will provide a random seed setting to AxClient, which will make the initial quasi-random trials fully reproducible and the subsequent trials somewhat reproducible (but as you generate more trials, they will start differing more and more).
Finally, if you want reproducibility because you want your Ax optimization to use data from previous optimization, you can attach those trials via attach_trial
and the data will be used in optimization. However, do make sure that those trials are actually applicable to the new optimization (for example, if you evaluate the trials on some data that has non-stationarity, then attaching data from an old optimization will worsen your new optimization results).
Let me know if that helps / makes sense!
from ax.
Closing this issue because the random_seed
argument for AxClient
is now included into the latest stable version –– 0.1.5. Note that it does not make optimization fully reproducible, as described above.
from ax.
Related Issues (20)
- Safe optimization in the Service API HOT 5
- The same point is evaluated multiple times during Integer Optimization with BO. HOT 5
- ax_client.generation_strategy.trials_as_df HOT 9
- Managing Objective Function Evaluation Failures in Ax for MOO HOT 6
- [GENERAL SUPPORT]: Managing Objective Function Evaluation Failures in Ax for MOO HOT 3
- [GENERAL SUPPORT]: Using qNegIntegratedPosteriorVariance HOT 3
- [GENERAL SUPPORT]: Reference point for multi-objective bayesian optimization HOT 4
- [GENERAL SUPPORT]: Adjusting search space or accommodating out-of-bounds initial data HOT 19
- [GENERAL SUPPORT]: Manual configuration, HOT 1
- [Bug]: Custom metric issue HOT 4
- [GENERAL SUPPORT]: CI_Level Paretofrontier
- [Bug]: Large sample time increase in ax-platform >= version 0.3.5 HOT 6
- [GENERAL SUPPORT]: Reference Point for Multi-Objective Bayesian Optimization HOT 5
- [GENERAL SUPPORT]: Plotting Pareto fronts / Posterior mean model HOT 6
- [GENERAL SUPPORT]: Getting best predicted point of a botorch model HOT 10
- [GENERAL SUPPORT]: Logical-or in outcome constraints HOT 4
- [GENERAL SUPPORT]: Parallelism and arbitrary parameter type support. HOT 5
- [GENERAL SUPPORT]: How to Standardize and Normalize Data for Service API HOT 6
- [FEATURE REQUEST]: Implement ImprovementGlobalStoppingStrategy for batch trials HOT 2
- [GENERAL SUPPORT]: Plot observed pareto frontier alongside remaining Arms HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ax.