Comments (5)
Can you just reparameterize your problem in this case? Like define integer parameters size_per_head
and log_batch_size
and then in your evaluation code have
hidden_size = size_per_head * num_attention_heads
batch_size = 2**log_batch_size
Or would that mean you need a joint constraint on size_per_head
and num_attention_heads
?
@sdsingh From an optimization/candidate generation perspective (other than difficulty of the problem) there is no issue with imposing non-linear constraints on the parameter space. I assume the linearity assumption is mainly imposing structure for representing the constraints in Ax?
from ax.
All parameter constraints are implemented as linear constraints in the modeling layer. When we pass points into our GPs, they are all normalized to [0,1]^d, where then our constraints are applied. You can see how these are implemented w/ a simple matrix multiply in the Botorch model. As a result, we can only support constraints that can be mapped into a linear constraint, however creatively that may be. Unfortunately, I don't see a way of doing that for either of these constraint types.
Both of these constraint types seem pretty useful, though. Let me think about what we can do here, but unfortunately I don't think this will be a quick fix.
from ax.
Thank you, that's indeed a great alternative way.
I just thought we can make it elegant, but after I read the explanation, I knew that's not easy as I thought.
But stiil a very useful library to tune, thanks for your works !!
from ax.
Closing this issue for now as there is an easy workaround and adding modulo constraints is not on our roadmap for the foreseeable future.
from ax.
Can you just reparameterize your problem in this case? Like define integer parameters
size_per_head
andlog_batch_size
and then in your evaluation code have
hidden_size = size_per_head * num_attention_heads
batch_size = 2**log_batch_size
Or would that mean you need a joint constraint on
size_per_head
andnum_attention_heads
?@sdsingh From an optimization/candidate generation perspective (other than difficulty of the problem) there is no issue with imposing non-linear constraints on the parameter space. I assume the linearity assumption is mainly imposing structure for representing the constraints in Ax?
Passing comment on an old thread: similarly, could reparameterize as hidden_size_multiplier
which reflects the same constraint and search space if I'm not mistaken. Also, there is some support for non-linear constraints in Ax now #769, though not sure if modulus is a differentiable function.
from ax.
Related Issues (20)
- "Hyperparameter Optimization via Raytune" link in website is broken. HOT 2
- Using `evaluate_acquisition_function` on `AxClient` causes subsequent optimziation errors HOT 2
- Question : modifications of compute_posterior_pareto_frontier HOT 2
- Tracking of auxiliary metrics HOT 2
- `qMaxValueEntropy` doesn't seem to work with `ObjectiveProperties(minimize=True)` HOT 4
- when should we end the Bandit Optimization HOT 4
- Nontrivial parameter constraints HOT 2
- Can't control arguments in fit_gpytorch_mll under the hood. Getting ABNORMAL_TERMINATION_IN_LNSRCH warning HOT 1
- Different Errors when initializing my loop with Service API and Developer API HOT 7
- `_random_seed` not retained when using `ax_client.save_to_json_file()` and `AxClient.load_from_json_file()` HOT 2
- Question: SEBO optimization with parameter dependency | logistic parameter constrains HOT 4
- Tutorial Request: Deploying Runners on Clusters, Debugging Runners/Schedulers HOT 4
- Issue with tolerance for floating point and its relevance when using log_scale = True HOT 7
- Question: does Ax support working with Tensorflow models? HOT 2
- Feature Request: Conditional Parameter Constraints HOT 5
- Questions about define how to evaluate HOT 3
- get_countour_plot() not plotting all trials HOT 4
- Error: A list of 'ChoiceParameter' is not iterable HOT 4
- [Bug] Generation Strategy equality check error without call to repr HOT 1
- ax.dev, GitHub readme, and loop tutorial examples are ignoring `minimize` kwarg HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ax.