Comments (2)
Hey Tonmoy,
sorry for the delay.
I am not sure what kinds of exceptions you want to handle. You can always put a try-except statement into your worker's compute method and repeat the computation yourself.
Catching network connectivity errors and retrying is a bit more tricky. Do you see those a lot, because I have not experiences those ever, but AFAIK, you run really long jobs, right?
from hpbandster.
@sfalkner I would like to use BOHB for long running training jobs as well, on preemptible instances. I can't imagine that all workers will survive the whole optimization process. It would definitely be important to retry failed configurations at least a couple of times, and the high-budget jobs are especially likely to fail at some point due to longer running times.
Come to think of it, the way to handle this best would probably be to have the Worker save periodic checkpoints to cloud storage, and be able to re-create the Worker if it fails and resume from that checkpoint, until it completes its allotted budget successfully.
Of course, this means that TCP socket connectivity is NOT the way to find out that a Worker is permanently dead - you have to handle the situation where the same job is now resumed by a new Worker instance. So instead of sockets, it should be persistent event queues or DB entries or something.
In effect, I guess this means getting rid of the NameServer as it is now and replacing it with a persistent cloud data structure to represent current state, which can be written and read by Workers and the Master.
from hpbandster.
Related Issues (20)
- A bunch of errors when using categoricals
- Interferance between two BOHB instances when they have the same networking parameters? HOT 2
- Why get_incumbent_id selects the incumbent only from runs with max_budget? HOT 1
- How to stop a worker
- How to plot pictures like your work
- Support scipy 1.3+ HOT 1
- Download code button for keras example is not working
- Missing Optimizer
- Minor documentation fixes
- Machine precision problem in determening s_max
- Fit reinforcement learning model with hyperparameter optimization method
- Hyperband bracket generation is inconsistent with the original Hyperband paper HOT 1
- Continuing from running halfway
- Warm start: results.json and configs.json (from live json_result_logger) do not contain results from previous runs HOT 3
- "Ideally, the optimizer would adjust the budgets online" -- when will it be implemented and using what method?
- Issue in case of warmstarting case
- Saving optimization process HOT 1
- conda package? HOT 2
- License file for conda package
- some problem HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hpbandster.