[Question] Tune Sklearn Algorithm about tune-sklearn

[Yard1]

I can answer some of those questions:
3 - AFAIK max_iters is the hard limit of iterations, but the early stopping algorithm may choose to end it earlier.
4 - try setting verbosity to 2 and see what's being printed out. For me, the number of trials was equal to what I set.

[rohan-gt]

@Yard1 ,
3 - If max_iters is the hard limit, what is n_iter used for?
4 - I did set verbosity=2 and noticed that only 10 trials are run even when I set both to 50

[Yard1]

3 - they are two different things. max_iters is the number of iteration per trial, and n_iter is the number of trials. So the maximum amount of total iterations is max_iters * n_iter.

[inventormc]

1. I'm actually not entirely sure how this is supposed to behave. Maybe @richardliaw can answer this one better?
2. n_iter=10 means you sample 10 hyperparameters from the hyperparameter space, and cv=3 means each of the 10 models will be cross validated using 3 fold cross validation. The way we figure out how to early stop at the moment is just to get the average performance across all folds, as this is how cross_validation is generally done. I'm not sure it's safe to conclude the model is bad just because it does worse on one fold.
3. @Yard1 gave a good answer to this so I don't need to add much here.
4. Did you check the cv_results_ dictionary?
5. n_jobs is used to figure out how many trainables can be run in parallel. So if you specify it to be -1, it will do the maximum number of parallel jobs, using 1 core per job. sk_n_jobs is just used to set the n_jobs parameter of the underlying sklearn estimator. This defaults to -1 to tell sklearn to use all the cores available to it, and can usually be ignored unless you run into errors. I'm not sure why it's using 1/2 a core, but it'd be helpful to have more information/output. Are there 2 cores total on your machine?

[richardliaw]

@rohan-gt these are great questions, I've pushed a PR to address them.

1. great catch, #81 should fix it up and you'll see the expected behavior.
2. I've renamed n_iter -> n_trials in #81.

3. max_iters + early_stopping => ASHA + absolute limit of max_iters. @Yard1 gave a great explanation.
4. Maybe an artifact of BOHB. I've pushed some changes and will ping you to try it out on master.
5. n_jobs=-1 should imply that there are 2 parallel trials running. Did you set n_iter=1? If not, could you post a new issue about how to repro?

Thanks a bunch for trying things out and asking questions - we really want to make you successful!