[Enhancement] Smart error handling and hyperparameter resampling about tune-sklearn

[richardliaw]

Thanks for opening this @rohan-gt! Would it be OK if we proceeded with tuning without resampling?

[rohan-gt]

@richardliaw in TuneSearchCV n_iter is the total number of trials and max_iters the early stopping for the same right? Without resampling won't the trails reduce drastically especially in a case like if 7/10 or even 10/10 cases error out? Like is there a way to keep the number of trials consistent?

[richardliaw]

Hmm... right. Would conditional search spaces (like HyperOpt or Optuna) work for you?

[rohan-gt]

@richardliaw Yes that could actually work. Btw why was skopt chosen over optuna for this implementation? Isn't optuna superior?

[richardliaw]

IIRC Tree-Parzen Estimators (which is the algorithm that optuna implements) shouldn't be much different to standard Bayesian Optimization (that skopt provides).

[rohan-gt]

@richardliaw Yes but scikit-optimize isn't an active project and throws errors with the latest scikit-learn. Also on second thought it would be a much cleaner implementation if it was possible to sample new hyperparameter combinations if existing ones error out than having to specify so many conditional search spaces

[richardliaw]

I think this is addressed with hyperopt thanks to #68!