Benchmarking about tune-sklearn

[anthonyhsyu]

Running on a free machine was definitely helpful, but instead we discovered that early stopping and non-early stopping didn't run differently. This may be due to #11 where early stopping is activated whether the estimator supports early stopping or not, although we don't seem to find evidence that FIFO early stops, and it performs similarly or even better than MedianStoppingRule. It's possible that the median was not a good metric since all configs tested yielded similar results, so it couldn't early stop. More needs to be looked before closing this, but it's most likely due to the early stopping based on whether the estimator supports it or not, introduced in #11.

[richardliaw]

Make sure we check accuracy

[anthonyhsyu]

These are confirmed to be accurate and have been compared to Dask. Even with significant speed up in fit times from Tune via early stopping, it is able to correctly align good and bad hyperparameter configurations.

Below is a plot detailing the average test set score. We want the blue and orange bars for each configuration (along the x-axis) to be of equal height, or if it is a bad parameter, both bars having significantly lower heights. This behavior is achieved below.