Greetings,
I tried to run the example from the README.md. All steps run without errors, until I try to score the test set. Then I get a ValueError: No models fitted!. The console output shows various runtime warnings, among them indications about missing files. What could be the cause?
(Unfortunately, github promptly refuses to accept a text file for some reason, so I pasted the console output during fitting and the error trace when trying to score below.)
My setup is a Python 2.7.6 virtualenv, running from IPython 4.0.0. The installed packages are as follows:
argparse (1.2.1)
AutoSklearn (0.0.1.dev0)
cma (1.1.06)
decorator (4.0.2)
funcsigs (0.4)
HPOlib (0.1.0)
HPOlibConfigSpace (0.1dev)
ipython (4.0.0)
ipython-genutils (0.1.0)
liac-arff (2.1.0)
lockfile (0.10.2)
matplotlib (1.4.3)
mock (1.3.0)
networkx (1.10)
nose (1.3.7)
numpy (1.9.0)
pandas (0.16.2)
ParamSklearn (0.1dev)
path.py (8.1.1)
pbr (1.8.0)
pexpect (3.3)
pickleshare (0.5)
pip (1.5.4)
protobuf (3.0.0-alpha-1)
psutil (3.2.1)
pyMetaLearn (0.1dev)
pymongo (3.0.3)
pyparsing (2.0.3)
python-dateutil (2.4.2)
pytz (2015.6)
PyYAML (3.11)
scikit-learn (0.15.2)
scipy (0.14.0)
setuptools (18.3.2)
simplegeneric (0.8.1)
six (1.9.0)
traitlets (4.0.0)
wheel (0.24.0)
wsgiref (0.1.2)
Thanks for your response.
Console output:
[INFO] [09-24 17:58:47:AutoML_54da6690e2c896d2d9aafe349b066645_1]: Remaining time after reading 54da6690e2c896d2d9aafe349b066645 3600.00 sec
/media/selects/venv/py27/local/lib/python2.7/site-packages/numpy/lib/nanfunctions.py:1057: RuntimeWarning: Degrees of freedom <= 0 for slice.
warnings.warn("Degrees of freedom <= 0 for slice.", RuntimeWarning)
/media/selects/venv/py27/local/lib/python2.7/site-packages/numpy/lib/nanfunctions.py:598: RuntimeWarning: Mean of empty slice
warnings.warn("Mean of empty slice", RuntimeWarning)
[WARNING] [09-24 17:58:47:pyMetaLearn.input.aslib_simple]: Not found: /media/selects/venv/py27/local/lib/python2.7/site-packages/AutoSklearn-0.0.1.dev0-py2.7-linux-x86_64.egg/autosklearn/metalearni
ng/files/multiclass.classification_dense_acc_metric/ground_truth.arff (maybe you want to add it)
[WARNING] [09-24 17:58:47:pyMetaLearn.input.aslib_simple]: Not found: /media/selects/venv/py27/local/lib/python2.7/site-packages/AutoSklearn-0.0.1.dev0-py2.7-linux-x86_64.egg/autosklearn/metalearni
ng/files/multiclass.classification_dense_acc_metric/citation.bib (maybe you want to add it)
[WARNING] [09-24 17:58:47:pyMetaLearn.input.aslib_simple]: Not found: /media/selects/venv/py27/local/lib/python2.7/site-packages/AutoSklearn-0.0.1.dev0-py2.7-linux-x86_64.egg/autosklearn/metalearni
ng/files/multiclass.classification_dense_acc_metric/cv.arff (maybe you want to add it)
[INFO] [09-24 17:58:48:autosklearn.metalearning.metalearning]: Reading meta-data took 0.59 seconds
['133', '132', '131', '130', '137', '136', '135', '134', '139', '138', '24', '25', '26', '27', '20', '21', '22', '23', '28', '29', '4', '8', '120', '121', '122', '123', '124', '125', '126', '127',
'128', '129', '59', '58', '55', '54', '57', '56', '51', '50', '53', '52', '115', '114', '88', '89', '111', '110', '113', '112', '82', '83', '80', '81', '119', '118', '84', '85', '3', '7', '108', '1
09', '102', '103', '100', '101', '106', '107', '104', '105', '39', '38', '33', '32', '31', '30', '37', '36', '35', '34', '60', '61', '62', '63', '64', '65', '66', '67', '68', '69', '2', '6', '99',
'98', '91', '90', '93', '92', '95', '94', '97', '96', '11', '10', '13', '12', '15', '14', '17', '16', '19', '18', '117', '116', '41', '48', '49', '46', '86', '44', '45', '42', '43', '40', '87', '1'
, '5', '9', '142', '140', '141', '77', '76', '75', '74', '73', '72', '71', '70', '79', '78', '47']
[WARNING] [09-24 17:58:48:pyMetaLearn.optimizers.metalearn_optimizer.metalearner]: Could not find runs for instance 1118_bac
[WARNING] [09-24 17:58:48:pyMetaLearn.optimizers.metalearn_optimizer.metalearner]: Could not find runs for instance 314_bac
[WARNING] [09-24 17:58:48:pyMetaLearn.optimizers.metalearn_optimizer.metalearner]: Could not find runs for instance 454_bac
[WARNING] [09-24 17:58:48:pyMetaLearn.optimizers.metalearn_optimizer.metalearner]: Could not find runs for instance 809_bac
[WARNING] [09-24 17:58:48:pyMetaLearn.optimizers.metalearn_optimizer.metalearner]: Could not find runs for instance 948_bac
[WARNING] [09-24 17:58:48:pyMetaLearn.metalearning.kNearestDatasets.kND]: Found no best configuration for instance 1118_bac
[WARNING] [09-24 17:58:48:pyMetaLearn.metalearning.kNearestDatasets.kND]: Found no best configuration for instance 948_bac
[WARNING] [09-24 17:58:48:pyMetaLearn.metalearning.kNearestDatasets.kND]: Found no best configuration for instance 454_bac
[WARNING] [09-24 17:58:48:pyMetaLearn.metalearning.kNearestDatasets.kND]: Found no best configuration for instance 809_bac
[WARNING] [09-24 17:58:48:pyMetaLearn.metalearning.kNearestDatasets.kND]: Found no best configuration for instance 314_bac
[INFO] [09-24 17:58:48:AutoML_54da6690e2c896d2d9aafe349b066645_1]: Time left for 54da6690e2c896d2d9aafe349b066645 after finding initial configurations: 3598.94sec
Calling: smac --numRun 1 --scenario /tmp/autosklearn_tmp_14167_1103/54da6690e2c896d2d9aafe349b066645.scenario --initial-challengers " -balancing:strategy 'weighting' -classifier 'lda' -imputation:s
trategy 'median' -kernel_pca:gamma '0.0290194572424' -kernel_pca:kernel 'rbf' -kernel_pca:n_components '1971' -lda:n_components '232' -lda:tol '0.000804876897084' -preprocessor 'kernel_pca' -rescal
ing:strategy 'min/max'" --initial-challengers " -balancing:strategy 'weighting' -classifier 'libsvm_svc' -imputation:strategy 'median' -liblinear_svc_preprocessor:C '18592.5543358' -liblinear_svc_p
reprocessor:class_weight 'auto' -liblinear_svc_preprocessor:dual 'False' -liblinear_svc_preprocessor:fit_intercept 'True' -liblinear_svc_preprocessor:intercept_scaling '1' -liblinear_svc_preprocess
or:loss 'l2' -liblinear_svc_preprocessor:multi_class 'ovr' -liblinear_svc_preprocessor:penalty 'l2' -liblinear_svc_preprocessor:tol '0.040232270855' -libsvm_svc:C '6111.7121149' -libsvm_svc:class_w
eight 'None' -libsvm_svc:coef0 '0.844884936773' -libsvm_svc:degree '5' -libsvm_svc:gamma '0.117882960246' -libsvm_svc:kernel 'poly' -libsvm_svc:max_iter '-1' -libsvm_svc:shrinking 'False' -libsvm_s
vc:tol '0.00109298090501' -preprocessor 'liblinear_svc_preprocessor' -rescaling:strategy 'min/max'" --initial-challengers " -balancing:strategy 'weighting' -classifier 'liblinear_svc' -imputation:s
trategy 'mean' -kernel_pca:gamma '1.6331524928' -kernel_pca:kernel 'rbf' -kernel_pca:n_components '761' -liblinear_svc:C '44.5016816038' -liblinear_svc:class_weight 'auto' -liblinear_svc:dual 'Fals
e' -liblinear_svc:fit_intercept 'True' -liblinear_svc:intercept_scaling '1' -liblinear_svc:loss 'l2' -liblinear_svc:multi_class 'ovr' -liblinear_svc:penalty 'l2' -liblinear_svc:tol '0.0018788986680
6' -preprocessor 'kernel_pca' -rescaling:strategy 'normalize'" --initial-challengers " -balancing:strategy 'weighting' -classifier 'libsvm_svc' -imputation:strategy 'mean' -libsvm_svc:C '50.8707992
587' -libsvm_svc:class_weight 'auto' -libsvm_svc:gamma '4.72168867253' -libsvm_svc:kernel 'rbf' -libsvm_svc:max_iter '-1' -libsvm_svc:shrinking 'True' -libsvm_svc:tol '1.67692533041e-05' -preproces
sor 'select_rates' -rescaling:strategy 'normalize' -select_rates:alpha '0.318343160914' -select_rates:mode 'fdr' -select_rates:score_func 'f_classif'" --initial-challengers " -balancing:strategy 'n
one' -classifier 'ridge' -imputation:strategy 'median' -kernel_pca:gamma '2.43149422021' -kernel_pca:kernel 'rbf' -kernel_pca:n_components '1194' -preprocessor 'kernel_pca' -rescaling:strategy 'nor
malize' -ridge:alpha '1.30657587648e-05' -ridge:fit_intercept 'True' -ridge:tol '0.000760986834404'" --initial-challengers " -adaboost:algorithm 'SAMME.R' -adaboost:learning_rate '0.400363929326' -
adaboost:max_depth '5' -adaboost:n_estimators '319' -balancing:strategy 'none' -classifier 'adaboost' -imputation:strategy 'most_frequent' -preprocessor 'no_preprocessing' -rescaling:strategy 'min/
max'" --initial-challengers " -balancing:strategy 'none' -classifier 'qda' -imputation:strategy 'mean' -pca:keep_variance '0.748479656855' -pca:whiten 'False' -preprocessor 'pca' -qda:reg_param '3.
82874880102' -qda:tol '0.0130621640728' -rescaling:strategy 'normalize'" --initial-challengers " -balancing:strategy 'weighting' -classifier 'libsvm_svc' -imputation:strategy 'mean' -libsvm_svc:C '
18807.7593252' -libsvm_svc:class_weight 'None' -libsvm_svc:gamma '0.940704535703' -libsvm_svc:kernel 'rbf' -libsvm_svc:max_iter '-1' -libsvm_svc:shrinking 'True' -libsvm_svc:tol '0.00148731196993'
-preprocessor 'select_rates' -rescaling:strategy 'min/max' -select_rates:alpha '0.126666738937' -select_rates:mode 'fdr' -select_rates:score_func 'f_classif'" --initial-challengers " -balancing:str
ategy 'weighting' -classifier 'lda' -imputation:strategy 'mean' -kitchen_sinks:gamma '1.48108179896' -kitchen_sinks:n_components '3450' -lda:n_components '25' -lda:tol '0.0426553560955' -preprocess
or 'kitchen_sinks' -rescaling:strategy 'min/max'" --initial-challengers " -balancing:strategy 'weighting' -classifier 'random_forest' -feature_agglomeration:affinity 'manhattan' -feature_agglomerat
ion:linkage 'average' -feature_agglomeration:n_clusters '76' -imputation:strategy 'median' -preprocessor 'feature_agglomeration' -random_forest:bootstrap 'True' -random_forest:criterion 'entropy' -
random_forest:max_depth 'None' -random_forest:max_features '1.60908385606' -random_forest:max_leaf_nodes 'None' -random_forest:min_samples_leaf '2' -random_forest:min_samples_split '12' -random_for
est:n_estimators '100' -rescaling:strategy 'min/max'" --initial-challengers " -balancing:strategy 'none' -classifier 'ridge' -imputation:strategy 'mean' -nystroem_sampler:coef0 '0.476829591723' -ny
stroem_sampler:degree '3' -nystroem_sampler:gamma '0.0817500204362' -nystroem_sampler:kernel 'poly' -nystroem_sampler:n_components '7840' -preprocessor 'nystroem_sampler' -rescaling:strategy 'min/m
ax' -ridge:alpha '3.52478796331e-06' -ridge:fit_intercept 'True' -ridge:tol '2.63925768895e-05'" --initial-challengers " -balancing:strategy 'none' -classifier 'random_forest' -imputation:strategy
'mean' -preprocessor 'no_preprocessing' -random_forest:bootstrap 'True' -random_forest:criterion 'gini' -random_forest:max_depth 'None' -random_forest:max_features '1.0' -random_forest:max_leaf_nod
es 'None' -random_forest:min_samples_leaf '1' -random_forest:min_samples_split '2' -random_forest:n_estimators '100' -rescaling:strategy 'min/max'" --initial-challengers " -balancing:strategy 'none
' -classifier 'lda' -imputation:strategy 'median' -lda:n_components '203' -lda:tol '0.0935342136025' -preprocessor 'select_rates' -rescaling:strategy 'normalize' -select_rates:alpha '0.048178281695
5' -select_rates:mode 'fwe' -select_rates:score_func 'f_classif'" --initial-challengers " -balancing:strategy 'none' -classifier 'sgd' -imputation:strategy 'mean' -preprocessor 'no_preprocessing' -
rescaling:strategy 'min/max' -sgd:alpha '0.0001' -sgd:eta0 '0.01' -sgd:fit_intercept 'True' -sgd:learning_rate 'optimal' -sgd:loss 'hinge' -sgd:n_iter '20' -sgd:penalty 'l2'" --initial-challengers
" -balancing:strategy 'weighting' -classifier 'liblinear_svc' -imputation:strategy 'mean' -kitchen_sinks:gamma '1.62106650658' -kitchen_sinks:n_components '6034' -liblinear_svc:C '780.976275468' -l
iblinear_svc:class_weight 'auto' -liblinear_svc:dual 'False' -liblinear_svc:fit_intercept 'True' -liblinear_svc:intercept_scaling '1' -liblinear_svc:loss 'l2' -liblinear_svc:multi_class 'ovr' -libl
inear_svc:penalty 'l2' -liblinear_svc:tol '2.60869016302e-05' -preprocessor 'kitchen_sinks' -rescaling:strategy 'min/max'" --initial-challengers " -balancing:strategy 'weighting' -classifier 'libsv
m_svc' -feature_agglomeration:affinity 'manhattan' -feature_agglomeration:linkage 'average' -feature_agglomeration:n_clusters '89' -imputation:strategy 'most_frequent' -libsvm_svc:C '246.452178174'
-libsvm_svc:class_weight 'auto' -libsvm_svc:gamma '0.0442300193285' -libsvm_svc:kernel 'rbf' -libsvm_svc:max_iter '-1' -libsvm_svc:shrinking 'False' -libsvm_svc:tol '0.0180487670379' -preprocessor
'feature_agglomeration' -rescaling:strategy 'standard'" --initial-challengers " -balancing:strategy 'weighting' -classifier 'passive_aggresive' -imputation:strategy 'median' -passive_aggresive:C '
1.31125616578' -passive_aggresive:fit_intercept 'True' -passive_aggresive:loss 'hinge' -passive_aggresive:n_iter '948' -preprocessor 'select_percentile_classification' -rescaling:strategy 'min/max'
-select_percentile_classification:percentile '83.3669247487' -select_percentile_classification:score_func 'chi2'" --initial-challengers " -balancing:strategy 'none' -classifier 'sgd' -imputation:s
trategy 'most_frequent' -preprocessor 'no_preprocessing' -rescaling:strategy 'min/max' -sgd:alpha '0.00292211727831' -sgd:epsilon '0.0116887099622' -sgd:eta0 '0.080560671307' -sgd:fit_intercept 'Tr
ue' -sgd:learning_rate 'invscaling' -sgd:loss 'modified_huber' -sgd:n_iter '754' -sgd:penalty 'l1' -sgd:power_t '0.463498329665'" --initial-challengers " -balancing:strategy 'none' -classifier 'ran
dom_forest' -imputation:strategy 'mean' -preprocessor 'select_rates' -random_forest:bootstrap 'False' -random_forest:criterion 'entropy' -random_forest:max_depth 'None' -random_forest:max_features
'4.67839426105' -random_forest:max_leaf_nodes 'None' -random_forest:min_samples_leaf '10' -random_forest:min_samples_split '10' -random_forest:n_estimators '100' -rescaling:strategy 'standard' -sel
ect_rates:alpha '0.167486470473' -select_rates:mode 'fdr' -select_rates:score_func 'f_classif'" --initial-challengers " -balancing:strategy 'none' -classifier 'sgd' -imputation:strategy 'most_frequ
ent' -preprocessor 'select_rates' -rescaling:strategy 'min/max' -select_rates:alpha '0.155334914856' -select_rates:mode 'fpr' -select_rates:score_func 'f_classif' -sgd:alpha '6.49185336268e-05' -sg
d:eta0 '0.0665593974375' -sgd:fit_intercept 'True' -sgd:learning_rate 'optimal' -sgd:loss 'log' -sgd:n_iter '189' -sgd:penalty 'l2'" --initial-challengers " -balancing:strategy 'weighting' -classif
ier 'sgd' -imputation:strategy 'median' -preprocessor 'no_preprocessing' -rescaling:strategy 'min/max' -sgd:alpha '0.000134377776157' -sgd:epsilon '0.000256156800074' -sgd:eta0 '0.05222815237' -sgd
:fit_intercept 'True' -sgd:learning_rate 'constant' -sgd:loss 'modified_huber' -sgd:n_iter '429' -sgd:penalty 'l1'" --initial-challengers " -balancing:strategy 'weighting' -classifier 'passive_aggr
esive' -imputation:strategy 'median' -liblinear_svc_preprocessor:C '0.306520222754' -liblinear_svc_preprocessor:class_weight 'None' -liblinear_svc_preprocessor:dual 'False' -liblinear_svc_preproces
sor:fit_intercept 'True' -liblinear_svc_preprocessor:intercept_scaling '1' -liblinear_svc_preprocessor:loss 'l2' -liblinear_svc_preprocessor:multi_class 'ovr' -liblinear_svc_preprocessor:penalty 'l
2' -liblinear_svc_preprocessor:tol '4.83193374386e-05' -passive_aggresive:C '0.000522592495213' -passive_aggresive:fit_intercept 'True' -passive_aggresive:loss 'hinge' -passive_aggresive:n_iter '31
3' -preprocessor 'liblinear_svc_preprocessor' -rescaling:strategy 'min/max'" --initial-challengers " -balancing:strategy 'none' -classifier 'libsvm_svc' -imputation:strategy 'median' -libsvm_svc:C
'19690.0557441' -libsvm_svc:class_weight 'None' -libsvm_svc:gamma '4.89593584562e-05' -libsvm_svc:kernel 'rbf' -libsvm_svc:max_iter '-1' -libsvm_svc:shrinking 'True' -libsvm_svc:tol '0.019646836528
3' -preprocessor 'random_trees_embedding' -random_trees_embedding:max_depth '4' -random_trees_embedding:max_leaf_nodes 'None' -random_trees_embedding:min_samples_leaf '16' -random_trees_embedding:m
in_samples_split '9' -random_trees_embedding:n_estimators '52' -rescaling:strategy 'standard'" --initial-challengers " -balancing:strategy 'none' -classifier 'sgd' -imputation:strategy 'mean' -prep
rocessor 'no_preprocessing' -rescaling:strategy 'min/max' -sgd:alpha '0.0001' -sgd:eta0 '0.01' -sgd:fit_intercept 'True' -sgd:learning_rate 'optimal' -sgd:loss 'hinge' -sgd:n_iter '20' -sgd:penalty
'l2'" --initial-challengers " -balancing:strategy 'weighting' -classifier 'random_forest' -extra_trees_preproc_for_classification:bootstrap 'True' -extra_trees_preproc_for_classification:criterion
'entropy' -extra_trees_preproc_for_classification:max_depth 'None' -extra_trees_preproc_for_classification:max_features '3.61796566599' -extra_trees_preproc_for_classification:min_samples_leaf '6'
-extra_trees_preproc_for_classification:min_samples_split '2' -extra_trees_preproc_for_classification:n_estimators '100' -imputation:strategy 'mean' -preprocessor 'extra_trees_preproc_for_classifi
cation' -random_forest:bootstrap 'True' -random_forest:criterion 'entropy' -random_forest:max_depth 'None' -random_forest:max_features '0.857466092817' -random_forest:max_leaf_nodes 'None' -random_
forest:min_samples_leaf '14' -random_forest:min_samples_split '15' -random_forest:n_estimators '100' -rescaling:strategy 'normalize'"
Calling: runsolver --watcher-data /dev/null -W 3598 -d 5 python /media/selects/venv/py27/local/lib/python2.7/site-packages/AutoSklearn-0.0.1.dev0-py2.7-linux-x86_64.egg/autosklearn/ensemble_selecti
on_script.py /tmp/autosklearn_tmp_14167_1103 54da6690e2c896d2d9aafe349b066645 multiclass.classification acc_metric 3593.92797899 /tmp/autosklearn_output_14167_1103 50 1 /tmp/autosklearn_tmp_14167_1
103/ensemble_indices_1
Out[16]: <AutoSklearnClassifier(AutoSklearnClassifier-1, initial)>
In [17]: >>> print automl.score(X_test, y_test)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-17-6a5d99e9c9c3> in <module>()
----> 1 print automl.score(X_test, y_test)
/media/selects/venv/py27/local/lib/python2.7/site-packages/AutoSklearn-0.0.1.dev0-py2.7-linux-x86_64.egg/autosklearn/automl.pyc in score(self, X, y)
358
359 def score(self, X, y):
--> 360 prediction = self.predict(X)
361 return evaluator.calculate_score(y, prediction, self.task_,
362 self.metric_, self.target_num_)
/media/selects/venv/py27/local/lib/python2.7/site-packages/AutoSklearn-0.0.1.dev0-py2.7-linux-x86_64.egg/autosklearn/estimators.pyc in predict(self, X)
137 The predicted classes.
138 """
--> 139 return super(AutoSklearnClassifier, self).predict(X)
140
141
/media/selects/venv/py27/local/lib/python2.7/site-packages/AutoSklearn-0.0.1.dev0-py2.7-linux-x86_64.egg/autosklearn/automl.pyc in predict(self, X)
327
328 if len(models) == 0:
--> 329 raise ValueError("No models fitted!")
330
331 if self.ohe_ is not None:
ValueError: No models fitted!