non-zero exit status 1 about sklearn2pmml

[vruusmann]

You should re-run your code with debug on:

sklearn2pmml(.., debug = True)

This will give you estimator and mapper Pickle files. Can you convert them using the JPMML-SkLearn command-line application?

[leslietso]

Thank you for the quick reply.

Unfortunately, I am unable to convert the Pickle files when using the command-line.

When running java -jar target/converter-executable-1.1-SNAPSHOT.jar --pkl-input estimator.pkl --pmml-output estimator.pmml

I get the error:

Oct 01, 2016 12:59:30 AM org.jpmml.sklearn.Main run
INFO: Parsing Estimator PKL..
Oct 01, 2016 1:00:12 AM org.jpmml.sklearn.Main run
SEVERE: Failed to parse Estimator PKL
java.lang.IllegalArgumentException
    at numpy.DType.toDescr(DType.java:64)
    at joblib.NumpyArrayWrapper.toArray(NumpyArrayWrapper.java:39)
    at org.jpmml.sklearn.PickleUtil$1.dispatch(PickleUtil.java:249)
    at net.razorvine.pickle.Unpickler.load(Unpickler.java:99)
    at org.jpmml.sklearn.PickleUtil.unpickle(PickleUtil.java:259)
    at org.jpmml.sklearn.Main.run(Main.java:169)
    at org.jpmml.sklearn.Main.main(Main.java:107)

Exception in thread "main" java.lang.IllegalArgumentException
    at numpy.DType.toDescr(DType.java:64)
    at joblib.NumpyArrayWrapper.toArray(NumpyArrayWrapper.java:39)
    at org.jpmml.sklearn.PickleUtil$1.dispatch(PickleUtil.java:249)
    at net.razorvine.pickle.Unpickler.load(Unpickler.java:99)
    at org.jpmml.sklearn.PickleUtil.unpickle(PickleUtil.java:259)
    at org.jpmml.sklearn.Main.run(Main.java:169)
    at org.jpmml.sklearn.Main.main(Main.java:107)

Is there a problem with the types I use? Every element should be either a float or integer.

[vruusmann]

This exception is most likely the root cause of the above CallProcessError:

Exception in thread "main" java.lang.IllegalArgumentException
    at numpy.DType.toDescr(DType.java:64)
    at joblib.NumpyArrayWrapper.toArray(NumpyArrayWrapper.java:39)

It means that one of your Numpy arrays contains atypical dtype specification. I would need to see inside your Pickle files in order to analyze and solve it - please attach your estimator and mapper Pickle files here and/or send them to my e-mail.

Also, what are your Numpy and Python versions? Please paste the version information (as printed by sklearn2pmml(.., debug = True)) here.

[leslietso]

Thank you for the reply. Here are my python and numpy versions.
('python: ', '2.7.12')
('sklearn: ', '0.18')
('sklearn.externals.joblib:', '0.10.2')
('sklearn_pandas: ', '1.1.0')
('sklearn2pmml: ', '0.11.2')
Numpy version: 1.11.1 (was not printed out by debug mode)

Furthermore, I just double-checked and the dtype of the elements in "data" is float64.

As for the estimator and mapper Pickle files, I have emailed them to you.

[vruusmann]

Upgraded my testing environment from Scikit-Learn 0.17.1 to 0.18. This exception now occurs with Python 2.7, but it doesn't occur with Python 3.4.

So, as a temporary workaround, you might try downgrading to Scikit-Learn 0.17.1, or upgrading to Python 3.4.

[leslietso]

Unfortunately, I get the same error when using Python 2.7 and Python 3.4 with Scikit-Learn 0.17.1.

Python 2.7 Versions:
('python: ', '2.7.12')
('sklearn: ', '0.17.1')
('sklearn.externals.joblib:', '0.9.4')
('sklearn_pandas: ', '1.1.0')
('sklearn2pmml: ', '0.11.2')

Python 2.7 Error:

Traceback (most recent call last):
  File "/Users/leslie/Downloads/test_sklearn2pmml.py", line 68, in <module>
    sklearn2pmml(neigh, df_mapper, "FeaturesAsFeaturesKNN.pmml", with_repr = True, debug = True)
  File "/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/__init__.py", line 65, in sklearn2pmml
    subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['java', '-cp', '/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/guava-19.0.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/istack-commons-runtime-2.21.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jaxb-core-2.2.11.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jaxb-runtime-2.2.11.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jcommander-1.48.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jpmml-converter-1.1.1.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jpmml-sklearn-1.1.2.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jpmml-xgboost-1.1.1.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/pmml-agent-1.3.3.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/pmml-model-1.3.3.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/pmml-model-metro-1.3.3.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/pmml-schema-1.3.3.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/pyrolite-4.13.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/serpent-1.12.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/slf4j-api-1.7.21.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/slf4j-jdk14-1.7.21.jar', 'org.jpmml.sklearn.Main', '--pkl-estimator-input', '/var/folders/ph/8rj4z16x1w3cy86hj3qh7nxh0000gn/T/estimator-fbU4yD.pkl.z', '--repr-estimator', "KNeighborsRegressor(algorithm='kd_tree', leaf_size=30, metric='minkowski',\n          metric_params=None, n_jobs=-1, n_neighbors=400, p=2,\n          weights='uniform')", '--pkl-mapper-input', '/var/folders/ph/8rj4z16x1w3cy86hj3qh7nxh0000gn/T/mapper-OpShIL.pkl.z', '--repr-mapper', "DataFrameMapper(features=[(['participant', 'condition', 'yaw', 'touchX', 'touchY', 'touchW', 'touchH', 'S0cx', 'S0cy', 'S0eo', 'S0evp', 'S0evm', 'S0ee', 'S1cx', 'S1cy', 'S1eo', 'S1evp', 'S1evm', 'S1ee', 'S2cx', 'S2cy', 'S2eo', 'S2evp', 'S2evm', 'S2ee', 'T0cx', 'T0cy', 'T0eo', 'T0evp', 'T0evm', 'T0ee', 'T1cx', 'T1cy', 'T1eo', 'T1evp', 'T1evm', 'T1ee', 'T2cx', 'T2cy', 'T2eo', 'T2evp', 'T2evm', 'T2ee', 'Ucx', 'Ucy', 'Ueo', 'Uevp', 'Uevm', 'Uee'], None), ('pitch', None)],\n        sparse=False)", '--pmml-output', 'FeaturesAsFeaturesKNN.pmml']' returned non-zero exit status 1

Python 3.4 Versions:
python: 3.4.3
sklearn: 0.17.1
sklearn.externals.joblib: 0.9.4
sklearn_pandas: 1.1.0
sklearn2pmml: 0.11.2

Python 3.4 Error:

Traceback (most recent call last):
  File "/Users/leslie/Downloads/test_sklearn2pmml.py", line 68, in <module>
    sklearn2pmml(neigh, df_mapper, "FeaturesAsFeaturesKNN.pmml", with_repr = True, debug = True)
  File "/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/__init__.py", line 65, in sklearn2pmml
    subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 561, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['java', '-cp', '/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/guava-19.0.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/istack-commons-runtime-2.21.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jaxb-core-2.2.11.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jaxb-runtime-2.2.11.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jcommander-1.48.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jpmml-converter-1.1.1.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jpmml-sklearn-1.1.2.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jpmml-xgboost-1.1.1.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/pmml-agent-1.3.3.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/pmml-model-1.3.3.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/pmml-model-metro-1.3.3.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/pmml-schema-1.3.3.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/pyrolite-4.13.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/serpent-1.12.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/slf4j-api-1.7.21.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/slf4j-jdk14-1.7.21.jar', 'org.jpmml.sklearn.Main', '--pkl-estimator-input', '/var/folders/ph/8rj4z16x1w3cy86hj3qh7nxh0000gn/T/estimator-2rsvly58.pkl.z', '--repr-estimator', "KNeighborsRegressor(algorithm='kd_tree', leaf_size=30, metric='minkowski',\n          metric_params=None, n_jobs=-1, n_neighbors=400, p=2,\n          weights='uniform')", '--pkl-mapper-input', '/var/folders/ph/8rj4z16x1w3cy86hj3qh7nxh0000gn/T/mapper-mi8u4pgb.pkl.z', '--repr-mapper', "DataFrameMapper(features=[(['participant', 'condition', 'yaw', 'touchX', 'touchY', 'touchW', 'touchH', 'S0cx', 'S0cy', 'S0eo', 'S0evp', 'S0evm', 'S0ee', 'S1cx', 'S1cy', 'S1eo', 'S1evp', 'S1evm', 'S1ee', 'S2cx', 'S2cy', 'S2eo', 'S2evp', 'S2evm', 'S2ee', 'T0cx', 'T0cy', 'T0eo', 'T0evp', 'T0evm', 'T0ee', 'T1cx', 'T1cy', 'T1eo', 'T1evp', 'T1evm', 'T1ee', 'T2cx', 'T2cy', 'T2eo', 'T2evp', 'T2evm', 'T2ee', 'Ucx', 'Ucy', 'Ueo', 'Uevp', 'Uevm', 'Uee'], None), ('pitch', None)],\n        sparse=False)", '--pmml-output', 'FeaturesAsFeaturesKNN.pmml']' returned non-zero exit status 1

Recreating the estimator and mapper Pickle files with Scikit-Learn 0.17.1 and manually converting it using the command-line returns the error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.jpmml.converter.DOMUtil.createRow(DOMUtil.java:49)
    at sklearn.neighbors.KNeighborsUtil.encodeNeighbors(KNeighborsUtil.java:111)
    at sklearn.neighbors.KNeighborsRegressor.encodeModel(KNeighborsRegressor.java:56)
    at sklearn.neighbors.KNeighborsRegressor.encodeModel(KNeighborsRegressor.java:31)
    at sklearn.EstimatorUtil.encodePMML(EstimatorUtil.java:54)
    at org.jpmml.sklearn.Main.run(Main.java:189)
    at org.jpmml.sklearn.Main.main(Main.java:107)

[vruusmann]

You can resolve java.lang.OutOfMemoryError by increasing JVM heap size using -Xms and -Xmx command-line options:

java -Xms4G -Xmx16G -jar target/converter-executable-1.1-SNAPSHOT.jar ..

[leslietso]

It worked! Thank you very much!

[vruusmann]

I'm reopening this issue, as I'm still working on the permanent fix.

My testing shows that this dtype attribute issue only affects MLP and KNN model types. All other model types work correctly on Python 2.7 + Scikit-Learn 0.18.

[vruusmann]

I've released sklearn2pmml version 0.12.0, which should be able to handle Python 2.7 in combination with Scikit-Learn 0.18. At least I was able to convert your Pickle files.

Anyway, I must warn you that if you're interested in evaluating your KNN models within JPMML/Openscoring software stack, then you'll be very likely disappointed with its performance. Your model contains 250k training instances, and during model scoring it will be necessary to select 400 of them (and average their predictions). Currently, JPMML/Openscoring performs the selection of closest training instances using brute-force scanning, which is clearly suboptimal here.

[gerardsimons]

I am also facing this issue now on Mac OS X, but due to a FileNotFoundError on the pkl path argument. I think it is trying to write the pkl in a temp folder (/var/folders/51/9s8qnl5x55b_1mx6fsf2gn480000gp/T/estimator-blIOk7.pkl.z) but there is nothing there when I check. Maybe it could help if you would be able to set the intermediary Pickle file location yourself?

EDIT: Actually the Pickle is there, it does fail in an IPython notebook though, but when I run the command from the debug afterwards in the terminal it does work ...