Comments (11)
You should re-run your code with debug on:
sklearn2pmml(.., debug = True)
This will give you estimator and mapper Pickle files. Can you convert them using the JPMML-SkLearn command-line application?
from sklearn2pmml.
Thank you for the quick reply.
Unfortunately, I am unable to convert the Pickle files when using the command-line.
When running java -jar target/converter-executable-1.1-SNAPSHOT.jar --pkl-input estimator.pkl --pmml-output estimator.pmml
I get the error:
Oct 01, 2016 12:59:30 AM org.jpmml.sklearn.Main run
INFO: Parsing Estimator PKL..
Oct 01, 2016 1:00:12 AM org.jpmml.sklearn.Main run
SEVERE: Failed to parse Estimator PKL
java.lang.IllegalArgumentException
at numpy.DType.toDescr(DType.java:64)
at joblib.NumpyArrayWrapper.toArray(NumpyArrayWrapper.java:39)
at org.jpmml.sklearn.PickleUtil$1.dispatch(PickleUtil.java:249)
at net.razorvine.pickle.Unpickler.load(Unpickler.java:99)
at org.jpmml.sklearn.PickleUtil.unpickle(PickleUtil.java:259)
at org.jpmml.sklearn.Main.run(Main.java:169)
at org.jpmml.sklearn.Main.main(Main.java:107)
Exception in thread "main" java.lang.IllegalArgumentException
at numpy.DType.toDescr(DType.java:64)
at joblib.NumpyArrayWrapper.toArray(NumpyArrayWrapper.java:39)
at org.jpmml.sklearn.PickleUtil$1.dispatch(PickleUtil.java:249)
at net.razorvine.pickle.Unpickler.load(Unpickler.java:99)
at org.jpmml.sklearn.PickleUtil.unpickle(PickleUtil.java:259)
at org.jpmml.sklearn.Main.run(Main.java:169)
at org.jpmml.sklearn.Main.main(Main.java:107)
Is there a problem with the types I use? Every element should be either a float or integer.
from sklearn2pmml.
This exception is most likely the root cause of the above CallProcessError
:
Exception in thread "main" java.lang.IllegalArgumentException
at numpy.DType.toDescr(DType.java:64)
at joblib.NumpyArrayWrapper.toArray(NumpyArrayWrapper.java:39)
It means that one of your Numpy arrays contains atypical dtype
specification. I would need to see inside your Pickle files in order to analyze and solve it - please attach your estimator and mapper Pickle files here and/or send them to my e-mail.
Also, what are your Numpy and Python versions? Please paste the version information (as printed by sklearn2pmml(.., debug = True)
) here.
from sklearn2pmml.
Thank you for the reply. Here are my python and numpy versions.
('python: ', '2.7.12')
('sklearn: ', '0.18')
('sklearn.externals.joblib:', '0.10.2')
('sklearn_pandas: ', '1.1.0')
('sklearn2pmml: ', '0.11.2')
Numpy version: 1.11.1 (was not printed out by debug mode)
Furthermore, I just double-checked and the dtype of the elements in "data" is float64.
As for the estimator and mapper Pickle files, I have emailed them to you.
from sklearn2pmml.
Upgraded my testing environment from Scikit-Learn 0.17.1 to 0.18. This exception now occurs with Python 2.7, but it doesn't occur with Python 3.4.
So, as a temporary workaround, you might try downgrading to Scikit-Learn 0.17.1, or upgrading to Python 3.4.
from sklearn2pmml.
Unfortunately, I get the same error when using Python 2.7 and Python 3.4 with Scikit-Learn 0.17.1.
Python 2.7 Versions:
('python: ', '2.7.12')
('sklearn: ', '0.17.1')
('sklearn.externals.joblib:', '0.9.4')
('sklearn_pandas: ', '1.1.0')
('sklearn2pmml: ', '0.11.2')
Python 2.7 Error:
Traceback (most recent call last):
File "/Users/leslie/Downloads/test_sklearn2pmml.py", line 68, in <module>
sklearn2pmml(neigh, df_mapper, "FeaturesAsFeaturesKNN.pmml", with_repr = True, debug = True)
File "/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/__init__.py", line 65, in sklearn2pmml
subprocess.check_call(cmd)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 541, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['java', '-cp', '/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/guava-19.0.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/istack-commons-runtime-2.21.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jaxb-core-2.2.11.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jaxb-runtime-2.2.11.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jcommander-1.48.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jpmml-converter-1.1.1.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jpmml-sklearn-1.1.2.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/jpmml-xgboost-1.1.1.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/pmml-agent-1.3.3.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/pmml-model-1.3.3.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/pmml-model-metro-1.3.3.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/pmml-schema-1.3.3.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/pyrolite-4.13.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/serpent-1.12.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/slf4j-api-1.7.21.jar:/Users/leslie/Library/Python/2.7/lib/python/site-packages/sklearn2pmml/resources/slf4j-jdk14-1.7.21.jar', 'org.jpmml.sklearn.Main', '--pkl-estimator-input', '/var/folders/ph/8rj4z16x1w3cy86hj3qh7nxh0000gn/T/estimator-fbU4yD.pkl.z', '--repr-estimator', "KNeighborsRegressor(algorithm='kd_tree', leaf_size=30, metric='minkowski',\n metric_params=None, n_jobs=-1, n_neighbors=400, p=2,\n weights='uniform')", '--pkl-mapper-input', '/var/folders/ph/8rj4z16x1w3cy86hj3qh7nxh0000gn/T/mapper-OpShIL.pkl.z', '--repr-mapper', "DataFrameMapper(features=[(['participant', 'condition', 'yaw', 'touchX', 'touchY', 'touchW', 'touchH', 'S0cx', 'S0cy', 'S0eo', 'S0evp', 'S0evm', 'S0ee', 'S1cx', 'S1cy', 'S1eo', 'S1evp', 'S1evm', 'S1ee', 'S2cx', 'S2cy', 'S2eo', 'S2evp', 'S2evm', 'S2ee', 'T0cx', 'T0cy', 'T0eo', 'T0evp', 'T0evm', 'T0ee', 'T1cx', 'T1cy', 'T1eo', 'T1evp', 'T1evm', 'T1ee', 'T2cx', 'T2cy', 'T2eo', 'T2evp', 'T2evm', 'T2ee', 'Ucx', 'Ucy', 'Ueo', 'Uevp', 'Uevm', 'Uee'], None), ('pitch', None)],\n sparse=False)", '--pmml-output', 'FeaturesAsFeaturesKNN.pmml']' returned non-zero exit status 1
Python 3.4 Versions:
python: 3.4.3
sklearn: 0.17.1
sklearn.externals.joblib: 0.9.4
sklearn_pandas: 1.1.0
sklearn2pmml: 0.11.2
Python 3.4 Error:
Traceback (most recent call last):
File "/Users/leslie/Downloads/test_sklearn2pmml.py", line 68, in <module>
sklearn2pmml(neigh, df_mapper, "FeaturesAsFeaturesKNN.pmml", with_repr = True, debug = True)
File "/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/__init__.py", line 65, in sklearn2pmml
subprocess.check_call(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 561, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['java', '-cp', '/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/guava-19.0.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/istack-commons-runtime-2.21.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jaxb-core-2.2.11.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jaxb-runtime-2.2.11.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jcommander-1.48.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jpmml-converter-1.1.1.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jpmml-sklearn-1.1.2.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/jpmml-xgboost-1.1.1.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/pmml-agent-1.3.3.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/pmml-model-1.3.3.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/pmml-model-metro-1.3.3.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/pmml-schema-1.3.3.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/pyrolite-4.13.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/serpent-1.12.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/slf4j-api-1.7.21.jar:/Users/leslie/Library/Python/3.4/lib/python/site-packages/sklearn2pmml/resources/slf4j-jdk14-1.7.21.jar', 'org.jpmml.sklearn.Main', '--pkl-estimator-input', '/var/folders/ph/8rj4z16x1w3cy86hj3qh7nxh0000gn/T/estimator-2rsvly58.pkl.z', '--repr-estimator', "KNeighborsRegressor(algorithm='kd_tree', leaf_size=30, metric='minkowski',\n metric_params=None, n_jobs=-1, n_neighbors=400, p=2,\n weights='uniform')", '--pkl-mapper-input', '/var/folders/ph/8rj4z16x1w3cy86hj3qh7nxh0000gn/T/mapper-mi8u4pgb.pkl.z', '--repr-mapper', "DataFrameMapper(features=[(['participant', 'condition', 'yaw', 'touchX', 'touchY', 'touchW', 'touchH', 'S0cx', 'S0cy', 'S0eo', 'S0evp', 'S0evm', 'S0ee', 'S1cx', 'S1cy', 'S1eo', 'S1evp', 'S1evm', 'S1ee', 'S2cx', 'S2cy', 'S2eo', 'S2evp', 'S2evm', 'S2ee', 'T0cx', 'T0cy', 'T0eo', 'T0evp', 'T0evm', 'T0ee', 'T1cx', 'T1cy', 'T1eo', 'T1evp', 'T1evm', 'T1ee', 'T2cx', 'T2cy', 'T2eo', 'T2evp', 'T2evm', 'T2ee', 'Ucx', 'Ucy', 'Ueo', 'Uevp', 'Uevm', 'Uee'], None), ('pitch', None)],\n sparse=False)", '--pmml-output', 'FeaturesAsFeaturesKNN.pmml']' returned non-zero exit status 1
Recreating the estimator and mapper Pickle files with Scikit-Learn 0.17.1 and manually converting it using the command-line returns the error:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.jpmml.converter.DOMUtil.createRow(DOMUtil.java:49)
at sklearn.neighbors.KNeighborsUtil.encodeNeighbors(KNeighborsUtil.java:111)
at sklearn.neighbors.KNeighborsRegressor.encodeModel(KNeighborsRegressor.java:56)
at sklearn.neighbors.KNeighborsRegressor.encodeModel(KNeighborsRegressor.java:31)
at sklearn.EstimatorUtil.encodePMML(EstimatorUtil.java:54)
at org.jpmml.sklearn.Main.run(Main.java:189)
at org.jpmml.sklearn.Main.main(Main.java:107)
from sklearn2pmml.
You can resolve java.lang.OutOfMemoryError
by increasing JVM heap size using -Xms
and -Xmx
command-line options:
java -Xms4G -Xmx16G -jar target/converter-executable-1.1-SNAPSHOT.jar ..
from sklearn2pmml.
It worked! Thank you very much!
from sklearn2pmml.
I'm reopening this issue, as I'm still working on the permanent fix.
My testing shows that this dtype
attribute issue only affects MLP and KNN model types. All other model types work correctly on Python 2.7 + Scikit-Learn 0.18.
from sklearn2pmml.
I've released sklearn2pmml version 0.12.0, which should be able to handle Python 2.7 in combination with Scikit-Learn 0.18. At least I was able to convert your Pickle files.
Anyway, I must warn you that if you're interested in evaluating your KNN models within JPMML/Openscoring software stack, then you'll be very likely disappointed with its performance. Your model contains 250k training instances, and during model scoring it will be necessary to select 400 of them (and average their predictions). Currently, JPMML/Openscoring performs the selection of closest training instances using brute-force scanning, which is clearly suboptimal here.
from sklearn2pmml.
I am also facing this issue now on Mac OS X, but due to a FileNotFoundError on the pkl path argument. I think it is trying to write the pkl in a temp folder (/var/folders/51/9s8qnl5x55b_1mx6fsf2gn480000gp/T/estimator-blIOk7.pkl.z
) but there is nothing there when I check. Maybe it could help if you would be able to set the intermediary Pickle file location yourself?
EDIT: Actually the Pickle is there, it does fail in an IPython notebook though, but when I run the command from the debug afterwards in the terminal it does work ...
from sklearn2pmml.
Related Issues (20)
- Unpickling error: invalid pickle opcode: 64 HOT 3
- Support for `numpy.datetime64` data type(s) HOT 23
- Failure to create a pmml file when using CountVectorizer with analyzer = 'char' HOT 1
- sklearn2pmml fails to create pmml for XGBoost model HOT 1
- Patching 0.51.1 for unknown categorical variables HOT 4
- Postprocessing the probabilities of an XGBoost Model HOT 3
- Link Estimator with prefit=True doesn't work
- Support for the `OneHotEncoder.infrequent_categories_` attribute HOT 1
- Revert PMML file to Python source code HOT 1
- Error when using `TargetEncoder` within `DataFrameMapper` within pipeline HOT 6
- Missing dependency `dill` from `setup.py`. HOT 7
- Support for Python power operator `**` HOT 1
- Assembling a pipeline from pre-fitted components HOT 3
- looking for support on quantile_forest for RandomForestQuantileRegressor HOT 2
- Failing to convert `CalibratedClassifierCV` plus `XGBClassifier` combination HOT 12
- Effectively debugging XGBoost pipelines (mis-matching predictions between Python and (J)PMML) HOT 14
- Revisit the missing value handling of custom transformers HOT 1
- Support for AgglomerativeClustering HOT 2
- `ExpressionTransformer` should try to rectify feature type information HOT 5
- `Alias` object has no attribute `transformer_` HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sklearn2pmml.