seldonio / seldon-server

Machine Learning Platform and Recommendation Engine built on Kubernetes

Update January 2018

• Seldon Core open sourced.
  • Seldon Core focuses purely on deploying a wide range of ML models on Kubernetes, allowing complex runtime serving graphs to be managed in production. Seldon Core is a progression of the goals of the Seldon-Server project but also a more restricted focus to solving the final step in a machine learning project which is serving models in production. Please have a look at the project page which includes extensive documentation to investigate further.

Seldon Server : * * Archived * *

This project is not actively maintained anymore please see Seldon Core.

Seldon Server is a machine learning platform that helps your data science team deploy models into production.

It provides an open-source data science stack that runs within a Kubernetes Cluster. You can use Seldon to deploy machine learning and deep learning models into production on-premise or in the cloud (e.g. GCP, AWS, Azure).

Seldon supports models built with TensorFlow, Keras, Vowpal Wabbit, XGBoost, Gensim and any other model-building tool — it even supports models built with commercial tools and services where the model is exportable.

It includes an API with two key endpoints:

1. Predict - Build and deploy supervised machine learning models created in any machine learning library or framework at scale using containers and microservices.
2. Recommend - High-performance user activity and content based recommendation engine with various algorithms ready to run out of the box.

Other features include:

• Complex dynamic algorithm configuration and combination with no downtime: run A/B and Multivariate tests, cascade algorithms and create ensembles.
• Command Line Interface (CLI) for configuring and managing Seldon Server.
• Secure OAuth 2.0 REST and gRPC APIs to streamline integration with your data and application.
• Grafana dashboard for real-time analytics built with Kafka Streams, Fluentd and InfluxDB.

Seldon is used by some of the world's most innovative organisations — it's the perfect machine learning deployment platform for start-ups and can scale to meet the demands of large enterprises.

Get Started

It takes a few minutes to install Seldon on a Kubernetes cluster. Visit our install guide and read our tech docs.

Community & Support

• Join the Seldon Users Group.
• Register for our newsletter to be the first to receive updates about our products and events.
• Visit our website, follow @seldon_io on Twitter and like our Facebook page.
• If you're in London, meet us at TensorFlow London - a community of over 1200 data scientists that we co-organise.
• We also offer commercial support plans and managed services.

License

Seldon is available under Apache Licence, Version 2.0

seldon-server's Issues

tagaffinityrecommender tag store not found

I'm trying tagaffinity recommender. the model looks fine. but i can't get any recommendations.
I get this error in api logs
2016-05-11 13:00:02,173 DEBUG http-nio-8080-exec-10 UserTagAffinityRecommender [client] [test123] [] - Failed to get tag store for client client 2016-05-11 13:00:02,173 DEBUG http-nio-8080-exec-10 RecommendationPeer [client] [test123] [] - Recommender userTagAffinityRecommender returned 0 results
what's tag store? why seldon can't find it?

In case it's related, this is my recommender configuration.
/all_clients/client/algs:
{"algorithms": [{"name":"userTagAffinityRecommender","filters":[],"includers":["recentItemsIncluder"], "config":[{"name":"io.seldon.algorithm.tags.attrid","value":7}, {"name":"io.seldon.algorithm.tags.useitemdim","value":"false"}, {"name":"io.seldon.algorithm.inclusion.itemsperincluder","value":1000}, {"name":"io.seldon.algorithm.clusters.minnumberitemsforvalidclusterresult","value":10}, {"name":"io.seldon.algorithm.clusters.decayratesecs","value":10800}, {"name":"io.seldon.algorithm.clusters.categorydimensionname","value":"category"}]}, {"name":"recentItemsRecommender","filters":[],"includers":[],"config":[]}], "combiner":"firstSuccessfulCombiner"}

and I got tags.attrid from item_attr table in seldon db
| attr_id | name | type | item_type | semantic |
| 7 | tags | VARCHAR | 1 | NULL |

NoTableManagedException

After the server started, I visit http://localhost:8080/seldon-server/token?consumer_key=seldonAll&consumer_secret=seldon

The server throw NoTableManagedException

2015-06-19 13:09:34,517 DEBUG http-bio-8080-exec-3 Log4JLogger.debug(58) - QueryCompilation:
[filter:DyadicExpression{PrimaryExpression{consumerKey} = ParameterExpression{c}}]
[symbols: c type=java.lang.String, this type=io.seldon.api.jdo.Consumer]
2015-06-19 13:09:34,519 DEBUG http-bio-8080-exec-3 Log4JLogger.debug(58) - JDOQL Query : Compiling "SELECT FROM io.seldon.api.jdo.Consumer WHERE consumerKey == c PARAMETERS java.lang.String c" for datastore
2015-06-19 13:09:40,765 WARN http-bio-8080-exec-3 Log4JLogger.warn(106) - Query for candidates of io.seldon.api.jdo.Consumer and subclasses resulted in no possible candidates
Persistent class "io.seldon.api.jdo.Consumer" has no table in the database, but the operation requires it. Please check the specification of the MetaData for this class.
org.datanucleus.store.rdbms.exceptions.NoTableManagedException: Persistent class "io.seldon.api.jdo.Consumer" has no table in the database, but the operation requires it. Please check the specification of the MetaData for this class.
at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:692)
at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getStatementForCandidates(RDBMSQueryUtils.java:425)

Errors creating kubernetes/conf/examples

When I try kubectl create -f import-data-job.json I get the following error:

could not read an encoded object from import-data-job.json: API version "batch/v1" in "import-data-job.json" isn't supported, only supports API versions ["v1"]

Same thing happens with examples/ml100k/ml100k-import.json. Any hint about this issue?

Anyway this new dockerized architecture seems great! Thank you!

java.lang.AssertionError: assertion failed: lapack.dppsv returned 1. at scala.Predef$.assert(Predef.scala:179)

I am getting below error when training matrix-factorization on latest version of seldon-server 1.3.2 and 1.3.3.

16/06/16 06:26:41 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS 16/06/16 06:26:41 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS 16/06/16 06:26:45 WARN TaskSetManager: Lost task 0.0 in stage 57.0 (TID 39, 172.17.0.10): java.lang.AssertionError: assertion failed: lapack.dppsv returned 1. at scala.Predef$.assert(Predef.scala:179) at org.apache.spark.ml.recommendation.ALS$CholeskySolver.solve(ALS.scala:393) at org.apache.spark.ml.recommendation.ALS$$anonfun$org$apache$spark$ml$recommendation$ALS$$computeFactors$1.apply(ALS.scala:1170) at org.apache.spark.ml.recommendation.ALS$$anonfun$org$apache$spark$ml$recommendation$ALS$$computeFactors$1.apply(ALS.scala:1131) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$mapValues$1$$anonfun$apply$41$$anonfun$apply$42.apply(PairRDDFunctions.scala:700) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$mapValues$1$$anonfun$apply$41$$anonfun$apply$42.apply(PairRDDFunctions.scala:700) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:278) at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78) at org.apache.spark.rdd.RDD.iterator(RDD.scala:262) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1944) at org.apache.spark.rdd.RDD$$anonfun$aggregate$1.apply(RDD.scala:1082) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.rdd.RDD.withScope(RDD.scala:310) at org.apache.spark.rdd.RDD.aggregate(RDD.scala:1075) at org.apache.spark.ml.recommendation.ALS$.computeYtY(ALS.scala:1182) at org.apache.spark.ml.recommendation.ALS$.org$apache$spark$ml$recommendation$ALS$$computeFactors(ALS.scala:1123) at org.apache.spark.ml.recommendation.ALS$$anonfun$train$3.apply(ALS.scala:578) at org.apache.spark.ml.recommendation.ALS$$anonfun$train$3.apply(ALS.scala:575) at scala.collection.immutable.Range.foreach(Range.scala:141) at org.apache.spark.ml.recommendation.ALS$.train(ALS.scala:575) at org.apache.spark.mllib.recommendation.ALS.run(ALS.scala:239) at org.apache.spark.mllib.recommendation.ALS$.trainImplicit(ALS.scala:417)

smrutiranjans@smrutirn-ub:~/seldon-server/kubernetes/conf$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
seldonio/seldon-stream 1.3.3 3d4b5505b2e6 2 days ago 728.3 MB
seldonio/seldon-server 1.3.3 a17472a3a306 2 days ago 1.182 GB
seldonio/mysql 1.0.1 eee2261bd6af 2 days ago 324.2 MB
seldonio/seldon-control 2.0.1_v1 7c4e65c50538 2 days ago 3.283 GB
seldonio/td-agent-server 1.0.5 80ae143f2cb5 4 days ago 433.1 MB
wurstmeister/kafka latest 9d084d73a156 7 days ago 221.1 MB
seldonio/grafana 1.0 abccee8b4538 2 weeks ago 230 MB
seldonio/td-agent-node 1.0.2 453456d4e4b6 2 weeks ago 433.1 MB
seldonio/zookeeper-k8s 1.0 bea89b8d2cbf 2 weeks ago 367.2 MB
seldonio/influxdb 1.0 48951c1855e1 2 weeks ago 271.1 MB
gcr.io/google_containers/hyperkube-amd64 v1.2.4 3c4f38def75b 5 weeks ago 316.8 MB
gcr.io/google_containers/exechealthz 1.0 82a141f5d06d 11 weeks ago 7.116 MB
gcr.io/google_containers/spark 1.5.2_v1 22712970844d 3 months ago 989.9 MB
gcr.io/google_containers/etcd-amd64 2.2.1 3ae398308ded 4 months ago 28.19 MB
gcr.io/google_containers/kube2sky 1.12 d4f3fadabe2b 6 months ago 24.48 MB
gcr.io/google_containers/etcd 2.2.1 a6cd91debed1 7 months ago 28.19 MB
gcr.io/google_containers/skydns 2015-10-13-8c72f8c 2927189088d7 8 months ago 40.55 MB
gcr.io/google_containers/pause 2.0 2b58359142b0 8 months ago 350.2 kB
seldonio/memcached 0.1 ed67562d41be 13 months ago 210.9 MB

root@seldon-control:/usr/lib/libblas# ls
libblas.a libblas.so libblas.so.3 libblas.so.3.0

Most popular items

Hi,

how can I setup seldon-server to recommend most popular items?
I try to set it with "cluster-by-dimension" model, but it always faild on model train action with java.lang.ClassNotFoundException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException

Thanks
mt.

Seldon building on Kubernetes

Hi,
I installed Kubernetes on AWS using the official guidance.
Now, I have a node master (m3.medium) and two workers (t2.medium) on which I need to run Seldon.
I'm using the official guidance: http://docs.seldon.io/install.html
But when i run the seldon-up.sh on Kubernetes, I obtained the following error during the creation of containers seldon-control-3388397084-pvr8k and influxdb-grafana-3089120915-fzxt4 :

Multi-Attach error for volume “pvc-d97313ee-93e1-11e7-ae85-02f1ec9e7a70” Volume is already exclusively attached to one node and can’t be attached to another Unable to mount volumes for pod “influxdb-grafana-3089120915-fzxt4_default(e21ad2fc-93e1-11e7-ae85-02f1ec9e7a70)“: timeout expired waiting for volumes to attach/mount for pod “default”/“influxdb-grafana-3089120915-fzxt4". list of unattached/unmounted volumes=[storage] Error syncing pod

How could I solve this issue??
Thank you for your time in advance.

Semantic vectors model generation problem

Hi
I'm trying to set up seldon based on semantic vector model. I have comma separated unicode string of tags in my item attributes. I followed this. Eventhough docker command looks positive:
Activating model in zookeper at node /all_clients/client/svtext with data /seldon-models/client/svtext/16913
there is no folder under /seldon-models and I get this error in api.log
2016-04-23 21:29:36,781 ERROR pool-3-thread-4 SemanticVectorsManager [] [] [] - Couldn't reloadFeatures for client client java.io.FileNotFoundException: /seldon-models/client/svtext/16913/termvectors.txt (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at io.seldon.resources.external.LocalFileStreamer.getResourceStream(LocalFileStreamer.java:49) at io.seldon.resources.external.ExternalResourceStreamer.getResourceStream(ExternalResourceStreamer.java:58) at io.seldon.sv.SemanticVectorsManager.loadModel(SemanticVectorsManager.java:84) at io.seldon.sv.SemanticVectorsManager.loadModel(SemanticVectorsManager.java:50) at io.seldon.recommendation.model.ModelManager$1.run(ModelManager.java:85) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
How can I get it working?

ConcurrentLinkedHashMap --> Caffeine when Java-8 based

When transitioning to requiring Java 8, please upgrade to Caffeine. The performance should be relatively the same and the per-instance memory usage should be smaller.

ConcurrentLinkedHashMap changes will continue to be minimal, even more so now, and driven by requests from Java 6 users unable to upgrade. Caffeine is ideally the upgrade path for Guava cache users too, which due to Android cannot be significantly modified.

Using helm for k8s install

You could do a lot of what your scripts are accomplishing by using helm charts. Users would simply call

helm repo add seldom HTTPS://github.com/seldonio
helm up
helm fetch seldonio/server
helm install server

Anywyas just a suggestio. Cool project!

seldom-shell command not found

I've downloaded the docker image to run seldon-shell but i obtain this:

vagrant@localhost:~$ docker run -it 2dd4d85f8bbe
root@eb924b3d3596:/# ls
BayesianOptimization bin boot dev etc home lib lib64 media mnt opt proc python-modules root run sbin scripts selinux srv sys tmp usr var
root@eb924b3d3596:/# seldon-shell
bash: seldon-shell: command not found

what i'm doing wrong?

Item Static Import: complex client_id

i tryed to import i file with two row like this:(obviously the schema it’s correct the file is only an example)
id,name,attr1,attr2,attr3,attr4,attr5
"563e146b1e0b688a0946a887”,"nam1”,"attr1_1","attr2_1","attr3_1","attr4_1","attr5_1"
"563e146b1e0b688a0946a884”,"nam2”,"attr1_2","attr2_2","attr3_2","attr4_2",”attr5_2"
i got this error:
inserting items into the db
Unexpected error ... <class '_mysql_exceptions.IntegrityError'>
Clearing DB of items and attributes
(1062, "Duplicate entry '9223372036854775807' for key 'PRIMARY’")
if i change the previous file in:
id,name,attr1,attr2,attr3,attr4,attr5
"563”,"nam1”,"attr1_1","attr2_1","attr3_1","attr4_1","attr5_1"
"5”,"nam2”,"attr1_2","attr2_2","attr3_2","attr4_2",”attr5_2”
using numeric id istead of UUID it goes ok!
Seems that the problem is not the UUID because using a RESTFUL POST /item it add the items correctly

Can't run server with "seldon-up.sh" - mysql status is forever Pending

I run Kubernetes locally via Minikube 0.12.1. My OS: Ubuntu 16.04. RAM: 16G (and 16GB swap).
Problem: when I run "seldon-up.sh" (Seldon v1.3.11), I see mysql node status is forever Pending.
When I run "kubectl get events", I see this:
24m 35m 40 mysql Pod Warning FailedScheduling {default-scheduler } pod (mysql) failed to fit in any node
fit failure on node (minikube): Insufficient memory

But I have 8.4GB RAM free.

Error with seldon-cli, does kubectl 1.6.2 supported?

Do you have the following error running seldon-cli?

error: expected 'exec POD_NAME COMMAND [ARG1] [ARG2] ... [ARGN]'.
POD_NAME and COMMAND are required arguments for the exec command
See 'kubectl exec -h' for help and examples.

Is 1.6.2 supported or should I use the older version?

my kubectl version is:
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:33:11Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:22:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Error in reuters-example in the Serve Recommendations part

Hello,
I am trying to run reuters-example following this http://docs.seldon.io/content-recommendation-example.html but when I execute run_recommendation_microservice.sh reuters-example seldonio/reuters-example 1.0 reuters I have this error:

deployment "reuters-example" configured service "reuters-example" configured WARNING:kazoo.client:Connection dropped: socket connection error: None WARNING:kazoo.client:Connection dropped: socket connection error: None connecting to zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181Traceback (most recent call last): File "/opt/conda/bin/seldon-cli", line 4, in <module> __import__('pkg_resources').run_script('seldon==2.0.1', 'seldon-cli') File "/opt/conda/lib/python2.7/site-packages/setuptools-18.5-py2.7.egg/pkg_resources/__init__.py", line 742, in run_script File "/opt/conda/lib/python2.7/site-packages/setuptools-18.5-py2.7.egg/pkg_resources/__init__.py", line 1667, in run_script File "/opt/conda/lib/python2.7/site-packages/seldon-2.0.1-py2.7.egg/EGG-INFO/scripts/seldon-cli", line 5, in <module> seldon.cli.start_seldoncli() File "/opt/conda/lib/python2.7/site-packages/seldon-2.0.1-py2.7.egg/seldon/cli/__init__.py", line 3, in start_seldoncli cli_main.main() File "/opt/conda/lib/python2.7/site-packages/seldon-2.0.1-py2.7.egg/seldon/cli/cli_main.py", line 346, in main start_zk_client(opts) File "/opt/conda/lib/python2.7/site-packages/seldon-2.0.1-py2.7.egg/seldon/cli/cli_main.py", line 301, in start_zk_client gdata["zk_client"].start() File "/opt/conda/lib/python2.7/site-packages/kazoo/client.py", line 546, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.threading.KazooTimeoutError: Connection time-out error: error executing remote command: Error executing command in container: Error executing in Docker Container: 1

PS: I have installed seldon in a cloud kubernetes cluster and i have my DNS pod which is working well.

Here are my running pods:
default influxdb-grafana-7mpow 2/2 Running 0 2h kubernetes-2 default kafka-controller-q5n79 1/1 Running 14 2h kubernetes-2 default memcached1-0mm1t 1/1 Running 0 2h kubernetes-2 default memcached2-xy165 1/1 Running 0 2h kubernetes-2 default reuters-example-1808766350-tq0ic 1/1 Running 0 2h kubernetes-2 default seldon-control 1/1 Running 0 2h kubernetes-3 default seldon-server-1847842434-nyksb 2/2 Running 35 2h kubernetes-2 default spark-master-controller-xuqfw 1/1 Running 0 19h kubernetes-3 default spark-worker-controller-f2aum 1/1 Running 0 1d kubernetes-3 default spark-worker-controller-l131c 1/1 Running 0 19h kubernetes-3 my-system elasticsearch-logging-v1-d13tc 1/1 Running 0 1d kubernetes-3 my-system fluentd-elasticsearch-kubernetes-2 1/1 Running 0 2h kubernetes-2 my-system fluentd-elasticsearch-kubernetes-3 1/1 Terminating 3 1d kubernetes-3 my-system heapster-f9lsf 2/2 Running 0 1d kubernetes-3 my-system infludb-grafana-m3fqi 2/2 Running 0 1d kubernetes-3 my-system kibana-logging-v1-7c8qr 1/1 Running 2 1d kubernetes-3 my-system kube-apiserver-kubernetes-1 1/1 Running 0 2h kubernetes-1 my-system kube-controller-manager-kubernetes-1 1/1 Running 55 2h kubernetes-1 my-system kube-dns-v8-fdk73 5/5 Running 0 1d kubernetes-3 my-system kube-scheduler-kubernetes-1 1/1 Running 44 2h kubernetes-1 my-system scheduler-master-kubernetes-1 2/2 Running 15 2h kubernetes-1
Could please help me or even give me an idea to explore.
Thank you in advance

Nadia

zookeeper cluster is not ready when deploying on kubernetes

Hi,
When tried to setup seldon on k8s cluster, it seemed that zookeeper cluster was not running as expected. I got some error as below,

2017-10-20 17:47:32,812 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@149] - Resolved hostname: zookeeper-2 to address: zookeeper-2/172.30.123.16
2017-10-20 17:47:35,819 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@400] - Cannot open channel to 3 at election address zookeeper-3/172.30.134.85:3888
java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:426)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:822)
2017-10-20 17:47:35,822 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@149] - Resolved hostname: zookeeper-3 to address: zookeeper-3/172.30.134.85
2017-10-20 17:47:35,823 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@852] - Notification time out: 60000

It seemed that the address setting is not correct.
How should I fix such issues?

Thx,
Jared

user similarity

What are the possibilities to having user similarity as part of seldon? Would exposing the nearest neighbor in the collab filtering accomplish this?

zookeeper cluster cannot communicate with each other

Hi all,
when I installing the seldon, I faced a problem.
After I installed kubernets, I run seldon-up.sh, the progress keep checking zookeeper status.
I investigate the problem, it was caused by that in the zookeeper pods, cannot get communication with each other with the hostname.
I try to using the service ip of each zookeeper, still failed. Because that the service ip cannot be accessed by the pod.
can you give some advise? thanks very much !!!

API to return past actions of a user

I there currently a way to return the past actions of a specific user?

Python support

Hello!
Is it support python 3.x? Because when I started with start-microservice --type prediction --client test -i iris-xgboost seldonio/iris_xgboost:2.1 rest 1.0 after running kubernetes, it shows bug like

$ start-microservice --type prediction --client test -i iris-xgboost seldonio/iris_xgboost:2.1 rest 1.0
  File "/Users/kwangin/workspace/seldon-server/kubernetes/bin/start-microservice", line 144
    print "Replicas is ",self.replicas
                       ^
SyntaxError: Missing parentheses in call to 'print'

Also, it shows an error at python 2.x like below.

$ start-microservice --type prediction --client test -i iris-xgboost seldonio/iris_xgboost:2.1 rest 1.0
[Microservice(iris-xgboost,seldonio/iris_xgboost:2.1,rest,1.000000)]
Replicas is  1
kubectl apply -f /Users/kwangin/workspace/seldon-server/kubernetes/bin/../conf/microservices/microservice-iris-xgboost.json
deployment "iris-xgboost" created
service "iris-xgboost" created
error: expected 'exec POD_NAME COMMAND [ARG1] [ARG2] ... [ARGN]'.
POD_NAME and COMMAND are required arguments for the exec command
See 'kubectl exec -h' for help and examples.
Traceback (most recent call last):
  File "/Users/kwangin/workspace/seldon-server/kubernetes/bin/start-microservice", line 313, in <module>
    runner.run(args.type,args.client,args.i + args.p)
  File "/Users/kwangin/workspace/seldon-server/kubernetes/bin/start-microservice", line 214, in run
    self.predictCreate(client,services)
  File "/Users/kwangin/workspace/seldon-server/kubernetes/bin/start-microservice", line 203, in predictCreate
    raise MicroserviceError("failed to run seldon cli to create conf"+name)
NameError: global name 'name' is not defined

Is there something missing?

Thanks.

Cannot start seldon-server war after clone && package it

Hi,

I have tried to deploy the seldon-server following this link:
http://docs.seldon.io/seldon-server-setup.html

but get these errors on tomcat:
org.apache.catalina.startup.TldConfig execute
INFORMACIÓN: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

Is there any specific wway to deploy the war file on it?

Thanks!

Move other projects into Seldon Server

This makes versioning and documentation easier.

Semantic vectors only recommends based on today's actions

My user has older actions in system. Is there a way to include those in recommenndation context?

2016-04-30 21:26:17,001 DEBUG http-nio-8080-exec-2 RecentImpressionsFilter [client] [OTokwpZOMEJAUZ7zuYwiKAZj] [] - RecentActions for user with client client internal user id 1 num.0 2016-04-30 21:26:17,998 DEBUG http-nio-8080-exec-3 RecommendationContext [client] [OTokwpZOMEJAUZ7zuYwiKAZj] [] - Built new rec context object in mode INCLUSION 2016-04-30 21:26:17,998 DEBUG http-nio-8080-exec-3 SemanticVectorsRecommender [client] [OTokwpZOMEJAUZ7zuYwiKAZj] [] - Can't recommend as no recent item interactions 2016-04-30 21:26:17,999 DEBUG http-nio-8080-exec-3 RecommendationPeer [client] [OTokwpZOMEJAUZ7zuYwiKAZj] [] - Recommender semanticVectorsRecommender returned 0 results

seldon-cli api --client-name reuters Failed to find single auth key

Hello,
I am trying to run reuters-example following this http://docs.seldon.io/content-recommendation-example.html but when I execute seldon-cli api --client-name reuters --endpoint /js/recommendations --item 6020 --limit 3 I have this error:

connecting to zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 [SUCCEEDED]
Failed to find single auth key
[]
error: error executing remote command: Error executing command in container: Error executing in Docker Container: 1

Could please help me or even give me an idea to explore.
Thank you in advance

Here is my pods:

influxdb-grafana-7mpow             2/2       Running   0          22h
kafka-controller-q5n79             1/1       Running   18         21h
memcached1-0mm1t                   1/1       Running   0          21h
memcached2-xy165                   1/1       Running   0          21h
mysql                              1/1       Running   0          18h
reuters-example-1808766350-tq0ic   1/1       Running   0          22h
seldon-control                     1/1       Running   0          21h
seldon-server-1847842434-3uduu     2/2       Running   4          16h
spark-master-controller-xuqfw      1/1       Running   0          1d
spark-worker-controller-f2aum      1/1       Running   0          1d
spark-worker-controller-l131c      1/1       Running   0          1d
td-agent-server                    1/1       Running   0          18h
zookeeper-1                        1/1       Running   0          18h
zookeeper-2                        1/1       Running   0          18h
zookeeper-3                        1/1       Running   0          18h

and my jobs:

NAME                  DESIRED   SUCCESSFUL   AGE
reuters-import-data   1         1            1d

Nadia

Maven can not find dependency seldon-server-config-${config.version}

I am now using the version is 0.90 I no matter what version is used to solve this Maven dependency.
Please helo me!

Seldon dashboard with hundreds of REST requests

Seldon grafana dashboard shows every API request for a client. With Recommendation REST API each user generates a different entry in charts. So after hundreds of users the chart legend is huge and the web interface hangs up.

Regards, and thank you
Enrique

failed to create rec_alg client

Hi developers,
I faced a problem, when I trying to run the Recommendation example
I run the command:
seldon-cli --debug rec_alg --action create --client-name reuters -f reuters.conf
connecting to zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 [SUCCEEDED]
Invalid client[reuters]
error: error executing remote command: Error executing command in container: Error executing in Docker Container: 1

the file reuters.conf is below:

{
"defaultStrategy": {
"algorithms": [
{
"config": [
{
"name": "io.seldon.algorithm.inclusion.itemsperincluder",
"value": 1000
},
{
"name": "io.seldon.algorithm.external.url",
"value": "http://reuters-example:5000/recommend"
},
{
"name": "io.seldon.algorithm.external.name",
"value": "reuters-example"
}
],
"filters": [],
"includers": [
"recentItemsIncluder"
],
"name": "externalItemRecommendationAlgorithm"
}
],
"combiner": "firstSuccessfulCombiner"
},
"recTagToStrategy": {}
}

could you tell me what should I do to analysis the problem to workaround it?
or how to debug the problem. thanks very much

end point for item based recommendation

is it possible to have a RESTFUL api end point that returns a list of items based on one item?
e.g GET /items/{item_id}/recommendations
thank you

Recommendation score not returned in recommendation results

Crash spark-worker-controller pod

Hello,
I have installed seldon in my local machine and now i am trying to run the Reuters Newswire Recommendation example but i have problems with spark-worker-controller and reuters-import-data pods.
The problems start after running kubectl create -f import-data-job.json command.
PS: I use a proxy to connect to the internet and i have added env variables for http_proxy and https_proxy

Can you help me please? Thank you in advance.
Here are the logs of my pods:
for spark-worker-controller: sed: -e expression #1, char 51: unknown option tos
=== Cannot resolve the DNS entry for spark-master. Has the service been created yet, and is SkyDNS functional?
=== See http://kubernetes.io/v1.1/docs/admin/dns.html for more details on DNS integration.
=== Sleeping 10s before pod exit.`

for reuters-import-data pod it stucks on ContainerCreating:
WARNING:kazoo.client:Connection dropped: socket connection error: Name or service not known Traceback (most recent call last): File "/opt/conda/bin/seldon-cli", line 4, in <module> __import__('pkg_resources').run_script('seldon==2.0.0', 'seldon-cli') File "/opt/conda/lib/python2.7/site-packages/setuptools-18.5-py2.7.egg/pkg_resources/__init__.py", line 742, in run_script File "/opt/conda/lib/python2.7/site-packages/setuptools-18.5-py2.7.egg/pkg_resources/__init__.py", line 1667, in run_script File "/opt/conda/lib/python2.7/site-packages/seldon-2.0.0-py2.7.egg/EGG-INFO/scripts/seldon-cli", line 5, in <module> seldon.cli.start_seldoncli() File "/opt/conda/lib/python2.7/site-packages/seldon-2.0.0-py2.7.egg/seldon/cli/__init__.py", line 3, in start_seldoncli cli_main.main() File "/opt/conda/lib/python2.7/site-packages/seldon-2.0.0-py2.7.egg/seldon/cli/cli_main.py", line 346, in main start_zk_client(opts) File "/opt/conda/lib/python2.7/site-packages/seldon-2.0.0-py2.7.egg/seldon/cli/cli_main.py", line 301, in start_zk_client gdata["zk_client"].start() File "/opt/conda/lib/python2.7/site-packages/kazoo/client.py", line 546, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.threading.KazooTimeoutError: Connection time-out connecting to zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

Custom prediction output

Can seldon be used to output prediction in any JSON format? My model outputs predictions as well other custom data.

Can i change this below format to anything i want?

{
  "predictions": [
    {
      "score": 0.9,
      "classId": "1",
      "confidence":0.7
    }
  ]
}

seldon-cli attr --action apply doesnt pick up the attributes json file

upon executing the following command in the command line : -
seldon-cli attr --action apply --client-name univision --json attr.json
the following error occurs -
Traceback (most recent call last):
File "/opt/conda/bin/seldon-cli", line 4, in
import('pkg_resources').run_script('seldon==2.1', 'seldon-cli')
File "/opt/conda/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/pkg_resources/init.py", line 719, in run_script
connecting to zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 [SUCCEEDED]
File "/opt/conda/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/pkg_resources/init.py", line 1504, in run_script
File "/opt/conda/lib/python2.7/site-packages/seldon-2.1-py2.7.egg/EGG-INFO/scripts/seldon-cli", line 5, in
seldon.cli.start_seldoncli()
File "/opt/conda/lib/python2.7/site-packages/seldon-2.1-py2.7.egg/seldon/cli/init.py", line 3, in start_seldoncli
cli_main.main()
File "/opt/conda/lib/python2.7/site-packages/seldon-2.1-py2.7.egg/seldon/cli/cli_main.py", line 355, in main
cmds[cmd](opts,command_data, command_args)
File "/opt/conda/lib/python2.7/site-packages/seldon-2.1-py2.7.egg/seldon/cli/cmd_attr.py", line 250, in cmd_attr
actions[action](command_data, opts)
File "/opt/conda/lib/python2.7/site-packages/seldon-2.1-py2.7.egg/seldon/cli/cmd_attr.py", line 210, in action_apply
store_json(command_data,opts)
File "/opt/conda/lib/python2.7/site-packages/seldon-2.1-py2.7.egg/seldon/cli/cmd_attr.py", line 106, in store_json
f = open(opts.json)
IOError: [Errno 2] No such file or directory: 'attr.json'

the attr.json is present in the working directory
Shreyass-MBP:univision_poc shreyassrinivasan$ pwd
/Users/shreyassrinivasan/seldon/univision_poc
Shreyass-MBP:univision_poc shreyassrinivasan$ ls
attr.json syndication_report_20161215_to_20161223.csv
items.csv
Shreyass-MBP:univision_poc shreyassrinivasan$

td-agent configuration file

I recently deployed seldon-server 1.2.3. I can add actions using the api but no new action logs are created in ${TOMCAT_HOME}/logs/fluentd/ and GroupActionsJob is not working. td-agent.conf is set to parse the previous (not json) action logs. How can I fix it?

Support for Apache Spark 2.X

Hey guys, I see that the current version of seldon-server uses Spark 1.5.2 by default, given the dramatic performance and feature improvements in Spark 2.X, are there any plans to support it?

kubernetes yamls should utilize a default storage class

Seldon should consider just using a PVC without a PV to let the a PV be created using the user's cluster's default storage class.

This would make it trivial to deploy in a number of environments where PV provisioning is available without needing to put more yaml in an environment variable.

Could not load [org.apache.zookeeper.server.ZooTrace]

Hi everyone. I got this error while deploying seldon-server 1.3.11 outside Kubernetes.

This is not the first time i deploy seldon outside kubernetes and i never had problem.

Can someone help me?

17-Feb-2017 11:13:45.366 INFO [localhost-startStop-2] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /home/vagrant/apps/tomcat/webapps/seldonversion.war
17-Feb-2017 11:14:00.873 SEVERE [localhost-startStop-2] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
17-Feb-2017 11:14:00.874 SEVERE [localhost-startStop-2] org.apache.catalina.core.StandardContext.startInternal Context [/seldonversion] startup failed due to previous errors
17-Feb-2017 11:14:00.950 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [seldonversion] appears to have started a thread named [localhost-startStop-2-SendThread(172.17.0.4:2181)] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Thread.sleep(Native Method)
org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219)
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1109)
17-Feb-2017 11:14:00.959 INFO [localhost-startStop-2] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /home/vagrant/apps/tomcat/webapps/seldonversion.war has finished in 15,592 ms
17-Feb-2017 11:14:01.037 INFO [localhost-startStop-2-SendThread(172.17.0.4:2181)] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1353)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1341)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1206)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1167)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1115)

are we allowed to use multiple types for metadata?

Hey,

I see that on the content rec guide page it says that we're only supposed to use 1 as the type_id.

Reading later in the docs, I was confused on the REST api page, it talks about adding music items and video game items.

Which is correct?

Edit: These pages have a few contradictions that are confusing me. The other one is user data. The REST api page talks about demographics and other attributes, while the guide page says only id and username are possible.

Array of attributes for attributes

Reading through the docs, I don't see a way in the meta data structuring to add an attribute with an array data type. Here's my issue:

I want to add ~10 actors per movie and ~15 actors per tv show. I originally thought about denormalizing this into actor_1, actor_2, etc. But that won't work relating separate attributes unless an actor happens to be in actor_1 multiple times, right?

In a perfect world, I'd be able to something like:

{
  [...],
  "type_attrs": [
    {"name": "actors", "value_type": ["int"]}  //<-- array of ints
  ]
}

Is there anything I could do to get close to something like this? Is there something else I'm missing that would work instead?

The same thing happens with genres since a movie or tv show usually has multiple genres (ex: thriller, action vs. thriller, horror).

minikube json validation error

Minikube complains about a json validation error for mysql in the latest released version.

error validating "/seldon-server/kubernetes/bin/../conf/mysql.json": error validating data: [unexpected type: object, unexpected type: object, unexpected type: object, unexpected type: object]; if you choose to ignore these errors, turn validation off with --validate=false

REST API - actions

Hi,

i'm playing with Seldon and I have problem with actions REST API.

I send this testing request to seldon server
curl -X POST -H "Content-Type: application/json" -d '{"user": "user1", "item": "item1"}' "http://seldon-server/actions?oauth_token=5rf6f5r5adsjhlsma08u2rbs97"

Everything goes well if I send it for first time and I've got response like this
{"actionId":null,"user":"user1","item":"item1","type":0,"date":null,"value":null,"times":null,"comment":null,"tags":null,"recTag":null,"referrer":null,"extraData":null}

But when I send same curl request again I've got this error:
{"error_id":26,"error_msg":"The supplied fields are incorrect or incompatible with your current integration. Please contact Rummble Labs support.","http_response":400}

Am I doing something wrong or it's possible to have only one unique action for user - item combination?
I think that user can have multiple actions of same type for one item - for example multiple page views of one article but in different time.

mt.

Are action 'values' allowed to be negative?

For example, could I import an 'actions.csv' containing:

user_id,item_id,value,time
1,2,-1,1422128735
2,2,1,1422752450
3,3,1,1422735290

No dimensions for int and boolean attribute type

This is from the documentation at http://docs.seldon.io/api-oauth.html#actions

The item attribute definition is:
string name [attr_id 1]
string artist [attr_id 2]
enum category [attr_id 3]
double price [attr_id 4]
Where:
category is the enumeration
** (pop [value_id 1], rock [value_id 2], rap [value_id 3])
a range definition is created for the price ** (<10 [value_id 1], 10-20 [value_id 2], >20 [value_id 3])
We’ll have the following dimension definition:
dimension1 [dim_id 1, attr_id 3, value_id 1] (category = pop)
dimension2 [dim_id 2, attr_id 3, value_id 2] (category = rock)
dimension3 [dim_id 3, attr_id 3, value_id 3] (category = rap)
dimension4 [dim_id 4, attr_id 4, value_id 1] (category =<10)
dimension5 [dim_id 5, attr_id 4, value_id 2] (category = 10-20)
dimension6 [dim_id 6, attr_id 4, value_id 3] (category => 20)

Now the question is: when i define the attribute seldon automatically create dimensions only for the enum value.

What about the boolean?
What about the int?
how should i define the range of an int attribute?

Import historic actions, and add to database.

Is there a sensible way to import large amounts of historic actions?

Using seldon-cli import --action actions--client-name CLIENT_NAME --file-path PATH_TO_FILE imports them in some strange way that only Spark jobs can see.

No result for Movielens 100K Worked Example

I am trying to run ml100k example. All things are all right but when I tape seldon-cli api --client-name ml100k --endpoint /js/recommendations --item 50 --limit 4 and get this result

connecting to zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 [SUCCEEDED]
response code 200
{"size":0,"requested":4,"list":[]}

which is different from what is mentioned in the tutorial.

Could you help me please?
Nadia

Setting dimensions

Hi

I have a string attribute for my items which I can probably change to enum. How can I set dimensions for my items based on that? Will seldon automatically generate dimensions if i manually alter items table in database and change type of that column?

Ignored parameters in ItemSimilaritySparkJob

Hello,

It seems that sortAndLimit(..) method in SimilarItems.scala hardcodes limit parameter with value 100. So parameterization via luigi/spark.py parameters is overriden.

Moreover luigi/spark.py doesn't provide default value for 'sample' parameter in ItemSimilaritySparkJob.

Sledon-cli

Is there some command to delete client, user or item? I don't find nothing in docs.
What if I want to delete item which was removed from my app?

CORS: "Access-Control-Allow-Origin"

Is there currently support for adding the "Access-Control-Allow-Origin" header to the REST API requests? Nearly all modern browsers are set to block those sorts of requests without that header.

(Obviously this is ignoring JSONP, which has its share of security concerns.)

kubernetes yamls should use Deployments not raw Pods

I deleted a Pod to try to kick it to get it to start properly. Much to my surprise... it simply didn't come back. Why is it not using a Deployment?

JavaScript API has no 'value' parameter for 'action' ingestion.

Unless I am missing something, how would I use the JS API to ingest a user action with a non-zero 'value' parameter.

I think this is the relevant file.

recommendation filtering by dimension with MF

Hi everyone.
I got this error when i try to filter a recommendation from the MF alg

2017-03-09 15:38:05,055 DEBUG http-nio-8080-exec-132 UserDimensionMappingModelManager [user1] [10206938095543821] [] - dimensions in: [74]
2017-03-09 15:38:05,055 DEBUG http-nio-8080-exec-132 UserDimensionMappingModelManager [user1] [10206938095543821] [] - No mappings for client[user1]
2017-03-09 15:38:05,055 DEBUG http-nio-8080-exec-132 UserDimensionMappingModelManager [user1] [10206938095543821] [] - dimensions out: [74]

Otherwise if i use a recentItems recommendation al the dimensions works correctly
Can someone explain what this log means?