alphaville / jaqpot3 Goto Github PK

Prompt users to login when they try to access the algorithm web services (/algorithm/id) - although get is allowed. On the login page users should be informed that the public 'guest' account can be used.

Prediction features are created on the default service, regardless of the dataset service specified.

e.g.

http://opentox.ntua.gr:8080/model/5527b45d-3779-4732-a105-dd381540710e

the 403 error is not returned in the right content-type:

curl -H "Accept:application/rdf+xml" http://opentox.ntua.gr:8080/model/bbab3714-e90b-4990-7d3a30eece

return Content-Type: text/html instead of rdf-xml

This is a feature request.

1. The names (dc:title) of predicted variables in jaqpot3 models are automatically generated in the form

"Feature created as prediction feature for the MLR model http://opentox.ntua.gr:8080/model/536ce048-b78b-4fb3-8f0f-1c85447bfb41"

dc:title is supposed to be presented to the end user and should have more user friendly content. A possible suggestion is to copy the name of the dependent variable, which was usedto create the model.

2. owl:sameAs of the prediction variable is empty. This usually points to an ontology entry, for example denoting the toxicology endpoint being modeled. If the dependent variable has such entry assigned, it might be useful to assign the same entry to the predicted variable. This information is used in client applications to find out which toxicology endpoint is modeled.

for calculating DoA, external algorithms and their URIs should be supported

List all users that participate in jaqpot and access there publicly available data (username, name, statistics).

when user tasks exceed the user is redirected to ifacegeneric without a warning, to be fixed

perform check for type of substances

Tasks need to output more background information using rdfs:comment so that the client knows exactly at what stage the task is.

I logged in at http://opentox.ntua.gr:8080/login and tried to access my profile at
/user/{id}, but got a HTTP status code 500 with the message: "The server encountered an unexpected condition which prevented it from fulfilling the request".

When a user logs in for the first time and they have not created any models, then no user profile is created for them (and maybe this is wrong). However, an attempt to access their (non existing) profile should return 404 and not 500.

I then created a model on the server (hence, my profile was created and was registered in the database) and I could then access my profile page normally. I could also access my quota page without problems.

In the PMML representation of a MLR model as it is produced currently by jaqpot, the URI of the dataset is URL-encoded:

<DatasetID href="http%3A%2F%2Fapps.ideaconsult.net%3A8080%2Fambit2%2Fdataset%2FR545"/>

See https://gist.github.com/alphaville/b55a1d1e64e7a52bf3c4

cURL command to reproduce the issue:

export token=`curl -X POST -k 'https://opensso.in-silico.ch:443/auth/authenticate?uri=service=openldap' -d username=guest -d password=guest 2> /dev/null`;
token=`echo $token | cut -c10-80`;
echo "[TOKEN] $token"
curl -X GET http://opentox.ntua.gr:8080/model/e5ebc96d-a5d4-47f8-9962-3e52c07901fb \
-H subjectid:$token \
-H Accept:application/xml

I see that at https://github.com/alphaville/jaqpot3/blob/master/jaqpot3-standalone/src/main/java/org/opentox/jaqpot3/qsar/util/PMMLGenerator.java#L77 there is an invocation to URLEncoder.encode - I'm not 100% sure that this is necessary. We should take a closer look and remove it if it's not necessary to encode it.

http://opentox.ntua.gr/mediawiki/index.php/Custom_Algorithm_Generation to be deleted????
http://opentox.ntua.gr/mediawiki/index.php/Use_a_Customised_Algorithm to be deleted????

The HTTP request:

curl -v "http://opentox.ntua.gr:8080/model/c32910c9-73c8-4356-b22b-4f39b68e8af0"  \
-H subjectid:$token -H Accept:application/xml

returned the content type:

Content-Type: text/plain;charset=UTF-8

while it should have been application/xml.

An error when applying jaqpot3 model. Note it does not occur always.

Applying the model

 curl -X POST -d "dataset_uri=http://apps.ideaconsult.net:8080/ambit2/dataset/SOME_DATASET" -d "model_uri=http://opentox.ntua.gr:8080/model/536ce048-b78b-4fb3-8f0f-1c85447bfb41" -d "dataset_uri=http://apps.ideaconsult.net:8080/ambit2/dataset" http://apps.ideaconsult.net:8080/ambit2/algorithm/superservice -iv -H "Accept:text/uri-list" -H "subjectid:SOME_TOKEN"

Querying the superservice task status

 curl http://apps.ideaconsult.net:8080/ambit2/task/d721985d-8f56-4fce-a299-261979ab7d23 -i
HTTP/1.1 502 Bad Gateway


  
  
  
  
  
  
  
  
  
  
  
    http://apps.ideaconsult.net:8080/ambit2/task/d721985d-8f56-4fce-a299-261979ab7d23
    
      
        java.io.IOException: Server returned HTTP response code: 500 for URL: http://opentox.ntua.gr:8080/task/dc20d666-fa36-4d7a-a46f-bfda35b412e9
        http://opentox.ntua.gr:8080/task/dc20d666-fa36-4d7a-a46f-bfda35b412e9 Server returned HTTP response code: 500 for URL: http://opentox.ntua.gr:8080/task/dc20d666-fa36-4d7a-a46f-bfda35b412e9
        http://opentox.ntua.gr:8080/task/dc20d666-fa36-4d7a-a46f-bfda35b412e9 Server returned HTTP response code: 500 for URL: http://opentox.ntua.gr:8080/task/dc20d666-fa36-4d7a-a46f-bfda35b412e9
        502
      
    
    0.0
    Error
    1302680588251
    Apply Calls a remote service  
  
  
  

Querying the Jaqpot3 task

curl http://opentox.ntua.gr:8080/task/dc20d666-fa36-4d7a-a46f-bfda35b412e9 -i
HTTP/1.1 500 Internal Server Error


  
    
    Provides access to chemical compounds and their features (e.g. structural, physical-chemical, biological, toxicological properties)
    
  
  
    
  
  
    HTTP status
    
    
    
  
  
    
    Generic OpenTox Resource
    
  
  
    has status
    
    
    
  
  
    
    
  
  
    URI of the peer that produces the exception.
    error actor
    
    
    
  
  
    
    Provides access to OpenTox algorithms
    
  
  
    
    
    
    
    
    
  
  
    
    Instances of the class ErrorReport appear in ontological models to report some exceptional event that happened during the transaction of data from one server to another, request errors or internal server errors. An error report is characterized by its actor, the corresponding status code, some unique identifier for the exceptional event and possibly another error report that triggered its creation.
    
  
  
    
  
  
    Asynchronous task created for a background job initiated by the model: 536ce048-b78b-4fb3-8f0f-1c85447bfb41
    ERROR
    
    [email protected]
    0.0
    
    Asynchronous Task for Prediction
    500.0
    Failed task. 
    0
    
  
  
    Estimated percentage of completion of a running task.
    
    
    
  
  
    500
    http://opentox.ntua.gr:8080/
    org.opentox.jaqpot3.qsar.InstancesUtil.sortByFeatureAttrList(InstancesUtil.java:72)
org.opentox.jaqpot3.qsar.InstancesUtil.sortByFeatureList(InstancesUtil.java:35)
org.opentox.jaqpot3.qsar.InstancesUtil.sortForModel(InstancesUtil.java:27)
org.opentox.jaqpot3.qsar.regression.WekaPredictor.predict(WekaPredictor.java:57)
org.opentox.jaqpot3.www.services.PredictionService.run(PredictionService.java:81)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
java.util.concurrent.FutureTask.run(FutureTask.java:138)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)

    The Dataset you provided does not contain feature:http://apps.ideaconsult.net:8080/ambit2/feature/22137
    Error report produced by 'http://opentox.ntua.gr:8080/'
    http://opentox.ntua.gr:8080/error/f88736f6-3fc0-4ac8-99ba-6946b541ac9a
    
  
  
    
  
  
    
    
  
  
    For example: 'Prediction feature not provided'
    A simple message providing some simple description of the exceptional event.
    error message
    
    
    
  
  
    
  
  
    
    
  
  
    
    
  
  
    
    
    
  
  
    
  
  
    
    
    
  
  
    
  
  
    Detailed message including technical information about the exceptional event. Can be used to help both peers in the debugging.
    error details
    
    
    
  
  
    
    
  
  
    errorReport: Task --> ErrorReport
    Assigns an error report generated by a failed task.
    
    
    
  

The message imply some features are missing, but the superservice does calculate the required features and these should be present.

Sorry for the lack of formatting , can't find how to represent RDF.XML markup via GitHub markup.

When in the user quota page, e.g., http://opentox.ntua.gr:8080/user/[email protected]/quota, the link of the user ID that leads to the user's main page doesn't work. The reason is that @ is encoded as %40. We should take care of this issue.

Set up the data cleanup web service.

The service at /algorithm returns a 303 if the client does not provide an authorisation token. This is not compliant to the OpenTox API v1.2. I think the best option is that /algorithm always returns a list of all algorithm URIs and does not require user authentication.

This raises the need for a web-based user interface that is perfectly separated from the services. We should also consider removing the HTML support form JAQPOT at a certain point.

Implement PLS filtering web service.

Users that are not logged in get an Internal Server Error (500) when they try to access models.

To reproduce the error: What I did is that I logged in with my credentials and created a model, namely, http://enanomapper.ntua.gr:8080/model/d751fab1-1160-476c-b5e8-1a930a12df1e and I then logged out. I attempted to access the model I had previously created (of course this should fail because of insufficient access rights and return a status code 403 and an error report). However, I got a 500.

Wiki - Alg-Preproc - Training-Workflow : Shouldn’t be 6 to 5??? and 8 to 7???

There must be "logout" support that erases all AA-related cookies from the user's browser and invalidates the token at the SSO server.

Got the following exception...

SEVERE: null
org.opentox.toxotis.database.exception.DbException: failed to execute statement
at org.opentox.toxotis.database.engine.error.AddErrorReport.write(AddErrorReport.java:39)
at org.opentox.jaqpot3.www.services.RunnableTaskService.updateFailedTask(RunnableTaskService.java:45)
at org.opentox.jaqpot3.www.services.TrainingService.run(TrainingService.java:121)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.sql.BatchUpdateException: Data truncation: Data too long for column 'message' at row 1
at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:1065)
at com.mchange.v2.c3p0.impl.NewProxyStatement.executeBatch(NewProxyStatement.java:743)
at org.opentox.toxotis.database.engine.error.AddErrorReport.write(AddErrorReport.java:30)
... 8 more

In this case the task cannot be updated to 'error/failed' and appears to be running indefinitely.

Prompt users to login when they try to access the algorithm web services (/algorithm/id) - although get is allowed. On the login page users should be informed that the public 'guest' account can be used.

The services hang when lots of requests are issued. I repeated the following and the whole WS hung!

task_uri=`curl -X POST -H subjectid:AQIC5wM2LY4Sfcy5t0Dn3DKUsKQiKF_hWCiVWuTwZa7svaQ.*AAJTSQACMDE.* \
-H Accept:text/uri-list -d dataset_uri=http://apps.ideaconsult.net:8080/ambit2/dataset/R545 \
-d prediction_feature=http://apps.ideaconsult.net:8080/ambit2/feature/22200 \
http://enanomapper.ntua.gr:8080/algorithm/mlr`; 
curl $task_uri > /dev/null;

then I couldn't do almost anything and, for example, the following request:

 curl http://enanomapper.ntua.gr:8080/task/ -H Accept:text/uri-list;

returned an empty response body and status code 200.

I had to restart jaqpot to get things working again. I suspect this has to do with the system configuration (e.g., MySQL configuration is currently the default one). It seems (according to javamelody) that it is not some memory management issue.

Additional information:
At a certain point, after having restarted jaqpot (from the admin backend of tomcat) several times, I got the following error:

type Exception report

message

description The server encountered an internal error () that prevented it from fulfilling this request.

exception

javax.servlet.ServletException: javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: PermGen space
    org.apache.catalina.manager.StatusManagerServlet.doGet(StatusManagerServlet.java:305)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)

root cause

javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: PermGen space
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:841)
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
    com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
    org.apache.catalina.manager.StatusTransformer.writeProcessorState(StatusTransformer.java:368)
    org.apache.catalina.manager.StatusTransformer.writeConnectorState(StatusTransformer.java:301)
    org.apache.catalina.manager.StatusManagerServlet.doGet(StatusManagerServlet.java:290)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)

root cause

java.lang.OutOfMemoryError: PermGen space

note The full stack trace of the root cause is available in the Apache Tomcat/7.0.26 logs.

ToxOtis/C3P0/Application Logs
Logs are not very suggestive as of what the problem may be. When hell breaks loose, there are no particular messages in the logs!

Java Configuration
I changed the Java configuration inside catalina.sh with the optimal parameters found here; still no success...

dc:title is missing from the RDF Model representation

e.g. http://opentox.ntua.gr:8080/model/536ce048-b78b-4fb3-8f0f-1c85447bfb41

A minor issue that creates some log entries:

INFO: Exception or error caught in resource
java.lang.NullPointerException
at org.opentox.jaqpot3.resources.JaqpotResource.updatePrimaryId(JaqpotResource.java:133)
at org.opentox.jaqpot3.resources.ErrorResource.doInit(ErrorResource.java:44)
at org.restlet.resource.UniformResource.init(UniformResource.java:565)
at org.restlet.ext.wadl.WadlApplication.getResourceInfo(WadlApplication.java:495)
at org.restlet.ext.wadl.WadlApplication.getResourceInfo(WadlApplication.java:617)
at org.restlet.ext.wadl.WadlApplication.getResourceInfo(WadlApplication.java:651)
at org.restlet.ext.wadl.WadlApplication.getResourceInfos(WadlApplication.java:673)
at org.restlet.ext.wadl.WadlApplication.getApplicationInfo(WadlApplication.java:399)
at org.opentox.jaqpot3.www.JaqpotWebApplication.getApplicationInfo(JaqpotWebApplication.java:32)
at org.opentox.jaqpot3.www.JaqpotWebApplication.getAppInfo(JaqpotWebApplication.java:44)
at org.opentox.jaqpot3.resources.IndexResource.options(IndexResource.java:67)
at org.opentox.jaqpot3.resources.IndexResource.get(IndexResource.java:55)
at org.restlet.resource.ServerResource.doHandle(ServerResource.java:462)
at org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:522)
at org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:258)
at org.restlet.resource.ServerResource.handle(ServerResource.java:752)
at org.restlet.resource.Finder.handle(Finder.java:515)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Router.doHandle(Router.java:472)
at org.restlet.routing.Router.handle(Router.java:688)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:152)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.engine.ChainHelper.handle(ChainHelper.java:111)
at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:72)
at org.restlet.Application.handle(Application.java:379)
at org.restlet.ext.wadl.WadlApplication.handle(WadlApplication.java:769)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Router.doHandle(Router.java:472)
at org.restlet.routing.Router.handle(Router.java:688)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Router.doHandle(Router.java:472)
at org.restlet.routing.Router.handle(Router.java:688)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:152)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.engine.ChainHelper.handle(ChainHelper.java:111)
at org.restlet.Component.handle(Component.java:385)
at org.restlet.Server.handle(Server.java:474)
at org.restlet.engine.ServerHelper.handle(ServerHelper.java:71)
at org.restlet.engine.http.HttpServerHelper.handle(HttpServerHelper.java:150)
at org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1035)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:151)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:129)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)

Check again the method updatePrimaryId to cater for NPEs

Applies to branch: FileUpload
Through the web interface, MLR training fails with error message: "The parameter 'prediction_feature' is mandatory for this algorithm" (status code: 400).

I also tried via cURL and I get the same error. Here's the cURL command I used (change the subjectid):

curl -X POST -d dataset_uri=http://apps.ideaconsult.net:8080/ambit2/dataset/R545 \
-d prediction_feature=http://apps.ideaconsult.net:8080/ambit2/feature/22200 \
-H subjectid:AQIC5wM2LY4SfcylizxgADoZWKKGeX4hM00KIt6VStFbvto.*AAJTSQACMDE.* \
-H Accept:text/uri-list \
http://alphaville.xs:8080/algorithm/mlr

The exact error message is:

org.opentox.jaqpot3.qsar.exceptions.BadParameterException: The parameter 'prediction_feature' is mandatory for this algorithm.
    at org.opentox.jaqpot3.qsar.trainer.MlrRegression.doParametrize(MlrRegression.java:108)
    at org.opentox.jaqpot3.qsar.trainer.MlrRegression.doParametrize(MlrRegression.java:77)
    at org.opentox.jaqpot3.qsar.AbstractTrainer.parametrize(AbstractTrainer.java:115)
    at org.opentox.jaqpot3.www.services.TrainingService.run(TrainingService.java:106)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

scaling & normalization should also be defined in PMML.
If the user provides a PMML file any preprocessing parameters will be omitted.

In http://opentox.ntua.gr:8080/algorithm/svm the dc:subjects are the ones of MLR. We may also add a comment with a link to https://github.com/opentox-api/curl-examples.

Set up the MVH web service (http://opentox.ntua.gr:8080/algorithm/mvh).

When user id is URL-encoded, 401 is returned. e.g. http://opentox.ntua.gr:8080/user/hampos%40opensso.in-silico.ch .

In Jaqpot3, http://opentox.ntua.gr:8080/algorithm/leverages is still down. Set up the training and prediction service.