opensilex / phis-ws

Web services REST for PHIS

• OpenSILEX Developer's installation
• Required software
• Check your installed software
• Download sources
• Build project
• Create opensilex command alias
  • Linux
  • Windows
• Setup configuration
• Setup Databases with docker
• Initialize system data
  • With Netbeans
  • With command line
• Start OpenSILEX development server with Netbeans
  • For web services only (with compiled Vue.js code)
  • For webservices and Vue.js hot reload server
• Start OpenSILEX development server with command line
  • For web services only (with compiled Vue.js code)
  • For webservices and Vue.js hot reload server
• Access to OpenSilex & tools
  • OpenSilex Application
  • RDF4J workbench
  • MongoDB
• Generate documentation
• Other maven build profiles and options
  • Generate release
  • Skip unit and integration tests
  • Skip Vue js build parts
  • Generate documentation
  • Generate documentation with security report audit
  • Check javascript security issues
  • Special profile for eclipse
  • Default configuration example

OpenSILEX Developer's installation

This repository contains source code for Phenotyping Hybrid Information System (PHIS) as an OpenSILEX instance

NOTE: OpenSILEX Production's installation available at: https://github.com/OpenSILEX/opensilex/blob/master/opensilex-doc/src/main/resources/installation/production.md

Required software

First you need to have this software installed :

• Java JDK 11+ (Our project is tested with JDK versions 11, 13 and 14)
• Maven 3.6+
• Git 2.17.1+
• docker 19.03.1+
• docker-compose 1.24.1+

Note: the <BASE_DIR> variable referenced in this documentation is the root folder of your installation where your specific user must have read and write permissions.

Check your installed software

Following commands should work from everywhere in your system without errors:

java -version

mvn --version

git --version

docker --version

docker-compose --version

Download sources

cd <BASE_DIR>
git clone https://github.com/OpenSILEX/opensilex.git
cd opensilex

Build project

cd <BASE_DIR>/opensilex
mvn install

Create opensilex command alias

Linux

Create the file .bash_aliases in your home directory if it doesn't exist.

Add this line to the file replacing the variable <BASE_DIR>:

alias opensilex=<BASE_DIR>/opensilex/opensilex-release/target/opensilex/opensilex.sh

Reload bash aliases and test it:

cd ~
source .bash_aliases
opensilex help

Windows

Add the following directory to your PATH environment variable replacing the variable <BASE_DIR>:

path %PATH%;<BASE_DIR>\opensilex\opensilex-release\target\opensilex\
opensilex help

Setup configuration

Edit <BASE_DIR>/opensilex/opensilex-dev-tools/src/main/resources/config/opensilex.yml

Be careful if you change the host and/or the port of databases as you will have to update the docker-compose configuration file accordingly.

Be sure to configure properly the read and write rights for your specific user on configured folders.

The only mandatory options to set up are:

• file-system.storageBasePath: Base directory for file storage

Setup Databases with docker

On Linux OS, you should add your current user <USER> to the docker group in order to avoid using sudo:

sudo usermod -aG docker <USER>

You need to log out and log in again to make it work.

cd <BASE_DIR>
cd opensilex/opensilex-dev-tools/src/main/resources/docker
docker-compose up -d

Docker containers will be automatically started on your machine at startup.

You can change the "restart" parameter in "docker-compose.yml" file if you don't want this behavior, but you will have to run the previous command after each restart manually.

Initialize system data

With Netbeans

Right-click on OpenSILEX project and select "Open Required Projects" --> "Open All Projects"

Then right-click on the class org.opensilex.dev.Install within the opensilex-dev-tools projet and select "run" or "debug".

If you want to reset all your databases, you can do the same with the class org.opensilex.dev.InstallReset.

With command line

opensilex dev install

If you want to reset all your database use -r flag with the previous command.

Start OpenSILEX development server with Netbeans

For web services only (with compiled Vue.js code)

Right-click on the class org.opensilex.dev.StartServer within the opensilex-dev-tools projet and select "run" or "debug".

For webservices and Vue.js hot reload server

Right-click on the class org.opensilex.dev.StartServerWithFront within the opensilex-dev-tools projet and select "run" or "debug".

Start OpenSILEX development server with command line

When using the command line you need to recompile all modules in order to have taken into account the changes in the Java source code (but not for Vue-JS code if using hot reload).

For web services only (with compiled Vue.js code)

opensilex dev start --no-front-dev

For webservices and Vue.js hot reload server

opensilex dev start

Access to OpenSilex & tools

OpenSilex Application

• OpenSilex Vue App is accessible at: http://localhost:8666/

• OpenSILEX API is accessible at: http://localhost:8666/api-docs

If you start the server with Vue.js, the hot reload server will tell you on which port it is accessible (probably 8080 depending of which is available).

Set the Default Super Admin user which will give you access to all web services is created with:

• login: [email protected]
• password: admin

RDF4J workbench

RDF4J workbench is accessible by default at: http://localhost:8667/rdf4j-workbench

Upon first connection, you must configure RDF4J server URL with this value: http://opensilex-rdf4j:8080/rdf4j-server

MongoDB

MongoDB is accessible on port 8668

You could download MongoDB Compass to manage your database

Generate documentation

cd <BASE_DIR>
mvn site -Pwith-test-report

All documentation will be available in <BASE_DIR>/opensilex-doc/src/main/doc directory.

Other maven build profiles and options

Generate release

mvn install -Drevision=X.Y.Z

A Zip file for revision will be available at <BASE_DIR>/opensilex-release/target/opensilex-release-X.Y.Z.zip

An uncompressed version is available in <BASE_DIR>/opensilex-release/target/opensilex-release-X.Y.Z/

Skip unit and integration tests

It is not recommended, but it can be useful if you need to rebuild very often at some point in order to save time.

mvn install -DskipTests=true

Skip Vue js build parts

This command can be useful if you work only on web services to speed up build.

mvn install -DskipFrontBuild

Generate documentation

Make sure the environment variable JAVA_HOME is defined.

cd <BASE_DIR>/opensilex
mvn site -Pwith-test-report -DskipFrontBuild

Generate documentation with security report audit

This build may be very very very long because it as to download a lot of stuff from internet (OWASP report).

It would be better to configure it periodically on an automated platform.

mvn site -Pwith-test-report -Pwith-security-check -DskipFrontBuild

Check javascript security issues

mvn verify -DskipFrontAudit=false

Special profile for Eclipse

If you are using Eclipse you may need to enable this profile to avoid build errors

mvn install -Pfor-eclipse

Default configuration example

# ------------------------------------------------------------------------------
# Base system configuration OpenSilex (OpenSilexConfig)
system:
  # Default application language (String)
  defaultLanguage: en 
# ------------------------------------------------------------------------------
# Configuration for module: FileStorageModule (FileStorageConfig)
file-system:
  # File storage service (FileStorageService)
  fs:
    # Service implementation class for: fs (FileStorageService)
    implementation: org.opensilex.fs.service.FileStorageService
    config:
      # Default file system storage (String)
      defaultFS: gridfs
      # Map of custom path connection management (Map<String,String>)
      # customPath:
      #   datafile/: irods
      # Map of file storage connection definition by identifier (Map<String,FileStorageConnection>)
      connections:
        # irods:
        #   # Service implementation class for: irods (FileStorageConnection)
        #   implementation: org.opensilex.fs.irods.IrodsFileSystemConnection
        #   config:
        #     # Base path for file storage (String)
        #     basePath: /FranceGrillesZone/home/fg-phenome/PHIS
        gridfs:
          # Service implementation class for: gridfs (FileStorageConnection)
          implementation: org.opensilex.fs.gridfs.GridFSConnection
          config:
            # MongoDB main host (String)
            host: localhost
            # MongoDB main host port (int)
            port: 8668 
            timezone: UTC
            # MongoDB database (String)
            database: opensilex 
        local:
          # Service implementation class for: local (FileStorageConnection)
          implementation: org.opensilex.fs.local.LocalFileSystemConnection
          config:
            # Base path for file storage (String)
            basePath: ../../opensilex-data


# ------------------------------------------------------------------------------
# Configuration for module: NoSQLModule (NoSQLConfig)
big-data:
  # MongoDB data source (MongoDBService)
  mongodb:
    # Service implementation class for: mongodb (MongoDBService)
    implementation: org.opensilex.nosql.mongodb.MongoDBService
    config:
      # MongoDB main host (String)
      host: localhost
      # MongoDB main host port (int)
      port: 8668
      # timezone (String)
      timezone: UTC
      # MongoDB database (String)
      database: opensilex
 

# ------------------------------------------------------------------------------
# Configuration for module: SPARQLModule (SPARQLConfig)
ontologies:
  # Platform base URI (String)
  baseURI: http://opensilex.dev/
  # SPARQL data source (SPARQLServiceFactory)
  sparql:
    # Service implementation class for: sparql (SPARQLServiceFactory)
    implementation: org.opensilex.sparql.rdf4j.RDF4JServiceFactory
    config: 
      # RDF4J repository name (String)
      repository: opensilex
      # RDF4J Server URI (String)
      serverURI: http://localhost:8667/rdf4j-server/
  # Platform base URI alias (String)
  baseURIAlias: dev
 
# ------------------------------------------------------------------------------
# Configuration for module: CoreModule (CoreConfig)
core:
  # Activate access logs by user (boolean)
  enableLogs: false

phis-ws's Issues

phenotypes service : rename phenotypes in dataset

rename :

• classes
• service name
• comments
• variables

Get or save file directly in Mongo with an InputStream object

Maybe use UploadFromStream from mongo driver 3.6. Prevent the use of temporary directory.

http://mongodb.github.io/mongo-java-driver/3.6/driver/tutorials/gridfs/
Save
// Get the input stream

try {
    InputStream streamToUploadFrom = new FileInputStream(new File("/tmp/mongodb-tutorial.pdf"));
    // Create some custom options
    GridFSUploadOptions options = new GridFSUploadOptions()
                                        .chunkSizeBytes(358400)
                                        .metadata(new Document("type", "presentation"));

    ObjectId fileId = gridFSBucket.uploadFromStream("mongodb-tutorial", streamToUploadFrom, options);
} catch (FileNotFoundException e){
   // handle exception
}

Load

ObjectId fileId; //The id of a file uploaded to GridFS, initialize to valid file id 

try {
    FileOutputStream streamToDownloadTo = new FileOutputStream("/tmp/mongodb-tutorial.pdf");
    gridFSBucket.downloadToStream(fileId, streamToDownloadTo);
    streamToDownloadTo.close();
    System.out.println(streamToDownloadTo.toString());
} catch (IOException e) {
    // handle exception
}

Change the URIs generations

Uris are generated counting elements in database and by incrementing this number. This solution does not work when some elements are deleted.

Concerned Uri generation

• agronomical object
• experiment
• variable
• trait
• method
• unit
• sensor
• vector
• images

NB : the generateNewInstanceUri() must be updated to static

Migrate from OpenAPI Specification 2 to OpenAPI Specification 3 (Swagger 1.5 to 2.0)

Java implementation that support OpenAPI 3.0:

• Swagger-Core-JAX-RS-Project-Setup-2.0.X

There're a lot of new features that were added to this version, such as:

• Added support for multiple root URLs.
• Added support for content type negotiation.
• Extended JSON Schema support to include oneOf, anyOf and not support.
• Added a callback mechanism to describe Webhooks.

Helpers :

• Implementations of OpenAPI

service to get skos relations list

create a service which will return the list of the skos relations that the user can use to define relations between concepts

Variables - non required comments and empty comments

The GET variable service returns only the variables (with traits, methods and units) which has comments. It needs to be corrected.

In the same time, only the Variables are returned and not the ObservationVariables. We have to update the web service to return any variable subclass of Variable concept.

Concern :

• the web service : OpenSILEX/phis-ws
• the webapp : OpenSILEX/phis-webapp

Usefull to :

• insert the public data of the paper

Update User creation (add URI)

Add user URI when a user is created due to new URI generator #45

Change the uri format

A standardized format provided is now existing. The generated uri formats must be updated accordingly.

Uri formats to update :

• agent
• agronomical object
• annotation
• event
• experiment
• image
• method
• sensor
• species
• trait
• unit
• user
• variable
• variety
• vector

Usefull to :

• respect the provided standards

Ontology - change to oepo

Change the ontology uri to oepo such as the ontology of the paper.

Concerns

• the ontology : OpenSILEX/ontology-phis-oepo-field
• the web service : OpenSILEX/phis-ws
• the client : OpenSILEX/phis-webapp

Usefull to

• respect the ontology of the paper

Add the automatic generation of the files headers

The automatic generation of the files headers has been removed (#38).
It needs to be added again

Change logger variable name to uppercase.

Change
final static Logger logger = LoggerFactory.getLogger({className})
To
final static Logger LOGGER = LoggerFactory.getLogger({className})

There are some wrong services attributes types (DTO and GET request parameters)

Wrong type for some requested attribute. For example, wrong type for depth attribute. Boolean type value needed like true and String type given "true".

Services to update :

• GET /uri/{uri}/instances (depth)
• GET /users (admin, available)
• POST /layers
• GET /users/{email}
• GET /users
• POST /users
• PUT /users

Add verification functions in rules

Improve DTO verifcations rules. At this time, the rules function only check if field are empty or not by passing boolean to HashMap Key/value. It maybe interesting to apply a specific verification function to a DTO field.

Need to create a Rules class

Ideas :

• Add needed boolean verification
• Add independant function verification

Services GET of ontologies properties groups

Create services which will return a list of properties by type to have clients adapted to the web service.

Groups of properties :

• RDFS /vocabularies/rdfs/properties
  • GET
    • rdfs:label
    • rdfs:comment
• SKOS
• Devices
  • comments
  • labels
• DublinCore
• Contacts
  • comments
  • labels

InstanceDefinition concept

• create a DAO for the InstanceDefinition (if possible)
• clean the others DAO such as TraitDAO which use then concept of InstanceDefinition

Check paginations

Check and correct the pagination of each service.

Check :

• Projects
• Experimentations
• Agronomical objects
• Datasets
• Documents
• Groups
• Images
• Methods
• Traits
• Units
• Variables
• Sensors
• Vectors
• Users
• AcquisitionSession

Needs correction :

• Project : the pagination does not work
• Experimentation : the pagination does not work
• Agronomical Object : count instead of getting all the results
• Documents : count instead of getting all the results
• Groups : the pagination does not work, totalCount always equals to the pageSize
• Images : count instead of getting all the results
• Methods : count instead of getting all the results
• Traits : count instead of getting all the results
• Units : count instead of getting all the results
• Variables : count instead of getting all the results
• Sensors : count instead of getting all the results
• Vectors : count instead of getting all the results
• Users : the pagination does not work, totalCount always equals to the pageSize
• [] AcquisitionSession : the pagination does not work

Translate comments in english

Add annotation concepts

Need to create URI for annotation, agent and event concepts:

See #45

Annotation service

• Create GET Annotation method
• Create POST Annotation method
• Integrate Annotation service

Create object with uri and rdfType

Most of models have the uri and rdfType attributes.
A class with uri and rdfType should be created and reused

fix the respect of REST norms problem on the POST dataset

The POST dataset creates provenance and dataset an returns the provenance uri. It does not respect the REST norms. It will be needed to be fixed, maybe by creating a provenance service.

rename classes XXXDao : XXXDAO

for example :
ExperimentDao ---> ExperimentDAO

Create differents POST DTO and a PUT DTO

the same DTO is currently used for PUT and POST but PUT and POST does not require the same parameters...

Needs to list which DTO has to be created :

DTO to create (to continue fill) :

• OntologyReferenceDTO

Modify rules on some DTO

Rules key need to be a string that corresponding to a DTO attribute name.

Concerned DTO :

• AgronomicalObjectDTO
• DatasetDTO
• ExperimentDTO
• GroupDTO
• InstanceDefinitionDTO
• LayerDTO
• ProjectDTO
• PropertyDTO
• PropertyVocabularyDTO
• SensorProfileDTO
• TripletDTO
• UserDTO
• VariableDTO

Integrate and test validation rules

List all verifications done before integrating rules : (#76)

• Integration

DTO Checked

• All

Resource checked:

• AgronomicalObjectResourceService
• DatasetResourceService
• DocumentResourceService
• ExperimentResourceService
• GroupResourceService
• ImageResourceService
• LayerResourceService
• MethodResourceService
• ProjectResourceService
• SensorResourceService
• TokenResourceService
• TraitResourceService
• TripletsResourceService
• UnitResourceService
• UriResourceService
• UserResourceService
• VariableResourceService
• VectorResourceService
• VocabularyResourceService
• CallResourceService
• StudyDetailResourceService
• AnnotationResourceService

Annotation created and documented:

• Date
• ProvenanceDate
• Required
• URL
• GroupLevels
• GrantTypes

Crop Species

Fonctionnality Crop Species :

• Create
• Read
• Update
• Link to Experiments
• Link to Agronomical Objects

Logger query trace to debug

To get less log messages in production mode, use Logger::debug() instead of Logger::trace() for the queries to the databases.

RessourceUtils Date functions management

Need to move convertStringToDateTime(String stringDate, String pattern) function from RessourceUtils to Dates classes due to jwt merge.

SQLQueryBuilder - add create and update

Create a query builder for the queries of create and update in SQL

URINamespaces - relations attribute : rename the keys

the r information on the keys is redundant. It needs to be removed : rHasDocument -> hasDocument

Using slash in document title prevent to retreived a document

#40 May resolve this one. When a slash is using in the document name the path to the document is altered.

Manage, verify and update pom dependencies.

Dependencies must be checked in pom file to check unused dependencies.

• Check all dependencies
• Update all dependencies
• Test WS

Validate get parameters sent to a service

[POST] token (/brapi/v1/token) - fill userDisplayName attribute

the userDisplayName is actually "null null"

POST Variables - error when null trait method or unit

When a call to the POST variables is done and that a trait, method or unit has a null value, an error occured

Experiments [GET] : add search by project

Add to the GET of the experiments service a search by project(s)

URINamespaces : rename the attribute objects in concepts

Rename OBJECTS in CONCEPTS and change the key of all the values : remove the c. (e.g cDocuments -> documents)

Add properties filename constants to Properties Manager

We need to had constant variables as done in UriGenerator :
e.g. PropertiesFileManager.getConfigFileProperty(PROPERTIES_SERVICE_FILE_NAME, PROPERTIES_SERVICE_BASE_URI);
instead of use raw string like :
PropertiesFileManager.getConfigFileProperty("service", "baseURI")

These constant variables must be in PropertiesFileManager.class .

Create required folders in the web service

Currently, some folders are not automatically created by the web service and needs to be created when a new phis instance is deployed. We needs to create automatically thoses folders using the web service.

• developped
• tested
• integrated

Concerned folders :

1. documents

• developped
• tested
• integrated

2. phisLayers

• developped
• tested
• integrated

3. images

• developped
• tested
• integrated

Change the name of the package "brapi"

We should maybe change the name of the package "brapi" (phis2ws.service.view.brapi)

Add calls service in order to implement BRAPI API specifications

See service definition at :
https://brapi.docs.apiary.io/#reference/calls

Users - use uri instead of email

Use the uri instead of the emails as id for the users.

Usefull to :

• insert the public data of the paper

Create a ProvenanceDAO

check each service return respect the REST norms

A few services of the web service does not respect de REST norms.

1. List all services which does not respect the REST norms
2. Fix them

Update the License header with the new format

See the coding styles of the docs-community-dev.

URINamespaces split the context array : prefixes and contexts

The context array has two differents kind of data : prefixes and contexts. It needs to be split into two arrays :

• prefixes : rename the keys : pxPlatform -> platform
• contexts

Use a recommended date format (xsd:DateTime) for clients and the server

For now, we use a wrong date time format (yyyy-MM-dd HH:mm:ssZ).
We must use the recommended format xsd:DateTime.
We must add 'T'separator indicating that time-of-day follows the first part of the date.
e.g. (yyyy-MM-dd'T'HH:mm:ssZ).

This change will impact the all API clients (R, Python, Web app), internal WS search method parameters and class models with datetime attribute.

Superclass for the classes ResourceService

Create a superclass for the ResourceService with :

• noResultFound()
• queryError()
• sqlError()
• getData()

Add a composer in the webapp part

Add a composer to include javascript libraries in the web app part

Rename classes Resultat

• classes ResultatXXX must be in english : ResultXXXX

Add the support of xsd types in SPARQL

support xsd types in the SPARQL queries (update query builders ?).