stellio-hub / stellio-context-broker

Stellio is an NGSI-LD compatible context broker

Stellio context broker

Stellio is an NGSI-LD compliant context broker developed by EGM. NGSI-LD is an Open API and Datamodel specification for context management published by ETSI).

Stellio is a FIWARE Generic Enabler. Therefore, it can be integrated as part of any platform “Powered by FIWARE”. FIWARE is a curated framework of open source platform components which can be assembled together with other third-party platform components to accelerate the development of Smart Solutions. For more information check the FIWARE Catalogue entry for Core Context. The roadmap of this FIWARE GE is described here.

You can find more info at the FIWARE developers website and the FIWARE website. The complete list of FIWARE GEs and Incubated FIWARE GEs can be found in the FIWARE Catalogue.

NGSI-LD Context Broker Feature Comparison

The NGSI-LD Specification is regularly updated and published by ETSI. The latest specification is version 1.8.1 which was published in April 2024.

• An Excel file detailing the current compatibility of the development version of the Stellio Context Broker against the features of the 1.8.1 specification can be downloaded here

📚 Documentation	🐳 Docker Hub	🎯 Roadmap

Overview

Stellio is composed of 2 business services:

• Search service is in charge of managing the information context and handling the temporal (and geospatial) queries, it is backed by a TimescaleDB database
• Subscription service is in charge of managing subscriptions and subsequent notifications, it is backed by a TimescaleDB database

It is completed with:

• An API Gateway module that dispatches requests to downstream services
• A Kafka streaming engine that decouples communication inside the broker (and allows plugging other services seamlessly)

The services are based on the Spring Boot framework, developed in Kotlin, and built with Gradle.

Quick start

A quick way to start using Stellio is to use the provided docker-compose.yml file in the root directory (feel free to change the default passwords defined in the .env file):

docker-compose up -d && docker-compose logs -f

It will start all the services composing the Stellio context broker platform and expose them on the following ports:

• API Gateway: 8080
• Search service: 8083
• Subscription service: 8084

Please note that the environment and scripts are validated on Ubuntu and macOS. Some errors may occur on other platforms.

We also provide a configuration to deploy Stellio in a k8s cluster. For more information, please look in the stellio-k8s project

Docker images tagging

Starting from version 2.0.0, a new scheme is used for tagging of Docker images:

• Releases are tagged with the version number, e.g., stellio/stellio-search-service:2.0.0
• latest tag is no longer used for releases as it can be dangerous (for instance, triggering an unwanted major upgrade)
• On each commit on the develop branch, an image with the latest-dev tag is produced, e.g., stellio/stellio-search-service:latest-dev

The version number is obtained during the build process by using the version information in the build.gradle.kts file.

Development

Developing on a service

Requirements:

• Java 21 (we recommend using sdkman! to install and manage versions of the JDK)

To develop on a specific service, you can use the provided docker-compose.yml file inside each service's directory, for instance:

cd search-service
docker-compose up -d && docker-compose logs -f

Then, from the root directory, launch the service:

./gradlew search-service:bootRun

Running the tests

Each service has a suite of unit and integration tests. You can run them without manually launching any external component, thanks to Spring Boot embedded test support and to the great TestContainers library.

For instance, you can launch the test suite for entity service with the following command:

./gradlew search-service:test

Building the project

To build all the services, you can just launch:

./gradlew build

It will compile the source code, check the code quality (thanks to detekt) and run the test suite for all the services.

For each service, a self executable jar is produced in the build/libs directory of the service.

If you want to build only one of the services, you can launch:

./gradlew search-service:build

Committing

Commits follow the Conventional Commits specification.

Code quality

Code formatting and standard code quality checks are performed by Detekt.

Detekt checks are automatically performed as part of the build and fail the build if any error is encountered.

• You may consider using a plugin like Save Actions that applies changed code refactoring and optimized imports on a save.

• You can enable Detekt support with the Detekt plugin.

Working locally with Docker images

To work locally with a Docker image of a service without publishing it to Docker Hub, you can follow the below instructions:

• Build a tar image:

./gradlew search-service:jibBuildTar

• Load the tar image into Docker:

docker load --input search-service/build/jib-image.tar

• Run the image:

docker run stellio/stellio-search-service:latest

Releasing a new version

• Merge develop into master

git checkout master
git merge develop

• Update version number in build.gradle.kts (allprojects.version near the bottom of the file)
• Commit the modification using the following template message

git commit -am "chore: upgrade version to x.y.z"

• Push the modifications

git push origin master

The CI will then create and publish Docker images tagged with the published version number in https://hub.docker.com/u/stellio.

• On GitHub, check and publish the release notes in https://github.com/stellio-hub/stellio-context-broker/releases

Usage

To start using Stellio, you can follow the API quick start.

Minimal hardware requirements needed to run Stellio

The recommended system requirements may vary depending on factors such as the scale of deployment, usage patterns, and specific use cases. That said, here are the general guidelines for the minimum computer requirements:

• Processor: Dual-core processor or higher
• RAM: 4GB or higher (1.8GB is needed to just run it)
• Storage: At least 4GB of free disk space (3.8GB is needed to just run it)
• Operating System: Linux (recommended), macOS (also recommended), or Windows

Please note that these requirements may vary based on factors such as the size of your dataset, the number of concurrent users, and the overall complexity of your use case.

Further resources

For more detailed explanations on NGSI-LD or FIWARE:

• NGSI-LD v1.8.1 Specification
• NGSI-LD Primer
• NGSI-LD Tutorials
• FIWARE Training

License

Stellio is licensed under APL-2.0.

It mainly makes use of the following libraries and frameworks (dependencies of dependencies have been omitted):

Library / Framework	Licence
Spring	APL v2
Titanium JSON-LD	APL v2
Reactor	APL v2
Jackson	APL v2
JUnit	EPL v2
Mockk	APL v2
JsonPath	APL v2
WireMock	APL v2
Testcontainers	MIT

© 2020 - 2024 EGM

stellio-context-broker's Issues

Implement endpoint to delete an entity attribute (6.7.3.2)

Check entity type names, property names and relationship names (4.6.2)

As per clause 4.6.2. of the specification

Setup Testcontainers (with docker compose module) to run entity-service tests

Currently, tests in entity service are running using the embedded Neo4j database.
Since docker compose is already used in the serivce, this could be improved by integrating Testcontainers with docker compose module.

Improve queries in entity creation process

Current version does the job but is not very efficient in terms of queries made to the DB.

Improve that and also create some minimal indexes.

Use Spring WebFlux Tests for controllers testing

Currently, we are not using Spring WebFlux Tests for controllers testing.
With @WebFluxTest annotation, regular @component beans (unecessary at the web layer) will not be scanned.

Upgrade to Spring Boot 2.3

Starting a list of things to deal with when upgrading:

• https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#howto-testcontainers

Finalize integration of Testcontainers in all modules

Spring provides some test support with embedded DBs (e.g. for neo4j) or with some mockked abstractions (e.g. for Kafka), but they are not so easy to integrate and prone to break.

As it is now validated, use the docker-compose integration provided by Testcontainers, that allows us to reuse our existing configs without duplication effort.

Finally, make sure the integration is consistent between all the modules (this could even be - at least partially - in the shared lib).

Migrate and update the quick guide

Currently, the quick guide referenced in the README.md points to the one located in https://github.com/easy-global-market/ngsild-api-data-models.

This is prone to error as we may (and we did!) forget to update it when there are changes to the API and is also confusing for users that have to jump to another site to quick start using the project.

So the purpose of this issue is to bootstrap a new quick guide in this repository, with samples data in a samples directory, based on beekeeping data model.

Make something oriented like a real use-case (and not a showcase of requests like the existing one). For instance:

• Create context information (beekeeper, beehive, ...), atomic and / or in batch
• Perform 2 or 3 simple queries
• Create a subscription on update to values of properties of a beehive (and explain how to get notified)
• Update values of properties of a beehive
• Query the temporal evolution of the properties of a beehive

Improve entity handling process and introduce minimal validation

• Ensure an NGSI-LD entity has an id and a type at creation time
• Avoid an extra JSON parsing to bootstrap the Entity node
• Remove useless (and error prone) entity type checking on patch / update operations
• Move entity compaction method to ExpandedEntity class
• Rename neo4jservice to entityService

Find why the use of a jarcache for JSON-LD Java raises a ClassNotFoundException

When setting up a jarcache.json in the classpath, it is seen as expected by JSON-LD Java.

However, its use raises a ClassNotFoundException from the library as it does not find some shaded com.google... classes (to be reproduced to have the exact stacktrace).

Not blocking but not optimal for performance (even if the library has a local cache when running).

Implement handling of delete operations for subscriptions

Upgrade to Neo4j 4.0

Process is fairly easy: https://neo4j.com/blog/graphcast-migrating-from-neo4j-3-5-to-4-0

Basically:

• Update binaries
• Move DB data and rename default database
• Activate migration in neo4j conf
• Ensure JDK11+ (we already do)

Fix http client dependency conflicts

The liquigraph added in the following commit 8297c11 to manage neo4j migrations results version conflicts of the apache http client.
This raises method not found error when using jsonldjava.

Add entityOperations/upsert endpoint

As per clause 6.15.3.1 of the specification

Add support for throttling parameter on notifications

Finish implementation of HTTP response common requirements (6.3.6)

The content type of the response is well set on most responses.

However it has still be implemented in some queries (e.g. search on entities or list of subscriptions that return a text/plain content type)

Do not load TestContainersConfiguration when not needed

The TestContainersConfiguration is loaded for tests that don't need it.
This should be loaded only in tests needing a Neo4J

Only return createdAt / modifiedAt if asked by consumers (5.2.2)

add postman collection for quick guide

Following discussion in PR #68 : #68 (comment)
We agreed to have a postman collection for this guide to help users not used to cli.

Implement endpoint for batch entity update (6.16.3.1)

Refactor and improve the batch entity creation process

Summary of refactorings to be done in next sprint:

• Move new methods introduced in ParsingUtils into NgsiLdParsingUtils (they are parsing some NGSI-LD payload)
• Improve readability of methods signatures and code (e.g. for complex pairs containing a map and a list)
• Rename ValidationUtils to something more adapted to what it is doing (indeed it is not doing any validation). Maybe in a EntityOperationService ?
• Improve the processing of the graph of entities being created

Quick thoughts about processing of entities:

• Basically we are working with a (potentially circular) graph of entities
  • First step is to build it
  • It can be done manually via something like an adjency list or via the use of a library (for instance, Guava has such a datastructure) : to be decided together
• In this graph, we have to identify:
  • Referenced entities provided in the payload (they will have to be created first, starting from the leaves)
  • Referenced entities already existing in the DB (ok, similar to a leaf)
  • Referenced entities that do not exist (this is a 400 for the owning entity)
  • Circular dependencies : an entity for which there is a path leading to herself, marked at the higher (closest to the root) level possible in the graph
• Starting from the leaves, we can create entities that have no dependencies to a not-yet created entity from the payload
  • Then going up in the tree until the root entities
  • Postponing the creation of the incoming relationship on circular entities until all entities are created

Add support for JSON-LD structured value (4.6.3)

Integrate static code analysis with Detekt

• In complement to ktlint which is already integrated
• Cf https://detekt.github.io/
• Also make use of the provided SonarQube integration

Add support for attributes and format on notifications sent

Add support for operationSpace and observationSpace in Entity model (4.7)

Extract and store location of each attribute instance

Upon the reception of an entity event from Kafka, extract the location of the entity and add it to each created attribute instance.

Review and ensure modifiedAt information is updated whenever it needs to

• Not updated at entity level when an attribute is deleted
• Not updated at entity level when an attribute is patch with attrs

Add CONTRIBUTING.md file

Help to compose this file : https://help.github.com/en/github/building-a-strong-community/setting-guidelines-for-repository-contributors#adding-a-contributing-file

Fix temporal values injection

related to #1
The temporal values injection in the search service fails with the following error:
class java.time.OffsetDateTime cannot be cast to class java.time.ZonedDateTime (java.time.OffsetDateTime and java.time.ZonedDateTime are in module java.base of loader 'bootstrap')

As in the error detail, there is a type mismatch.
More precisely, the attributes observed_at values are returned as OffsetDateTime from the Postgres DB and should be casted correctly to ZonedDateTime.

Fix http request preconditions to accept a Mime Type with parameters

Currently, a request sent with the following content type header 'application/ld+json;charset=UTF-8' is not accepted.
The behaviour of the http request preconditions should be fixed to accept a Mime Type with parameters.

Fix temporal properties data type

Currently the temporal properties (createdAt, modifiedAt and observedAt) data type is OffsetDateTime that should be changed to ZonedDateTime to support timezone information.

Store entity payload in a dedicated table

Upgrade TimescaleDB extension to 1.7.0

Implement endpoint for batch entity delete (6.17.3.1)

Fix error response when creating an entity having non-existent relationship object

When creating an entity having a non-existent relationship object, the expected error response is a 400 with the detail (Target entity $objectId does not exist, create it first).

However, the error response is a 500 with the following payload

{
    "detail": "There has been an error during the operation execution",
    "title": "There has been an error during the operation execution",
    "type": "https://uri.etsi.org/ngsi-ld/errors/InternalError"
} 

Review and improve Cypher queries to apply best practices for efficiency

Check entity content does not use a prohibited character (4.6.4)

As per clause 4.6.4 of the specification

Add support for the simplified representation of an entity (4.5.4)

Via support for the keyValues value in options parameter.

Will have to be supported in every request returning some sort of results. Let's start with an implementation in entity service.

Improve the update of the subscription raised notification

When a new notification is raised by the subscription service, the related subscription is updated in the context information. This operation can be improved (it should be possible to go from 4 queries to 2).

Wrap JSON-LD expanded entity into a class

Instead of manipulating clumsy structures like a Pair(Map<String,Any>,List<String>), wrap such a structure into a class so that methods contracts and associated manipulations become more straightforward.

Implement all rules related to HTTP request pre-conditions

Some of the rules are already implemented, the task consists in implementing all the missing ones, all the code should go into the shared library.

For reference in the specification:

• API operations definitions: 5.5.2, 5.5.3
• HTTP binding: 6.3.2, 6.3.3, 6.3.4

Implement common rules related to error types

Based on architecture set up in done bbf536e

Some of the rules are already implemented, the task consists in implementing all the missing ones, all the code should go into the shared library.

Implement handling of Update and Delete operations for subscriptions

Search service fails to create new temporal property

The search service fails to create new temporal properties (Implements 6.20.3.1) for the following reasons:

• The call to addAttributeInstances() service function in the TemporalEntityHandler should be flatMapped and not mapped.
• We are storing the attributeName in an expanded format, but using the short type to call the getForEntityAndAttribute() service function.
  -> No temporalEntityAttributeUuid is returned.
• In the addAttributeInstances service function, we are calling the getPropertyValueFromMapAsDateTime() util function with the EGM_OBSERVED_BY parameter (https://ontology.eglobalmark.com/egm#observedBy), but I saw that it was expanded as (https://uri.etsi.org/ngsi-ld/observedAt)

return body
            .flatMapMany {
                Flux.fromIterable(expandJsonLdFragment(it, contextLink).asIterable()) ------> here expanded as https://uri.etsi.org/ngsi-ld/observedAt
            }
            .flatMap {
                temporalEntityAttributeService.getForEntityAndAttribute(entityId, it.key.extractShortTypeFromExpanded()) ------> here called with short type
                    .map { temporalEntityAttributeUuid ->
                        Pair(temporalEntityAttributeUuid, it) }
            }
            .map {
                attributeInstanceService.addAttributeInstances(it.first,
                    it.second.key.extractShortTypeFromExpanded(), ------> here called with short type
                    expandValueAsMap(it.second.value))
            }
            .collectList()
            .map {
                ResponseEntity.status(HttpStatus.NO_CONTENT).build<String>()
            }

Format all files aligned with ktlint and share the format settings

All files are not formatted the same way.
Currently we don't have any format configuration shared for the developers to matter the minimum about formatting.

We should format all files and share an Intellij configuration for developers to contribute better on the project.

Internal error when BadRequest exception occurs

When doing that request :

POST /ngsi-ld/v1/entities HTTP/1.1

{
  "id": "urn:ngsi-ld:Vehicle:A12388",
  "type": "Vehicle",
  "speed": {
    "type": "Property"
  },
  "@context": [
    "https://raw.githubusercontent.com/easy-global-market/ngsild-api-data-models/master/shared-jsonld-contexts/egm.jsonld",
    "https://raw.githubusercontent.com/easy-global-market/ngsild-api-data-models/master/aquac/jsonld-contexts/aquac.jsonld",
    "http://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld"
  ]
}

The response should be an error BadRequestData (400 http) like that, helping the user figure out what's wrong in the request :

{
    "detail": "Key $NGSILD _PROPERTY_VALUE not found in $propertyValues").
    "type": "https://uri.etsi.org/ngsi-ld/errors/BadRequestData",
    "title": "The request includes input data which does not meet the requirements of the operation"
}

But today we get this : (500 http)

{
    "detail": "There has been an error during the operation execution",
    "type": "https://uri.etsi.org/ngsi-ld/errors/InternalError",
    "title": "There has been an error during the operation execution"
}

Investigation :
This happens because the error occurs in a function with @transactional annotation which seems to not pass correctly the exception.

Perform case-insensitive search of a sensor / device by vendorId

As the case may vary, perform a case-insensitive search to be sure to get the associated sensor / device.

Ensure Neo4j is available before starting entity service

Currently there is no synchronization between entity service and neo4j at application startup.

When launched, the entity service cannot connect to neo4j (it is not started yet) which blocks database migrations.

Add @context in events sent to Kafka

Consumers may need to expand the operation payload received in an event.

Currently, they have to extract it from the entity payload, which implies the JSON parsing of the entity and that's not efficient.

A possible solution is to add the @context as a (mandatory) field in the event message.

This is now implemented as part of the Authorization PR.

As a consequence, remove the @context information from the payload field in the message (not from the entity one)

Handle users permissions when requesting subscriptions instances

To be added:

• Check on the subscription sub (subject of the users token) before updating/deleting a subscription
• Manage http responses (return 403 forbidden when users are not allowed to request the subscription instance)