magbak / otit_swt

Tools that make it easier to use Semantic Web Technologies in industrial settings: creating models that contextualize sensor data and querying that data

OT/IT with Semantic Web Technologies (SWT)

Now split in two and moved to chrontext and stOTTRs

This repository provides experimental tools for moving between the world of Operational Technology (OT) to the world of Information Technology (IT).

• Model Mapper: - An implementation of stOttr with extensions for mapping asset structures based on the Epsilon Transformation Language. Implemented with Apache Arrow in Rust using Pola.rs.
• Hybrid Query Engine: SPARQL- and Apache Arrow-based high throughput access to time series data residing in an arbitrary time series database which is contextualized by a knowledge graph. Built in Rust using pola.rs, spargebra, sparesults and oxrdf from the Oxigraph project.
• Domain Specific Query Language: A customizable query language for accessing time series data using simple generalized paths such as those found in the Reference Designation System or in OPC UA information models. The DSQL is parsed with nom and translated to the Hybrid Query language.

Currently, these tools are volatile works in progress, and should not be used by anyone for anything important.

Queries

We can make queries in Python. The code assumes that we have a SPARQL-endpoint and an Arrow Flight SQL-endpoint (Dremio) set up.

import pathlib
from otit_swt_query import Engine, ArrowFlightSQLDatabase, TimeSeriesTable

engine = Engine(OXIGRAPH_QUERY_ENDPOINT)
    tables = [
        TimeSeriesTable(
            schema="my_nas",
            time_series_table="ts.parquet",
            value_column="v",
            timestamp_column="ts",
            identifier_column="id",
            value_datatype="http://www.w3.org/2001/XMLSchema#unsignedInt")
    ]
arrow_flight_sql_database = ArrowFlightSQLDatabase(host=DREMIO_HOST, port=DREMIO_PORT, username="dremio",
                                                   password="dremio123", tables=tables)
engine.set_arrow_flight_sql(arrow_flight_sql_database)
df = engine.execute_hybrid_query("""
PREFIX xsd:<http://www.w3.org/2001/XMLSchema#>
PREFIX otit_swt:<https://github.com/magbak/otit_swt#>
PREFIX types:<http://example.org/types#>
SELECT ?w ?s ?t ?v WHERE {
    ?w a types:BigWidget .
    ?w types:hasSensor ?s .
    ?s otit_swt:hasTimeseries ?ts .
    ?ts otit_swt:hasDataPoint ?dp .
    ?dp otit_swt:hasTimestamp ?t .
    ?dp otit_swt:hasValue ?v .
    FILTER(?t > "2022-06-01T08:46:53"^^xsd:dateTime && ?v < 200) .
}
    """)

Mapping

We can easily map DataFrames to RDF-graphs using the Python library.

from otit_swt_mapper import Mapping
#We use polars dataframes instead of pandas dataframes. The api is pretty similar.
import polars as pl

#Define a stOttr document with a template:
doc = """
    @prefix ex:<http://example.net/ns#>.
    ex:ExampleTemplate [?MyValue] :: {
    ottr:Triple(ex:myObject, ex:hasValue, ?MyValue)
    } .
    """

#Define a data frame to instantiate:
df = pl.DataFrame({"Key": ["A", "B"],
                   "MyValue": [1, 2]})
#Create a mapping object
mapping = Mapping([doc])
#Expand the template using the data in the dataframe
mapping.expand("http://example.net/ns#ExampleTemplate", df)
#Export triples
triples = mapping.to_triples()
print(triples)

Results in:

[<http://example.net/ns#myObject> <http://example.net/ns#hasValue> "1"^^<http://www.w3.org/2001/XMLSchema#long>, 
 <http://example.net/ns#myObject> <http://example.net/ns#hasValue> "2"^^<http://www.w3.org/2001/XMLSchema#long>]

An example mapping is provided in this jupyter notebook.

The Python API is documented here

Installing pre-built wheels

From the latest release, copy the appropriate .whl-file for your system, then run:

pip install https://github.com/magbak/otit_swt/releases/download/v0.1.5/otit_swt_mapper-0.1.12-cp310-cp310-manylinux_2_31_x86_64.whl
pip install https://github.com/magbak/otit_swt/releases/download/v0.1.5/otit_swt_query-0.1.12-cp310-cp310-manylinux_2_31_x86_64.whl

All code is licensed to Prediktor AS under the Apache 2.0 license unless otherwise noted, and has been financed by The Research Council of Norway (grant no. 316656) and Prediktor AS as part of a PhD Degree.