GithubHelp home page GithubHelp logo

jimjonesbr / lod4wfs Goto Github PK

View Code? Open in Web Editor NEW
13.0 4.0 7.0 57.17 MB

Server application for publishing Geographic Linked Open Datasets via Web Feature Services.

License: MIT License

Shell 0.26% R 6.57% Java 92.35% JavaScript 0.82% CSS 0.01%

lod4wfs's Introduction

LOD4WFS - Linked Open Data for Web Feature Services Adapter (Beta)

Note

This project has been discontinued. If you wish to load geographical data from RDF triplestores into GIS, please take a look at the PostgreSQL Foreign-data Wrapper rdf_fdw.

The LOD4WFS Adapter (Linked Open Data for Web Feature Services) is a service to provide access to Linked Geographic Data from Geographical Information Systems (GIS). It implements a service which listens to WFS requests and converts these requests into the SPARQL Query Language for RDF. After the SPARQL Query is processed, the LOD4WFS Adapter receives the RDF result set from the Triple Store, encodes it as a WFS XML document, and returns it to the client, e.g. a GIS. This approach enables current GIS to transparently have access to geographic LOD data sets, using their implementation of WFS, without any adaptation whatsoever being necessary. In order to reach a higher number of GIS, the currently most common implementation of WFS has been adopted for the LOD4WFS Adapter, namely the OGC Web Feature Service Implementation Specification 1.0.0. It provides the basic functions offered by the WFS reference implementation, GeoServer.

LOD4WFS Documentation

The LOD4WFS Adapter enables access to geographic LOD data sets in two different ways: Standard Data Access (SDA) and Federated Data Access (FDA). To cut a long story short: SDA very easy to use but, but not very flexibel. FDA a bit more complex to use (you have to write you own SPARQL Queries), but very flexibel.

Click here to download the latest stable releases.

Standard Data Access (SDA)

The Standard Data Access feature was designed in order to enable access to all geographic LOD data sets contained in a triple store. This feature basically works as an explorer, looking for geographic LOD data sets from a certain Triple Store and making them available via WFS. Due to the possibility of describing different types of geometries (polygons, lines, points) and many different coordinate reference systems, which are characteristic requirements of a WFS, we chose by default the GeoSPARQL Vocabulary as an input requirement for the Standard Data Access feature. The example bellow shows how geometries and their related attributes are expected to be structured. The geometries are encoded as WKT literals and the attributes of Features are linked to the instance of the geo:Geometry class via RDF Schema and Dublin Core Metadata Element Set vocabularies. However, there are no constraints on which vocabularies or properties may be used for describing attributes. For a full version of the data set click here.

@prefix geo: <http://www.opengis.net/ont/geosparql/1.0#>.
@prefix my: <http://ifgi.lod4wfs.de/resource/>.
@prefix sf: <http://www.opengis.net/ont/geosparql#>.
@prefix dc: <http://purl.org/dc/elements/1.1/>.
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>.
@prefix xsd: <http://www.w3.org/2001/XMLSchema#>.

my:FEATURE_RECIFE a geo:Feature ;
	rdf:ID "2611606"^^xsd:integer ;
	dc:description "Recife"^^xsd:string ;
	geo:hasGeometry my:GEOMETRY_RECIFE .
my:GEOMETRY_RECIFE a geo:Geometry ;
	geo:asWKT "POLYGON ((-35.0148559599999984 -8.0564907399999992,-34.9939074400000010 -8.0493884799999993, ...
-35.0148559599999984 -8.0564907399999992))"^^sf:wktLiteral .

In order to make the data sets discoverable via the Standard Data Access feature, additional metadata must be added to the data sets. We make use of Named Graphs for this purpose. Every Named Graph in the LOD data source must contain only objects of the same feature type. This approach facilitates the discovery of Features, speeding up queries that list the Features available in the triple store. In case a Named Graph contains multiple Feature types, the features can be split into different layers using the Federated Data Access (see next section). Finally, each Named Graph needs to be described by certain RDF properties, namely abstract, title and subject from the Dublin Core Terms Vocabulary. These properties help the adapter to classify all Features available in a Triple Store, so that they can be further on discovered by the WFS client through the WFS Capabilities Document.

@prefix dct: <http://purl.org/dc/terms/>.
@prefix xsd: <http://www.w3.org/2001/XMLSchema#>.

<http://ifgi.lod4wfs.de/graph/municipalities> dct:abstract "Municipalities of the Brazilian Federal States."^^xsd:string .
<http://ifgi.lod4wfs.de/graph/municipalities> dct:title "Brazilian Municipalities"^^xsd:string .
<http://ifgi.lod4wfs.de/graph/municipalities> dct:subject "municipalities boundaries"^^xsd:string .

Federated Data Access (FDA)

The Federated Data Access feature offers the possibility of accessing geographic LOD data sets based on predefined SPARQL Queries. Differently than the Standard Data Access, the Federated Data Access feature is able to execute SPARQL Queries to multiple SPARQL Endpoints, thus enabling WFS Features to be composed of data from different sources. As a proof of concept of what can be achieved, the SPARQL Query bellow shows an example of a federated query, combining data from DBpedia and Ordnance Survey of Great Britain. The SPARQL Query is executed against the Ordnance Survey’s SPARQL Endpoint, retrieving the GSS Code and geographic coordinates from districts of Great Britain – the coordinates are provided by the Ordnance Survey using the WGS84 lat/long Vocabulary, but this example converts them to WKT literals using the function CONCAT. Afterwards, the retrieved entries are filtered by matching the districts’ names with DBpedia entries from the east of England, which are written in English language. The result of this SPARQL Query can be further on listed as a single WFS Feature via the LOD4WFS Adapter, thereby providing a level of interoperability between data sets that is currently unachievable by any implementation of WFS, whether using Shapefiles or geographic databases.

PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX dbpo: <http://dbpedia.org/ontology/>
PREFIX dbp: <http://dbpedia.org/resource/>
PREFIX wgs84: <http://www.w3.org/2003/01/geo/wgs84_pos#>
PREFIX os: <http://data.ordnancesurvey.co.uk/ontology/admingeo/>

SELECT ?abstract ?name ?gss(CONCAT("POINT(", xsd:string(?long),"", xsd:string(?lat), ")") AS ?wkt)
WHERE { ?subject rdfs:label ?name .
		?subject wgs84:lat ?lat .
		?subject wgs84:long ?long .
		?subject os:gssCode ?gss .
		?subject a os:District
		SERVICE <http://dbpedia.org/sparql/> {
				?entry rdfs:label ?place .
				?entry dbpo:abstract ?abstract .
				?entry dbpo:isPartOf dbp:East_of_England
				FILTER langMatches(lang(?place), "EN")
				FILTER langMatches(lang(?abstract), "EN")
				FILTER ( STR(?place) = ?name )
		}
}

Setting up your environment

To setup your WFS Adapter environment properly, first you have to start playing with the settings file, which is - surprise, surprise - located at settings folder. The settings file is divided in the following sections.

[GetCapabilities]

Properties used for displaying the SDA Features metadata in the Capabilities Document.

title : RDF property for the Features’ title. Default value: <http://purl.org/dc/terms/title>

abstract : RDF property for the Features’ abstract. Default value: <http://purl.org/dc/terms/abstract>

keywords : RDF property for the Features’ subject. Default value: <http://purl.org/dc/terms/subject>

[Geometry]

Predicates and classes used for identifying geometries (SDA compliant) in SPARQL Endpoints.

geometryPredicate : <http://www.opengis.net/ont/geosparql/1.0#asWKT>

geometryClass : <http://www.opengis.net/ont/geosparql/1.0#Geometry>

predicatesContainer : <http://www.opengis.net/ont/geosparql/1.0#Feature>

featureConnector : <http://www.opengis.net/ont/geosparql/1.0#hasGeometry>

wktLiteral : <http://www.opengis.net/ont/geosparql#wktLiteral>

These properties are based in the following schema:

@PREFIX geo:  <http://www.opengis.net/ont/geosparql/1.0#> . 
@PREFIX sf:   <http://www.opengis.net/ont/geosparql#> . 

_:feature a geo:Feature .
_:geometry a geo:Geometry .
_:feature geo:hasGeometry _:geometry .
_:geometry geo:asWKT "POINT(15.60 50.72))"^^sf:wktLiteral .  

[SystemDefaults]

Most of the properties in this section, if not carefully configured, can possibily damage the service.

Let's start with the easy ones...

defaultCRS : Default Coordinate Reference System for the encoded geometries. You'll probably never need to touch this property, since geometries without an explicit CRS are to be interpreted as EPSG:4326 (WGS84). However, if your data set does not stick to the standards, here you have an option to change it.

sdaPrefix : Prefix used for SDA Features, displayed in the WFS documents. Default value: sda.

fdaPrefix : Prefix used for FDA Features, displayed in the WFS documents. Default value: fda.

solrPrefix : Prefix used for SOLR Features, displayed in the WFS documents. Default value: solr.

sdaEnabled : Property to enable or disable the usage of SDA Features. Since the SDA Features extrictly rely on a certain data schema, it's not applicable in most cases. Therefore, it is disabled by default. Default value: false.

fdaEnabled : Property to enable or disable the usage of FDA Features. Default value: true.

solrEnabled : Property to enable or disable the usage of SOLR Features. Default value: false.

Here there are the critical properties, so unless you really know what you're doing, just don't touch them. These properties are related to the default RDF literal types and namespaces used by the application, which are compliant to the current standards.

defaultLiteral : In case no literal type is provided, the type defined here is going to be assumed. Default value: <http://www.w3.org/2001/XMLSchema#string>

stringLiteral : String literal type. Default value: <http://www.w3.org/2001/XMLSchema#string>

integerLiteral : Integer literal type. Default value: <http://www.w3.org/2001/XMLSchema#int>

decimalLiteral : Decimal literal type. Default value: <http://www.w3.org/2001/XMLSchema#decimal>

longLiteral : Long literal type. Default value: <http://www.w3.org/2001/XMLSchema#long>

dateLiteral : Date literal type. Default value: <http://www.w3.org/2001/XMLSchema#date>

floatLiteral : Float literal type. Default value: <http://www.w3.org/2001/XMLSchema#float>

serviceName : Service used at the HTTP requests, e.g. http://localhost:8088/lod4wfs/wfs/?service=wfs&version=1.0.0&request=GetCapabilities. Default value: lod4wfs

[Server]

Application's settings.

defaultPort : Port in which the service will listen to the requests. Default value: 8088.

SPARQLEndpointURL : SPARQL Endpoint address where the SDA Features are located.

SPARQLDirectory : Directory where the FDA Features’ queries will be stored. Default value: features/

connectionTimeOut : Connection time-out (in milliseconds) for the given SPARQL Endpoint. Setting a time out will prevent that the requests to wait forever, in case of a server malfunction or network problems. I case you're more like "no risk, no fun", just set the property to 0. Default value: 1000.

[WebInterface]

[PreviewLimit] : Maximum number of records displayed at the web interface preview. Default value: 10

Starting the WFS Adapter

After the settings file is properly configured, you’re ready to start the LOD4WFS adapter. The application is compressed in a single jar file lod4wfs_releasex.x.jar located in the root directory and can be started in the following command:

$ java -Xmx1g -jar lod4wfs_release-x.x.jar

Note: The parameter -Xmx reserves the amount of heap space necessary for the application. So, if you’re dealing with large geometries, it might be necessary to increase the heap space, e.g. -Xmx5g. Since it is totally dependent on the geometries size, there is no optimum heap space size. For better maintainability, it is recommended to create a service in your operating system to manage the process created with command above mentioned.

If you're a developer and want to start the service directly from your IDE, just execute the main() method of the class de.ifgi.lod4wfs.web.Start. If you didn't miss anything, you should be seeing by now the following greetings in your console:

LOD4WFS Adapter (Linked Open Data for Web Feature Service) 

Institut für Geoinformatik | Universitäts- und Landesbibliothek 
Westfälische Wilhelms-Universität Münster
http://www.uni-muenster.de/

Application Version: BETA-0.4.4.16091509
Startup time: 15-09-2016 17:04:43
Java Runtime: 1.8.0_91
Operating System: Linux 4.2.0-42-generic (amd64)
Port: 8088
Default SPARQL Endpoint (SDA Features): http://any.endpoint.de:8081/parliament/sparql
Connection Time-out in ms: 1000
FDA Features Enabled: true
SDA Features Enabled: false
SOLR Features Enabled: false
FDA Features Directory: features/
Web Interface Preview Limit: 10

lod4wfs's People

Contributors

jimjonesbr avatar dependabot[bot] avatar

Stargazers

Nico Görlich avatar 胡实 avatar Andreas Kuckartz avatar Weiming Huang avatar Martynas Jusevičius avatar  avatar Surendra avatar Pano Maria avatar Mikel Egaña Aranguren avatar Martinez-Lopez Javier avatar  avatar Aaron Mihalik avatar Aleksandar Smiljanic avatar

Watchers

Andreas Kuckartz avatar Francesco Bartoli avatar Matthias Pfeil avatar  avatar

lod4wfs's Issues

Automatic GeometryVariable

Automatic generate GeometryVariable based on the GeometryPredicate variable. Apply the same rule used in the GetFeature SPARQL generation.

FDA Features Backup

Backup and Restore function for FDA features via web interface (features list)

  • Single features
  • All features

Invalid entries at features.log

The application is no longer able to parse the features.log file properly:

[2016-09-15 07:32:24,169] ERROR [FDAFeatures-Factory] - Invalid entry at features.log

Invalid canonical hostname

Canonical host name provided in the Capabilities Document currently depends on network settings, returning blank if the settings are unknown.

CRS list

Enhance the list of CRS in the reference_systems.crs file.

Capabilities Document

Capabilities Document not displayed when requested at the Web Interface on localhost.

Normal GeoJson

GeoJson only works through JSONP, however other WFS application provide GeoJson in a plain Json way. Would it be possible to implement a plain GeoJson way instead of the mandatory JSONP way. This makes it possible to handle LOD4WFS maps as normal WFS maps through uniform GeoJson..

Multiple CRS

Create a settings file for dealing with multiple CRS. A conversion from the CRS URL, in the WKT literal, to a CRS understood by GIS must be implemented. Currently working only with EPSG.

SPARQL Endpoint dependence

Enable listener to start even where there is no valid SPARQL Endpoint configured and SDA features are enabled. Currently there is no way to start the service if there is no Endpoint available.

Output format GeoJson error

According to the docs: outputformat=text/javascript will give GeoJson as result. However, adding the output format "outputformat=text/javascript" to the url throws a nullpointer exception in LOD4WFS:

[2016-09-08 10:50:17,058] WARN [org.eclipse.jetty.servlet.ServletHandler] - /lod4wfs/wfs java.lang.NullPointerException at de.ifgi.lod4wfs.factory.AdapterLOD4WFS.getFeature(AdapterLOD4WFS.java:239) at de.ifgi.lod4wfs.factory.FactoryWFS.getFeature(FactoryWFS.java:313) at de.ifgi.lod4wfs.facade.Facade.getFeature(Facade.java:45) at de.ifgi.lod4wfs.web.ServletWFS.doGet(ServletWFS.java:206) at javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:457) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745)

Test Requests

Create test requests at the web interface.

  • GetCapabilities
  • DescribeFeatureType
  • GetFeature

Log files

Create a month aggregation of log entries (different files).

Maxfeature

Implement maxfeature key to the KVP requests to limit the number of features returned via GetFeature.

properties with underscore in the name are not transfered

I've run into a problem where properties with underscores in their name do not get handled properly

If I have a sparql variable ?my_property it will end up in the WFS as 'property' with null value.

I've tested this on multiple datasets same result. I didn't have the time to check the problem in the code.

Output Formats

Supported output formats (GML2, GeoJSON) are currently not informed in the Capabilities Document.

WKT literals are case-insensitive case

WKT literals are case-insensitive, so [1] and [2] should both be processed. Currently, WKT literals of form [1] are not processed.

[1] Point (1.0 1.0)
[2] POINT(1.0 1.0)

Expected: WKT literals to be processed in a case-insensitive way.

CQL Support

Did you ever thought about CQL support, to extend the queries with automated filters to select a subset of features based on a attribute value ?

All non-geo values are interpreted as strings

All non-geo values in the SPARQL query are treated by QGIS as string. Is there a way to copy the datatype from the SPARQL result set to the feature set?

Have tried the following in AdapterLOD4WFS, however I don't know the correct encoding of the datatype in the XML format.

Element elementAttribute = document.createElement(layerPrefix + ":" + predicateWithoutPrefix);

RDFNode solnNode = soln.get("?"+predicateWithoutPrefix);

  if (solnNode.isLiteral()) {
    Literal solnLiteral = solnNode.asLiteral();
    elementAttribute.setAttribute("type", solnLiteral.getDatatypeURI());
    elementAttribute.appendChild(document.createCDATASection(solnLiteral.getLexicalForm()));
  } else {
       elementAttribute.appendChild(document.createCDATASection(solnNode.toString()));
  }
currentGeometryElement.appendChild(elementAttribute);

SSL support for SPARQL requests

When sending a SPARQL request to an endpoint that is served over HTTPS, an SSL error is thrown. SSL support does not seem to be enabled.

Expected: HTTPS requests to SPARQL endpoints to be sent with SSL support, if required.

Provide drop in WAR besides full server

Hi,

I would like to have LOD4WFS to be run inside a existing Tomcat deployment.
Is it possible to build a drop in war, instead of a full package which runs its own server ?

Bart

CORS Support

Currently the WFS service has no CORS support, this can only be achieved by putting a proxy in front of LOD4WFS.

I understand that a proxy is good thing anyway, but you don't want to mess with headers manually

Update dependencies

The following external libraries are obsolete and need to be upgraded in the POM file:

  • log4j
  • jetty
  • jena-libs
  • jena-arq
  • ini4j
  • solrj

Access to multiple triplestores

Enable access to geometries from multiple triplestores in a single request.

  • Add tag to the settings file listing triplestores.
  • Implement GetCapabilities to the triplestores specified in the settings file
  • Implement triplestore mapping for all geometries listed in the Capabilities Document.

Release Version

Enable release + revision numbering using maven build. Info currently being generated into resources/version.properties

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.