swirrl / datahost-prototypes Goto Github PK
View Code? Open in Web Editor NEWLicense: Eclipse Public License 1.0
License: Eclipse Public License 1.0
In a previous WIP prototype, called cubiql 2, we showed how we can support some of the extensibility of RDF schemas within a relatively fixed graphql schema, by the inclusion of specific property path fields at key points in the schema.
For example cubiql-2 supported this query:
{
endpoint {
datacube(id:"http://covid/infections") {
id
observations {
area_label: property_path_string(path:["sdmxd:refArea" "rdfs:label"])
}
}
}
}
The property_path_XXXX
fields encoded within their names the expected return types of those fields and would cast accordingly. We included the following set of fields:
property_path_literal_query
property_path_uri_query
property_path_int_query
property_path_float_query
property_path_string_query
property_path_lang_string_query
property_path_datetime
As these fields take an array it is possible to provide depth limits on the length of the path, without changing the schema.
The property path fields in cubiql-2 would also work with the default prefixes, but also an extended set of prefixes which could be provided in a parent graphql resolver.
It would be nice to selectively demonstrate this in this prototype in particular on the global object identification interface.
There will always be performance costs associated with this, so controling the context in which this feature is available and can be optimised is important, along with the depth of the paths supported in each context.
Look at Circle CI Dynamic config and Path Filtering Orbs to only build and deploy monorepo apps that have changed.
https://circleci.com/docs/dynamic-config/
https://circleci.com/developer/orbs/orb/circleci/path-filtering
Background reading:
https://hackernoon.com/continuous-integration-for-a-monorepo-on-circleci-example
We need to communicate the intention of the prototypes, so we need to improve the README with
Currently you get errors like:
Access to fetch at 'http://localhost:8888/api' from origin 'http://localhost:8080/' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
NOTE: we will also need to fix #32 and move to HTTPS to prevent mixed content errors to support client side js API access fully.
The prototype currently works off the live database, which is continually being changed.
We should extract a minimal/modest set of fixture data from the beta site by running some construct queries, and then load those fixtures into an in memory (or RDF4j native store) SPARQL repo.
The point here is to turn the catql-prototype repo into a mono repo where we have CI and docker setup for two app builds (the graphql prototype and the new linked data API prototype).
datahost-graphql
as the first step in turning this repo into a monorepo (containing two prototype services, not one)datahost-graphql
)datahost-graphql
project into the datahost-ld-openapi
(so we get the same project structure)datahost-ld-openapi
(though at this point the implementation will be a copy of the graphql service)tpximpact.datahost.ldapi
200 OK Hello World
Add routes for:
PUT /data/:series/:release
GET /data/:series/:release
POST /data/:series/:release/revision
...
POST /data/:series/:release/schema
Due to the document centric nature of some of the RDF here it would be nice to find a way to augment some of those documents with the inverse triples of key predicates... e.g.
my-release dcat:inSeries my-series => my-release dcat:seriesMember my-series
This could in principle be done either on write or on read.
Currently endpoint_id
can be any SPARQL endpoint. We should rename this parameter to be draftset_id
to prevent any assumptions that it works on arbitrary 3rd party endpoints. draftset-id
will then at some point in the future be a string indicating the draftset you are querying against.
If this parameter isn't provided we should default to the public endpoint.
e.g. this works:
{
endpoint {
catalog {
id
catalog_query {
datasets {
id
}
facets {
publishers {
id
label
}
creators {
id
label
}
themes {
id
label
}
}
}
}
}
}
But removing the datasets
sub query making this query, causes it to hang. I strongly suspect this is because of the promise
delivery between these two resolvers; and I think we're accidentally expecting the datasets
field
and resolver to always be present; where as the fields can be independent of each other.
i.e. just because the data dependency to compute the enabled
flag may exist from facets
-> datasets
it doesn't imply that the fields are always present in the query, and we're relying on the fields to execute the resolvers.
Here's the hanging query:
{
endpoint {
catalog {
id
catalog_query {
facets {
publishers {
id
label
}
creators {
id
label
}
themes {
id
label
}
}
}
}
}
}
The schema currently requires that you provide a valid catalog URI e.g:
{
endpoint {
catalog(id: "http://gss-data.org.uk/catalog/datasets") {
id
}
}
}
We should change it so this is optional, and that it will default to the hardcoded value above:
{
endpoint {
catalog {
id
}
}
}
There's currently no need to make this configurable; though we might as well lift it into the .edn
config as a hardcoded value, so it's closer to being configurable.
Discuss and decide on domain name strategy and whether we need to wire up a proxy pass through an nginx etc
(Probably don't need to do it for the prototype; but we need to have or start formulating a plan for the how the routes will map)
Some SPARQL queries are going to return only a modest size of results and will rarely be updated. We should look at caching the results of these with core.cache or an alternative, and using information revealed by drafter to expire caches when new data is published.
This task is to integrate the three strands into one, such that the series work is exposed via openapi 3 and reitit, and persistently stores data with duratom.
Other things to do:
Definition of done:
If you do a PUT /data/my-dataset-series
of the appropriate content then you should be able to GET /data/my-dataset-series
and get that document back out.
Right now we only have series implemented so this issue is just to inject dcterms:issued
into those on create. The relevant model code is in datahost/ldapi/models/series.clj
. Creates should be atomic, i.e. inject the timestamp before the db update is attempted, so either the whole jsonld document (with the issued timestamp) gets inserted or not. This could be implemented in such a way that the code to inject the timestamp could be re-used for other models (we will also be adding the timestamp to e.g. releases and revisions once those are implemented).
This issue is done when all new series include a dcterms:issued
key that reflects the time the series was created.
Tech choices for prototype:
Mount a docker volume for the LD API service in the production environment to store the database across application restarts.
Though perhaps we should be ok with it being transitory and lost on reboot during the prototype stage, and we should tackle this only when need persistence over restarts, and when we need that we should use a real DB server e.g. Jena/Stardog.
We've currently focussed on explicitly supporting an absolute bare minimum amount of metadata, e.g. dct:title
dct:description
and not much else beyond predicates to tie our model together.
We should review all the entities and the metadata terms that can be used on them...
e.g. For all entities we should probably add a managed predicate of dct:issued
(which cannot be changed once set), whilst dct:modified
should be managed and set on every upsert.
Take the example queries in #8 and implement (failing) tests for them, along with any other edge cases we may need, so we can potentially park this issue on a draft PR and come back to it later.
Idea is to start simple and get something going ASAP.
The current process is that we deploy docker containers for the datahost-graphql
prototype to our public docker registry, and then start them through google container optimised images.
The idea is to automate this workflow such that our CI becomes CI/CD, where deployments happen on every merge to the main
branch (or on a tag
).
As the above prototype is currently stateless (or the state is supplied by the beta.gss-data.org.uk
public sparql endpoint, we can kick the can down the road on some of the harder deployment questions.
In the short term deploys don't need to be downtimeless.
Some other ideas:
Once this is done, we can look at applying this same process to the datahost-ldopenapi
prototype. At that point this process will likely also need some state deployed with (so will require volumes mounted with data on) and we'll need an initial provisioning phase followed by a more routine CI/CD driven 'redeploy version' phase.
We haven't actually tried deploying this yet.
I expect it's 99% there; but suspect there may be a few small gotcha's... e.g. I think the ring prototype binds to localhost
and I think we need to change it to bind to 0.0.0.0
for it to work in the docker environment.
This ticket is basically to flush out any issues like this, by testing the deployment ASAP.
There are some inconsistencies in the existing faceting behaviour.
We have captured what the new behaviour should be in failing tests in PR #31, and merged them as pending this issue being resolved. They are flagged with ^:kaocha/pending
metadata so they don't currently break the build, and indicate functionality we've not yet implemented properly.
Some of the inconsisties/issues with faceting are also listed below:
{
endpoint {
catalog(id: "http://gss-data.org.uk/catalog/datasets") {
id
catalog_query {
datasets {
id
publisher
}
facets {
themes {
id
enabled
}
creators {
id
enabled
label
}
publishers {
id
enabled
}
}
}
}
}
}
Says all the facets are enabled = false
, when I think they should all be true
(if actually used by the data).
{
endpoint {
catalog(id: "http://gss-data.org.uk/catalog/datasets") {
id
catalog_query(themes: ["http://gss-data.org.uk/def/gdp#trade"]) {
datasets {
id
publisher
}
facets {
themes {
id
enabled
}
creators {
id
enabled
label
}
publishers {
id
enabled
}
}
}
}
}
}
I think in the results for this query many themes
should be enabled, as most options will expand the amount of query results. For example adding the balanceofpayments
theme to the list results increases the results from ~7 to ~20.
It might be nice to disable options which don't grow the results, though at the minute I don't think that is necessary.
So for example in a variant of our original query:
{
endpoint {
catalog(id: "http://gss-data.org.uk/catalog/datasets") {
id
catalog_query(themes:["http://gss-data.org.uk/def/gdp#trade"]) {
datasets {
id
label
}
facets {
themes {
id
}
publishers {
id
enabled
}
}
}
}
}
}
the publishers
returned shouldn't all be false
, but should implicitly include as enabled publishers
who published data into the trade theme, e.g.
So the above query is inconsistent with this one which locks both theme = trade
and creator = hmrc
as it has results.
I think enabled
is only set to true at the minute if you include that facet in your query... i.e. I think you're thinking of it as whether the checkbox has been ticked; where as I'm thinking of it as being whether you should enable/show or disable/hide that checkbox from the user.
This should probably follow a POST REDIRECT GET pattern.
datahost-prototypes/datahost-ld-openapi/doc/data-model-example.ttl
Lines 57 to 73 in a9d04fa
We should review the scratch schema code and integrate it as rest API routes.
Ultimately we may want to support many schemas being associated with a single release. However in this iteration of the prototype we can assume just one.
Multiple schemas may not be necessary; but they may support more cleanly allowing people to incrementally add commitments, or layer on other concerns (e.g. URI generation) to existing datasets (providing all data within that release historically stills conforms to the extra schema).
Implement POST / REDIRECT / GET pattern for creating schemas.
Route:
POST /data/:series-slug/release/:release-slug/schemas Accept: json+ld
BODY
{"dh:columns" [{"csvw:datatype" "string"
"csvw:name" "foo_bar"
"csvw:titles" ["Foo Bar"]}]}
Note normalising the incoming request body should:
"@type" "dh:TableSchema"
as a managed param to the table schema"@type" "dh:DimensionColumn"
to each column"datahost:appliesToRelease" "</data/:series-slug/release/:release-slug>"
"appropriate-csvw:modeling-of-dialect" "UTF-8,RFC4180"
Server responds:
303 /data/:series-slug/release/:release-slug/schemas/:auto-incrementing-schema-id
Route:
GET /data/:series-slug/release/:release-slug/schemas Accept: json+ld
Note the release will also get the inverse triple:
</data/:series-slug/release/:release-slug> datahost:hasSchema </data/:series-slug/release/:releases-slug/schemas/:schema-id>
Current schema looks like this:
{
endpoint(draftset_id: "xyz") {
endpoint_id
}
}
However switching the endpoint via a draftset_id
parameter probably isn't the right way to do this, as it won't play well with the GraphQL Global Object Identification pattern which recommends having a get(id:String)
field at the root of the schema, so for us draftset selection will need to be "over this".
Hence we should support a way of passing the draftset on the request.
Regardless I think endpoint metadata can stay in the schema, so this is really about moving where the draftset_id
flag is passed.
This avoids us having to worry about provisioning an extra database service for the prototype, we can run it in process.
We should wire it up to programmatically load some background data into a single graph on startup. We might want to copy some functionality from the muttnik loader around hashing to replace the background data if it is different, to what is already there. Initially the background data doesn't need to contain much more than:
@base <http://www.example.org/> .
</graphs/background-data> {
# Some background catalog metadata
example:datasets
a dcat:Catalog ;
rdfs:label "Datasets" .
}
Other things we need to do (at some point TBD):
Mount a docker volume for this service: #44
Requires #3 to be done first
Add the route /data/:series/release/:release-slug
mirroring the patterns established in the series route.
This ticket can be DONE when users can upload a json-ld of release metadata to the route and get it back out again.
Improvements to this area will be made in a future ticket #76 , where we add new representations to this resource for different content-types after the changesets/commits routes/model is complete.
The documents are currently stored in duratom which maintains the same interface as a clojure atom.
All clojure reference types along with the duratom
library we're using support add-watch
.
So an easy way to sync and serialise changes in the documents to RDF is by calling our ednld->rdf
on the document and writing the triples to the native-store triplestore through a watcher added to the duratom
.
If the triplestore write fails, we should retract the changes to the duratom too.
Short answer yes.
Medium answer: Yes, safe enough for now (eventually consistent)... though in a production system we would additional consistency checks to ensure eventual consistency, and a method for handling partial failure. The duratom will witness changes before the triple store, so we need to assume that; but as writes to the duratom
and triplestore are synchronous to the initial blocking request, providing a user blocks on their requests future requests should appear consistent.
Long answer for people who spotted one potential gotcha:
The docstring for add-watch
warns that the watcher may be called from multiple threads; and additionally watches on clojure atoms don't guarantee order semantics which would be bad as there'd be a risk of the databases containing different states.
However I have looked and it seems that duratom does protect against this, by wrapping the underlying atom in a lock (and relying on the fact that atom's synchronously block on their watches before returning from swap!
).
I've also confirmed this behaviour at a REPL:
(def a (db/duratom :local-file :file-path "/tmp/test-duratom.edn" :commit-mode :sync :init 0 ))
(def b (atom []))
(add-watch a :watcher (fn [k ref old new] (swap! b conj [old new])))
(dotimes [n 100000] (future (swap! a inc)))
(apply < (map first @b)) ;; => true
The true
here indicates that all starting (prior to inc
) states were linearised.
However if you repeat the experiment with a normal atom
you may witness the out of order behaviour mentioned:
(def a (atom 0))
(def b (atom []))
(add-watch a :watcher (fn [k ref old new] (swap! b conj [old new])))
(dotimes [n 100000] (future (swap! a inc)))
(apply < (map first @b)) ;; => false
We should probably find a more specific name for dh:DatasetSeries
, and rename it everywhere in the API.
The problem with DatasetSeries
is that the dcat terminology is quite broad:
Dataset series are defined in [ISO-19115] as a "collection of datasets […] sharing common characteristics". However, their use is not limited to geospatial data, although in other domains they can be named differently (e.g., time series, data slices) and defined more or less strictly (see, e.g., the notion of "dataset slice" in [VOCAB-DATA-CUBE]).
Our meaning of a DatasetSeries
is that it represents the name of a "dataset" which is stable over time, regardless of releases, revisions or schema and methodology changes. It is Heraclituses river, through which change flows. Examples are datasets like "The census" or "Indicies of Multiple Deprivation" which have seen various schema and methodology changes overtime and aren't necessarily directly comparable with previous releases. For us the series contains "releases" which in turn have stable schema, and contain "revisions". So for us the series allows for the representation of a series of "breaking changes" under the same "name", whilst still preserving all the notions of stability we care about in releases/revisions below the series level.
However most users probably think of a series as a "time series", which for us would be a cube with a stable schema and a time dimension.
Hence I think it would be worth us specialising dcat:DatasetSeries
with a subclass with a narrower name, my favourite so far is
dh:DatasetIdentity rdfs:subClassOf dcat:DatasetSeries .
Also:
Though we have done a large part of continuous deployment in #33 it is in a new GCP project, and the old prototype which has DNS setup is pointing to a fixed IP allocated in that project.
We can then do #91 and point that DNS to a new fixed IP which is in the new terraform deployment; at which point deploys will be fully automated for the prototype graphql service.
I've just noticed a bug in the implementation of facet results introduced in #63, in that it returns incorrect results in the presence of a search_string
.
I suspect this is because it is calculating the facets enabled
status before filtering by the search string.
Example queries to show the issue:
This query locks the trade
theme and searches within it for a string of "hmrc":
{
endpoint {
catalog {
catalog_query(search_string:"hmrc"
themes:["http://gss-data.org.uk/def/gdp#trade"]
) {
datasets {
title
creator
}
facets {
creators {
enabled
id
}
themes {
enabled
id
}
}
}
}
}
}
The elided results below first show all datasets in the results have a creator of hmrc
as expected:
"datasets": [
{
"creator": "https://www.gov.uk/government/organisations/hm-revenue-customs"
},
{
"creator": "https://www.gov.uk/government/organisations/hm-revenue-customs"
},
{
"creator": "https://www.gov.uk/government/organisations/hm-revenue-customs"
},
{
"creator": "https://www.gov.uk/government/organisations/hm-revenue-customs"
},
{
"creator": "https://www.gov.uk/government/organisations/hm-revenue-customs"
}
],
However in the facets branch for creator we see:
...
{
"enabled": true,
"id": "https://www.gov.uk/government/organisations/hm-revenue-customs"
},
...
{
"enabled": true,
"id": "https://www.gov.uk/government/organisations/department-for-digital-culture-media-sport"
},
...
Which is wrong, because the only enabled facet should be https://www.gov.uk/government/organisations/hm-revenue-customs
.
Removing the search string returns the expected distinct two creators :
{
"creator": "https://www.gov.uk/government/organisations/department-for-digital-culture-media-sport"
},
{
"creator": "https://www.gov.uk/government/organisations/hm-revenue-customs"
},
...
So it looks like we're just not applying the search string to the facets.
Goal is to make SSL work on these repos.
Right now we only have series implemented so this issue is just to inject dcterms:modified into those on update. The relevant model code is in datahost/ldapi/models/series.clj
. Creates should be atomic, i.e. inject the timestamp before the db update is attempted, so either the whole jsonld document (with the modified timestamp) gets inserted or not. This could be implemented in such a way that the code to inject the timestamp could be re-used for other models (we will also be adding the timestamp to e.g. releases and revisions once those are implemented).
This issue is done when all updated series include a dcterms:modified
key that reflects the time the series was updated.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.