RDFa Streaming Parser
A fast and lightweight streaming and 100% spec-compliant RDFa 1.1 parser, with RDFJS representations of RDF terms, quads and triples.
The streaming nature allows triples to be emitted as soon as possible, and documents larger than memory to be parsed.
Installation
$ npm install rdfa-streaming-parseror
$ yarn add rdfa-streaming-parserThis package also works out-of-the-box in browsers via tools such as webpack and browserify.
Require
import {RdfaParser} from "rdfa-streaming-parser";or
const RdfaParser = require("rdfa-streaming-parser").RdfaParser;Usage
RdfaParser is a Node Transform stream
that takes in chunks of RDFa data,
and outputs RDFJS-compliant quads.
It can be used to pipe streams to,
or you can write strings into the parser directly.
While not required, it is advised to specify the profile of the parser
by supplying a contentType or profile constructor option.
Print all parsed triples from a file to the console
const myParser = new RdfaParser({ baseIRI: 'https://www.rubensworks.net/', contentType: 'text/html' });
fs.createReadStream('index.html')
.pipe(myParser)
.on('data', console.log)
.on('error', console.error)
.on('end', () => console.log('All triples were parsed!'));Manually write strings to the parser
const myParser = new RdfaParser({ baseIRI: 'https://www.rubensworks.net/', contentType: 'text/html' });
myParser
.on('data', console.log)
.on('error', console.error)
.on('end', () => console.log('All triples were parsed!'));
myParser.write('<?xml version="1.0"?>');
myParser.write(`<!DOCTYPE html>
<html>
<head prefix="foaf: http://xmlns.com/foaf/0.1/">`);
myParser.write(`<link rel="foaf:primaryTopic foaf:maker" href="https://www.rubensworks.net/#me" />`);
myParser.write(`</head>`);
myParser.write(`<body>`);
myParser.write(`</body>`);
myParser.write(`</html>`);
myParser.end();Import streams
This parser implements the RDFJS Sink interface,
which makes it possible to alternatively parse streams using the import method.
const myParser = new RdfaParser({ baseIRI: 'https://www.rubensworks.net/', contentType: 'text/html' });
const myTextStream = fs.createReadStream('index.html');
myParser.import(myTextStream)
.on('data', console.log)
.on('error', console.error)
.on('end', () => console.log('All triples were parsed!'));Configuration
Optionally, the following parameters can be set in the RdfaParser constructor:
dataFactory: A custom RDFJS DataFactory to construct terms and triples. (Default:require('@rdfjs/data-model'))baseIRI: An initial default base IRI. (Default:'')language: A default language for string literals. (Default:'')vocab: The initial vocabulary. (Default:'')defaultGraph: The default graph for constructing quads. (Default:defaultGraph())features: A hash of features that should be enabled. Defaults to the features defined by the profile. (Default: all features enabled)profile: The RDFa profile to use. (Default: profile with all features enabled)contentType: The content type of the document that should be parsed. This can be used as an alternative to the 'profile' option. (Default: profile with all features enabled)htmlParseListener: An optional listener for the internal HTML parse events, should implementIHtmlParseListener(Default:null)
new RdfaParser({
dataFactory: require('@rdfjs/data-model'),
baseIRI: 'http://example.org/',
language: 'en-us',
vocab: 'http://example.org/myvocab',
defaultGraph: namedNode('http://example.org/graph'),
features: { langAttribute: true },
profile: 'html',
htmlParseListener: new MyHtmlListener(),
});Profiles
On top of RDFa Core 1.1, there are a few RDFa variants that add specific sets of rules, which are all supported in this library:
- HTML+RDFa 1.1: Internally identified as the
'html'profile with'text/html'as content type. - XHTML+RDFa 1.1: Internally identified as the
'xhtml'profile with'application/xhtml+xml'as content type. - SVG Tiny 1.2: Internally identified as the
'xml'profile with'application/xml','text/xml'and'image/svg+xml'as content types.
This library offers three different ways to define the RDFa profile or setting features:
- Content type: Passing a content type such as
'text/html'to thecontentTypeoption in the constructor. - Profile string: Passing
'','core','html','xhtml'or'svg'to theprofileoption in the constructor. - Features object: A custom combination of features can be defined by passing a
featuresoption in the constructor.
The table below lists all possible RDFa features and in what profile they are available:
| Feature | Core | HTML | XHTML | XML | Description |
|---|---|---|---|---|---|
baseTag |
✓ | ✓ | If the baseIRI can be set via the <base> tag. |
||
xmlBase |
✓ | If the baseIRI can be set via the xml:base attribute. |
|||
langAttribute |
✓ | ✓ | ✓ | If the language can be set via the language attribute. | |
onlyAllowUriRelRevIfProperty |
✓ | ✓ | ✓ | If non-CURIE and non-URI rel and rev have to be ignored if property is present. | |
inheritSubjectInHeadBody |
✓ | ✓ | If the new subject can be inherited from the parent object if we're inside <head> or <body> if the resource defines no new subject. |
||
datetimeAttribute |
✓ | ✓ | ✓ | If the datetime attribute must be interpreted as datetimes. |
|
timeTag |
✓ | ✓ | ✓ | If the <time> tag contents should be interpreted as datetimes. |
|
htmlDatatype |
✓ | ✓ | If rdf:HTML as datatype should cause tag contents to be serialized to text. |
||
copyRdfaPatterns |
✓ | ✓ | ✓ | If rdfa:copy property links can refer to rdfa:Pattern's for copying. |
|
xmlnsPrefixMappings |
✓ | ✓ | ✓ | ✓ | If prefixes should be extracted from xmlns. |
skipHandlingXmlLiteralChildren |
If children of rdf:XMLLiteral should not be handled as RDFa anymore. This is not part of the RDFa spec. | ||||
xhtmlInitialContext |
✓ | If the XHTML initial context should be included in the initial prefixes. | |||
roleAttribute |
✓ | ✓ | ✓ | If the role attribute should be handled. |
How it works
This tool makes use of the highly performant htmlparser2 library for parsing HTML in a streaming way. It listens to tag-events, and maintains the required tag metadata in a stack-based datastructure, which can then be emitted as triples as soon as possible.
Our algorithm closely resembles the suggested processing sequence, with a few minor changes to make it work in a streaming way.
If you want to make use of a different HTML/XML parser,
you can create a regular instance of RdfaParser,
and just call the following methods yourself directly:
onTagOpen(name: string, attributes: {[s: string]: string})onText(data: string)onTagClose()
Specification Compliance
This parser passes all tests from the RDFa 1.1 test suite. More specifically, the following manifests are explicitly tested:
- HTML+RDFa 1.1 (HTML4)
- HTML+RDFa 1.1 (HTML5)
- HTML+RDFa 1.1 (XHTML5)
- SVGTiny+RDFa 1.1
- XHTML+RDFa 1.1
- XML+RDFa 1.1
The following optional features for RDFa processors are supported:
The following optional features for RDFa processors are not supported (yet):
License
This software is written by Ruben Taelman.
This code is released under the MIT license.
![greenkeeper[bot] avatar](https://avatars.githubusercontent.com/in/505?v=4 "greenkeeper[bot]")
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent