Parsing spacebar in URI throws an error about server.js

[RubenVerborgh]

@migalkin Thanks to @mielvds, the downloads are now available at http://downloads.linkeddatafragments.org/hdt/fedx/

[migalkin]

Oh, my bad, the issue is apparently connected to the Client.js, not Server.js please move it to the Client.js repo :)

[RubenVerborgh]

@migalkin Thanks for reporting. In this case, the error is in the input file: the URI is invalid. For performance reasons, the server makes the assumption that the HDT file is built from a valid RDF input file.

When generating the HDT file, please use rdf2hdt -f turtle in.nt out.hdt. Even though your input is N-Triples, the Turtle parser (actually SERD) is doing a more precise job than the built-in parser.

My colleague @mielvds can probably tell you what we did to make the KEGG dataset valid.

[migalkin]

@RubenVerborgh thanks for the insights, I tried the Turtle parser but the result is still the same.
I found actually a lot of such URIs with a spacebar in KEGG/Bio2RDF, so fixing it manually is apparently not the best way

[mielvds]

Perhaps @hariharshankar can provide some insight? He has created a lot of HDT files from dirty RDF datasets

[RubenVerborgh]

@migalkin Strange, I expect the SERD parser to fail on URIs with a space. If not, that's a but in the SERD parser… do you have the latest version installed? Just for clarity: I'm talking about the C++ version of the HDT utility; I don't know about the Java utility.

Fixing manually is certainly not the best way. I have treated similar datafiles in the past with just a simple sed (if spaces are the only problem).

[migalkin]

@RubenVerborgh Yes, I use the latest C++ version available on github. Though the original dataset is in N3, first I had to use rdf2rdf parser to transform it to NT and then apply the C++ HDT tool. None of them threw any error.

If you have a script to eliminate spaces in URIs that would be great as my knowledge of reg exps is not enough for targeting only uris within <> and not affecting string literals for example.

[thadguidry]

@migalkin Perhaps your dealing with a non-breaking space that's affecting the parser ? I've had to deal with Unicode ALOT and sometimes you run into these situations. Here's a writeup I did for OpenRefine that describes the problem. https://github.com/OpenRefine/OpenRefine/wiki/Recipes#question-marks--showing-in-your-data

[RubenVerborgh]

Strange, serdi fails for me with invalid IRI character ` ' (escape %20).

This sed script should help you out:

sed 's/<\([^>]*\) \([^>]*\)>/<\1%20\2>/g' input.nt > input_fixed.nt

[hariharshankar]

The scripts I use to check and clean DBpedia dataset in NTriples format are here: https://bitbucket.org/hariharshankar/dbpedia_hdt. We mainly tailored the script to deal with issues in the DBpedia dataset.

Specifically, this is the script that contains regexes that you were interested in:
https://bitbucket.org/hariharshankar/dbpedia_hdt/src/dc11f79ebecbd5f7de28c336dfcb76436eb17a66/dbp_clean.py?at=master&fileviewer=file-view-default

Hope that helps.

[migalkin]

Thank you for your suggestions

@RubenVerborgh the script returns a different entity, e.g., for

<http://bio2rdf.org/pdb-ccd:NAD NAJ>

the result it

<D%20J>

I'm afraid it's an invalid URI again. Is there any way to retain the full string?

@hariharshankar
Executed the script and it (according to the docs) deleted all the 'dirty' triples.
I agree it might be a solution (apparently how LOD Laundromat works), but for us it is a problem as we are comparing federated query engines and all deleted for LDF triples might be still returned by some other RDF query engines (which use the original 'dirty' dataset), so it affects the cardinality of the answer and therefore makes the evaluation unreliable.

@thadguidry
Thanks for pointing this out, I'll try to apply.

[RubenVerborgh]

Sorry, should have been:

sed 's/<\([^>]*\) \([^>]*\)>/<\1%20\2>/g' input.nt > input_fixed.nt

Also corrected it above.

[RubenVerborgh]

Actually, is it FedX that you are running? Because several of its datasets have problems, and because of that, there is some uncertainty over what the correct results are. @mielvds has cleaned the datasets and converted them to HDT. We can send them to you if you like (and perhaps, we should just offer this on our website, as this is a common use case).

[migalkin]

@RubenVerborgh
Thank you for the script, it helped me a lot.
In some cases, e.g.,

<http://bio2rdf.org/ec:Acting on the CH-OH group of donors;> .

it replaces only the last space like

<http://bio2rdf.org/ec:Acting on the CH-OH group of%20donors;> .

so I just wrote a short additional script to fix found erroneous lines.
HDT parser worked fine and I can query the LDF server without any errors/warnings.

Indeed, we use Fedbench as one of the benchmarks. Frankly, I can't imagine why they decided to publish such an invalid data as a benchmark (wining, wining).
We would be very grateful if you could share cleaned dumps (including original NTriples dumps as we'll have to upload them to non-HDT stores).

[RubenVerborgh]

@mielvds Do you remember where we've put those dumps? I'll put them on the website.

[mielvds]

I'll get back to you on that.

[mielvds]

No guarantees that these dumps are error free though.

…

On 02 Dec 2016, at 22:43, Ruben Verborgh ***@***.***> wrote: @migalkin <https://github.com/migalkin> Thanks to @mielvds <https://github.com/mielvds>, the downloads are now available at http://downloads.linkeddatafragments.org/hdt/fedx/ <http://downloads.linkeddatafragments.org/hdt/fedx/> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#37 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA_DFJlbqfiEAjwcrYRmh33gfh-tZ3pXks5rEJEfgaJpZM4LA5G6>.