digitalbazaar / pyld

JSON-LD processor written in Python

PyLD

Introduction

This library is an implementation of the JSON-LD specification in Python.

JSON, as specified in RFC7159, is a simple language for representing objects on the Web. Linked Data is a way of describing content across different documents or Web sites. Web resources are described using IRIs, and typically are dereferencable entities that may be used to find more information, creating a "Web of Knowledge". JSON-LD is intended to be a simple publishing method for expressing not only Linked Data in JSON, but for adding semantics to existing JSON.

JSON-LD is designed as a light-weight syntax that can be used to express Linked Data. It is primarily intended to be a way to express Linked Data in JavaScript and other Web-based programming environments. It is also useful when building interoperable Web Services and when storing Linked Data in JSON-based document storage engines. It is practical and designed to be as simple as possible, utilizing the large number of JSON parsers and existing code that is in use today. It is designed to be able to express key-value pairs, RDF data, RDFa data, Microformats data, and Microdata. That is, it supports every major Web-based structured data model in use today.

The syntax does not require many applications to change their JSON, but easily add meaning by adding context in a way that is either in-band or out-of-band. The syntax is designed to not disturb already deployed systems running on JSON, but provide a smooth migration path from JSON to JSON with added semantics. Finally, the format is intended to be fast to parse, fast to generate, stream-based and document-based processing compatible, and require a very small memory footprint in order to operate.

Conformance

This library aims to conform with the following:

• JSON-LD 1.1, W3C Candidate Recommendation, 2019-12-12 or newer
• JSON-LD 1.1 Processing Algorithms and API, W3C Candidate Recommendation, 2019-12-12 or newer
• JSON-LD 1.1 Framing, W3C Candidate Recommendation, 2019-12-12 or newer
• Working Group test suite

The test runner is often updated to note or skip newer tests that are not yet supported.

Requirements

• Python (3.6 or later)
• Requests (optional)
• aiohttp (optional, Python 3.5 or later)

Installation

PyLD can be installed with a pip package

pip install PyLD

Defining a dependency on pyld will not pull in Requests or aiohttp. If you need one of these for a Document Loader then either depend on the desired external library directly or define the requirement as PyLD[requests] or PyLD[aiohttp].

Quick Examples

from pyld import jsonld
import json

doc = {
    "http://schema.org/name": "Manu Sporny",
    "http://schema.org/url": {"@id": "http://manu.sporny.org/"},
    "http://schema.org/image": {"@id": "http://manu.sporny.org/images/manu.png"}
}

context = {
    "name": "http://schema.org/name",
    "homepage": {"@id": "http://schema.org/url", "@type": "@id"},
    "image": {"@id": "http://schema.org/image", "@type": "@id"}
}

# compact a document according to a particular context
# see: https://json-ld.org/spec/latest/json-ld/#compacted-document-form
compacted = jsonld.compact(doc, context)

print(json.dumps(compacted, indent=2))
# Output:
# {
#   "@context": {...},
#   "image": "http://manu.sporny.org/images/manu.png",
#   "homepage": "http://manu.sporny.org/",
#   "name": "Manu Sporny"
# }

# compact using URLs
jsonld.compact('http://example.org/doc', 'http://example.org/context')

# expand a document, removing its context
# see: https://json-ld.org/spec/latest/json-ld/#expanded-document-form
expanded = jsonld.expand(compacted)

print(json.dumps(expanded, indent=2))
# Output:
# [{
#   "http://schema.org/image": [{"@id": "http://manu.sporny.org/images/manu.png"}],
#   "http://schema.org/name": [{"@value": "Manu Sporny"}],
#   "http://schema.org/url": [{"@id": "http://manu.sporny.org/"}]
# }]

# expand using URLs
jsonld.expand('http://example.org/doc')

# flatten a document
# see: https://json-ld.org/spec/latest/json-ld/#flattened-document-form
flattened = jsonld.flatten(doc)
# all deep-level trees flattened to the top-level

# frame a document
# see: https://json-ld.org/spec/latest/json-ld-framing/#introduction
framed = jsonld.frame(doc, frame)
# document transformed into a particular tree structure per the given frame

# normalize a document using the RDF Dataset Normalization Algorithm
# (URDNA2015), see: https://www.w3.org/TR/rdf-canon/
normalized = jsonld.normalize(
    doc, {'algorithm': 'URDNA2015', 'format': 'application/n-quads'})
# normalized is a string that is a canonical representation of the document
# that can be used for hashing, comparison, etc.

Document Loader

The default document loader for PyLD uses Requests. In a production environment you may want to setup a custom loader that, at a minimum, sets a timeout value. You can also force requests to use https, set client certs, disable verification, or set other Requests parameters.

jsonld.set_document_loader(jsonld.requests_document_loader(timeout=...))

An asynchronous document loader using aiohttp is also available. Please note that this document loader limits asynchronicity to fetching documents only. The processing loops remain synchronous.

jsonld.set_document_loader(jsonld.aiohttp_document_loader(timeout=...))

When no document loader is specified, the default loader is set to Requests. If Requests is not available, the loader is set to aiohttp. The fallback document loader is a dummy document loader that raises an exception on every invocation.

Commercial Support

Commercial support for this library is available upon request from Digital Bazaar: [email protected].

Source

The source code for the Python implementation of the JSON-LD API is available at:

https://github.com/digitalbazaar/pyld

Tests

This library includes a sample testing utility which may be used to verify that changes to the processor maintain the correct output.

To run the sample tests you will need to get the test suite files by cloning the json-ld-api, json-ld-framing, and normalization repositories hosted on GitHub:

• https://github.com/w3c/json-ld-api
• https://github.com/w3c/json-ld-framing
• https://github.com/json-ld/normalization

If the suites repositories are available as sibling directories of the PyLD source directory, then all the tests can be run with the following:

python tests/runtests.py

If you want to test individual manifest .jsonld files or directories containing a manifest.jsonld, then you can supply these files or directories as arguments:

python tests/runtests.py TEST_PATH [TEST_PATH...]

The test runner supports different document loaders by setting -l requests or -l aiohttp. The default document loader is set to Requests.

An EARL report can be generated using the -e or --earl option.

pyld's Issues

SSL certificate verify failures in tests

The tests that access https://json-ld.org/ have "SSL: CERTIFICATE_VERIFY_FAILED" failures. Perhaps due to the local VerifiedHTTPSHandler not supporting SNI?

does context URL have to be absolute?

I tried pyld with a context URL that was of the form '/foo/bar', but pyld refuses to load it. It wants a URL with netloc included. Is this a part of the standard I missed or a choice by pyld? What's the motivation?

compact then flatten breaks @id

Notice how when I compact and then flatten the JSON, the @id changes from 'http://example.com/api/things/1' -> 'http://example.com/api/things/'. I believe the same thing happens in the Javascript code. I don't have a good enough understanding of the spec to know what the proper behaviour is, though part of me expecting the least surprise thinks the @id should have became the empty string "" in the compacted form.

Sample code:

jsond = {
    '@id': 'http://example.com/api/things/1',
    'http://example.com' : "",
}
contextd = {
    '@base': 'http://example.com/api/things/1',
}

from pyld import jsonld
import pprint

compactd = jsonld.compact(jsond, contextd)
print "= compacted"
pprint.pprint(compactd, width = 1)

print "= flattened"
flattened = jsonld.flatten(compactd)
pprint.pprint(flattened, width = 1)

Output:

= compacted
{'@context': {'@base': 'http://example.com/api/things/1'},
 '@id': './',
 'http://example.com': ''}
= flattened
[{'@id': 'http://example.com/api/things/',
  'http://example.com': [{'@value': ''}]}]

Every .expand request downloads 130K file from schema.org

I suggest to add some caching because if the library is called many times it's painfully slow

New version on PyPi not installable with pip

Using pip 8.1.2 with Python 3.5, PyLD 0.7.1 is not installable (but is installable with Python2):

~/src/devsite/client-libraries-dev/python/tests : [master x] pip3.5 install PyLD==0.7.1
Collecting PyLD==0.7.1
Installing collected packages: PyLD
Successfully installed PyLD-0.7.0
~/src/devsite/client-libraries-dev/python/tests : [master x] pip install PyLD==0.7.1 
Collecting PyLD==0.7.1
Installing collected packages: PyLD
Successfully installed PyLD-0.7.1
~/src/devsite/client-libraries-dev/python/tests : [master x] 

Note that despite seeing 0.7.1 is there (it doesn't error that no such version exists), it actually collects and installs 0.7.0 for Python3.

(Workaround is to download the *.tar.gz and install with pip install -e, but I'm not sure that's a great long term plan.)

couldn't API stay compatible in 0.8.x?

just a question: couldn't there remain some load_document adapter to use flashy new requests_document_loader so projects using previous API could still use it? (sorry -- I didn't look in detail inside etc to see if that is feasible or not, decided just to ask)

compact with 'compactArrays': False always returns a graph and convert string value to list

I running into the issue where I'm calling jsonld.compact with 'compactArrays': False, and the result I'm getting always returns a graph and convert string value to list.

The json structure I'm using is:

{
  "@context": ...,
  "key1": "abc",
  "key2": ["abc", "def"]
}

where the value of key2 can be single value or multiple values.

I can't use jsonld.compact without 'compactArrays': False when key2 has single value, it will convert it into string.

It seems that during the compact function the code will call expand first, which will expand everything into array, and 'compactArrays': False will skip all of them. skipExpansion wouldn't work either since it convert the value ofkey1 into list of chars.

I would suggest to restructure the code so that it will not do a full expand before _compact

framing algorithm embedding options

I only faintly understand how framing works, but from my reading of the discussion here:

digitalbazaar/jsonld.js#68

there are three embed options, @never, @once and @always. But in the pyld sources I can see @last instead of @once. Isn't that inconsistent with the latest state of jsonld.js?

Testsuite fails on Expansion: #t0088: Value Expansion does not expand native values, such as booleans, to a node object

Hi,

The testsuite fails on (Python3.5):

Expansion: #t0088: Value Expansion does not expand native values, such as booleans, to a node object ...

Traceback (most recent call last):
  File "tests/runtests.py", line 228, in runTest
    result = getattr(jsonld, fn)(*params)
  File "tests/../lib/pyld/jsonld.py", line 158, in expand
    return JsonLdProcessor().expand(input_, options)
  File "tests/../lib/pyld/jsonld.py", line 876, in expand
    expanded = self._expand(active_ctx, None, document, options, False)
  File "tests/../lib/pyld/jsonld.py", line 2173, in _expand
    active_ctx, key, value, options, inside_list=False)
  File "tests/../lib/pyld/jsonld.py", line 1974, in _expand
    active_ctx, active_property, e, options, inside_list)
  File "tests/../lib/pyld/jsonld.py", line 1998, in _expand
    return self._expand_value(active_ctx, active_property, element)
  File "tests/../lib/pyld/jsonld.py", line 2667, in _expand_value
    return {'@id': self._expand_iri(active_ctx, value, base=True)}
  File "tests/../lib/pyld/jsonld.py", line 3998, in _expand_iri
    if ':' in value:
TypeError: argument of type 'bool' is not iterable
ERROR

======================================================================
ERROR: Expansion: https://json-ld.org/test-suite/tests/expand-manifest.jsonld#t0088: Value Expansion does not expand native values, such as booleans, to a node object
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/runtests.py", line 240, in runTest
    raise e
  File "tests/runtests.py", line 228, in runTest
    result = getattr(jsonld, fn)(*params)
  File "tests/../lib/pyld/jsonld.py", line 158, in expand
    return JsonLdProcessor().expand(input_, options)
  File "tests/../lib/pyld/jsonld.py", line 876, in expand
    expanded = self._expand(active_ctx, None, document, options, False)
  File "tests/../lib/pyld/jsonld.py", line 2173, in _expand
    active_ctx, key, value, options, inside_list=False)
  File "tests/../lib/pyld/jsonld.py", line 1974, in _expand
    active_ctx, active_property, e, options, inside_list)
  File "tests/../lib/pyld/jsonld.py", line 1998, in _expand
    return self._expand_value(active_ctx, active_property, element)
  File "tests/../lib/pyld/jsonld.py", line 2667, in _expand_value
    return {'@id': self._expand_iri(active_ctx, value, base=True)}
  File "tests/../lib/pyld/jsonld.py", line 3998, in _expand_iri
    if ':' in value:
TypeError: argument of type 'bool' is not iterable

----------------------------------------------------------------------
Ran 565 tests in 9.384s

FAILED (errors=1, skipped=142)

Problem compacting data

Hello,

When I try the following code:

 from pyld import jsonld
 doc = {"@type": "Movie",
       "actor": [{"name": "Myers Clark",
             "sameAs": "http://akas.imdb.com/name/nm2072773/"},
            {"name": "Kimberly Ridgeway",
             "sameAs": "http://akas.imdb.com/name/nm2064470/"}]}
 print jsonld.compact(doc, "http://schema.org/")

I get this as a respose:

{'@context': 'http://schema.org/', '@type': 'Movie'}

However, when I try this equivalent code in http://json-ld.org/playground/,

 {"@context": "http://schema.org",
   "@type": "Movie",
  "actor": [{"name": "Myers Clark",
             "sameAs": "http://akas.imdb.com/name/nm2072773/"},
            {"name": "Kimberly Ridgeway",
             "sameAs": "http://akas.imdb.com/name/nm2064470/"}]}

I get the expected json-ld in the compacted view:

 {
   "@type": "http://schema.org/Movie",
   "http://schema.org/actor": [
     {
       "http://schema.org/name": "Myers Clark",
       "http://schema.org/sameAs": {
         "@id": "http://akas.imdb.com/name/nm2072773/"
       }
     },
     {
       "http://schema.org/name": "Kimberly Ridgeway",
      "http://schema.org/sameAs": {
         "@id": "http://akas.imdb.com/name/nm2064470/"
       }
     }
   ]
 }

Any idea why jsonld.compact is ignoring the actors in this example?

Thanks :)

JsonLdProcessor should be new-style Python class

Minor oversight: JsonLdProcessor is defined as an old-style Python class. It should instead be defined as a new-style class:

class JsonLdProcessor(object):
    ...

This code will work with both Python 2 and Python 3 (in Python 3 all classes are new-style).

For more information, see:
http://stackoverflow.com/questions/54867/old-style-and-new-style-classes-in-python

Incorrect processing of lists

I believe that line 2713 is incorrect, when recursing into lists. The frame[prop] refers to the list, not to the objects within the list, and hence no further framing occurs as the objects do not match the list frame.

The fix is, if I understand the intent of the code and the framing spec correctly:

 self._match_frame(
         state, [o['@id']], frame[prop][0]['@list'],
         list_, '@list')

Thus, given the resources,
Frame:

{
    "@context" : "http://www.shared-canvas.org/ns/context.json",
    "@type": "sc:Manifest",
    "structures": {
        "@list": [
            {
                "@type":"sc:Range",
                "@embed": true,
                "canvases": {
                    "@list": [
                        {
                            "@embed":false
                        }
                    ]
                }
            }
        ]
    }
}

Instance:

{
  "@context":"http://www.shared-canvas.org/ns/context.json",
  "@type":"sc:Manifest",
  "@id":"http://www.example.org/iiif/book1/manifest.json",
  // ... references to canvases objects here ...
  "structures": [
    {
      "@id": "http://www.example.org/iiif/book1/range/1.json",
        "@type":"sc:Range",
        "label":"Introduction",
        "canvases": [
          "http://www.example.org/iiif/book1/canvas/1.json",
          "http://www.example.org/iiif/book1/canvas/2.json",
          "http://www.example.org/iiif/book1/canvas/3.json#xywh=0,0,750,300"
        ]
    }
  ]
}

Would embed the sc:Range object, but not the objects which are part of the "canvases" list.

Rob

regression? JsonLdError: <unprintable JsonLdError object>

Our @datalad testing build bots started to puke a new error which we haven't seen before:

======================================================================
ERROR: datalad.metadata.tests.test_search.test_our_metadataset_search
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 133, in run
    self.runTest(result)
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 151, in runTest
    test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 353, in run
    result.addError(self, sys.exc_info())
  File "/usr/lib/python2.7/dist-packages/nose/proxy.py", line 128, in addError
    formatted = plugins.formatError(self.test, err)
  File "/usr/lib/python2.7/dist-packages/nose/plugins/manager.py", line 99, in __call__
    return self.call(*arg, **kw)
  File "/usr/lib/python2.7/dist-packages/nose/plugins/manager.py", line 141, in chain
    result = meth(*arg, **kw)
  File "/usr/lib/python2.7/dist-packages/nose/plugins/logcapture.py", line 237, in formatError
    return (ec, self.addCaptureToErr(ev, records), tb)
  File "/usr/lib/python2.7/dist-packages/nose/plugins/logcapture.py", line 244, in addCaptureToErr
    records + \
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 652, in safe_str
    return str(val)
  File "/home/buildbot/datalad-pr-docker-dl-nd90/build/venv-ci/local/lib/python2.7/site-packages/pyld/jsonld.py", line 4923, in __str__
    rval += '\nCause: ' + str(self.cause)
  File "/home/buildbot/datalad-pr-docker-dl-nd90/build/venv-ci/local/lib/python2.7/site-packages/pyld/jsonld.py", line 4923, in __str__
    rval += '\nCause: ' + str(self.cause)
  File "/home/buildbot/datalad-pr-docker-dl-nd90/build/venv-ci/local/lib/python2.7/site-packages/pyld/jsonld.py", line 4923, in __str__
    rval += '\nCause: ' + str(self.cause)
  File "/home/buildbot/datalad-pr-docker-dl-nd90/build/venv-ci/local/lib/python2.7/site-packages/pyld/jsonld.py", line 4917, in __str__
    rval += '\nType: ' + self.type
TypeError: cannot concatenate 'str' and 'dict' objects

which when I went into that environment to troubleshoot showed us another face:

======================================================================
ERROR: datalad.metadata.tests.test_search.test_our_metadataset_search
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/buildbot/datalad-pr-docker-dl-nd90/build/datalad/tests/utils.py", line 840, in newfunc
    return func(*args, **kwargs)
  File "/home/buildbot/datalad-pr-docker-dl-nd90/build/datalad/tests/utils.py", line 627, in newfunc
    return t(*(arg + (filename,)), **kw)
  File "/home/buildbot/datalad-pr-docker-dl-nd90/build/datalad/metadata/tests/test_search.py", line 123, in test_our_metadataset_search
    assert list(ds.search('.', report='*', regex=True))
  File "/home/buildbot/datalad-pr-docker-dl-nd90/build/datalad/metadata/search.py", line 219, in __call__
    meta = flatten_metadata_graph(meta)
  File "/home/buildbot/datalad-pr-docker-dl-nd90/build/datalad/metadata/__init__.py", line 285, in flatten_metadata_graph
    return jsonld.flatten(obj, ctx={"@context": "http://schema.org/"})
  File "/home/buildbot/datalad-pr-docker-dl-nd90/build/venv-ci/local/lib/python2.7/site-packages/pyld/jsonld.py", line 178, in flatten
    return JsonLdProcessor().flatten(input_, ctx, options)
  File "/home/buildbot/datalad-pr-docker-dl-nd90/build/venv-ci/local/lib/python2.7/site-packages/pyld/jsonld.py", line 885, in flatten
    'jsonld.FlattenError', cause=cause)
JsonLdError: <unprintable JsonLdError object>

yet to figure out what is going on, but I feel that it relates to having recent 1.0.{0,1} releases of pyld out ;)
attn: @mih

All Rights Reserved

The project does not use an open source license. Could this please be changed?

Inefficient type testing for strings

pyld/lib/jsonld.py uses the following several

isinstance(var, (str, unicode))

These should be replaced with

isinstance(var, basestring)

Compaction fails for bool values

For the ld+json on this page:
http://io9.gizmodo.com/war-for-the-planet-of-the-apes-is-expanding-its-world-b-1789898134

Compaction fails with exception "argument of type 'bool' is not iterable"

compacted = jsonld.compact(doc, "http://schema.org")

The ld+json is copied below. It works fone on the ld_json playgrond (http://json-ld.org/playground/)

{
  "@context":"http://schema.org",
  "@type":"Article",
  "url":"http://io9.gizmodo.com/war-for-the-planet-of-the-apes-is-expanding-its-world-b-1789898134",
  "author":{
    "@type":"Person",
    "name":"Katharine Trendacosta"
  },
  "headline":"War for the Planet of the Apes Is Expanding Its World Beyond Caesar",
  "datePublished":"2016-12-09T11:23:00-05:00",
  "mainEntityOfPage":true,
  "image":{
    "@list":[
      {
        "@type":"ImageObject",
        "url":"https://i.kinja-img.com/gawker-media/image/upload/s--Hqn2havm--/c_scale,f_auto,fl_progressive,q_80,w_800/puk3lhu2eradznb76suk.png",
        "width":800,
        "height":449
      },
      {
        "@type":"ImageObject",
        "url":"https://i.kinja-img.com/gawker-media/image/upload/s--roBh1JKD--/c_scale,f_auto,fl_progressive,q_80,w_800/vrkerqbl9obklfbw34tl.png",
        "width":800,
        "height":86
      },
      {
        "@type":"ImageObject",
        "url":"https://i.kinja-img.com/gawker-media/image/upload/s--ky3BF80O--/c_scale,f_auto,fl_progressive,q_80,w_800/st48cto6bgc3l2xalbdl.png",
        "width":800,
        "height":335
      },
      {
        "@type":"ImageObject",
        "url":"https://i.kinja-img.com/gawker-media/image/upload/s--WarNmwdk--/c_scale,f_auto,fl_progressive,q_80,w_800/ljzzdte47tqh1rxwymti.png",
        "width":800,
        "height":334
      },
      {
        "@type":"ImageObject",
        "url":"https://i.kinja-img.com/gawker-media/image/upload/s--34F0OaZJ--/c_scale,f_auto,fl_progressive,q_80,w_800/h9yzgozhc7eqlvvzd3hv.png",
        "width":800,
        "height":335
      },
      {
        "@type":"ImageObject",
        "url":"https://i.kinja-img.com/gawker-media/image/upload/s--f6efnWre--/c_scale,f_auto,fl_progressive,q_80,w_800/vgyj2esofblnovpsqs2x.png",
        "width":800,
        "height":335
      },
      {
        "@type":"ImageObject",
        "url":"https://i.kinja-img.com/gawker-media/image/upload/s--_xSyT253--/c_scale,f_auto,fl_progressive,q_80,w_800/gtzfjztsemdtcl0pwnyx.png",
        "width":800,
        "height":368
      }
    ]
  },
  "thumbnailUrl":"https://i.kinja-img.com/gawker-media/image/upload/s--RXoSFT-B--/eh1hvjxamru5z6aobgwc.png",
  "dateCreated":"2016-12-09T11:23:00-05:00",
  "articleSection":"io9.gizmodo.com",
  "creator":[
    "Katharine Trendacosta"
  ],
  "keywords":[
    "war for the planet of the apes",
    "movies"
  ],
  "publisher":{
    "@type":"Organization",
    "name":"io9.gizmodo.com",
    "logo":{
      "@type":"ImageObject",
      "url":"https://i.kinja-img.com/gawker-media/image/upload/s--ZrPeeUzw--/veoxklw8coulpkai27fa.png"
    }
  }
}

Support application/n-quads.

Similar to digitalbazaar/jsonld.js#141:

According to the W3C Recommendation, the standard MIME type for N-Quads is "application/n-quads" (see https://www.w3.org/TR/n-quads/#h2_sec-mediaReg), while jsonld currently requires "application/nquads" in .normalize(), .toRDF() and fromRDF().
I propose it should accept both "application/n-quads" and "application/nquads", returning N-Quads in both cases.

Examples of usage with Google's Structured Data JSON LD?

Despite this being the premier JSON-LD library for python, the examples are sparse and leave a lot to the imagination. Or maybe it's just me. In frustration I ended up resorting to dicts and json for now, but wanted to see if we could get some more examples and documentation up.

Here are examples of outputting jsonld Google's way:

https://developers.google.com/search/docs/guides/

Breadcrumbs: https://developers.google.com/search/docs/data-types/breadcrumbs

<script type="application/ld+json">
{
  "@context": "http://schema.org",
  "@type": "WebSite",
  "url": "https://www.example.com/",
  "potentialAction": [{
    "@type": "SearchAction",
    "target": "https://query.example.com/search?q={search_term_string}",
    "query-input": "required name=search_term_string"
  },{
    "@type": "SearchAction",
    "target": "android-app://com.example/https/query.example.com/search/?q={search_term_string}",
    "query-input": "required name=search_term_string"
  }]
}
</script>

https://developers.google.com/search/docs/data-types/logo

{
  "@context": "http://schema.org",
  "@type": "Organization",
  "url": "http://www.example.com",
  "logo": "http://www.example.com/images/logo.png"
}

https://developers.google.com/search/docs/data-types/articles

{
  "@context": "http://schema.org",
  "@type": "NewsArticle",
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://google.com/article"
  },
  "headline": "Article headline",
  "image": {
    "@type": "ImageObject",
    "url": "https://google.com/thumbnail1.jpg",
    "height": 800,
    "width": 800
  },
  "datePublished": "2015-02-05T08:00:00+08:00",
  "dateModified": "2015-02-05T09:20:00+08:00",
  "author": {
    "@type": "Person",
    "name": "John Doe"
  },
   "publisher": {
    "@type": "Organization",
    "name": "Google",
    "logo": {
      "@type": "ImageObject",
      "url": "https://google.com/logo.jpg",
      "width": 600,
      "height": 60
    }
  },
  "description": "A most wonderful article"
}

JSON-LD 1.1

Hi,

I'm wondering is there any plan to make the PyLD compatible with the current JSON-LD version 1.1.

Thanks!

Does PyLD do scoped contexts within node objects ?

See Example 28 in http://www.w3.org/TR/json-ld/ for the kind of JSON-LD that I am having trouble with.

I have some code producing JSON-LD containing a sub-dictionary with a context of its own. It appears that PyLD does not process that context, while it handles the main context of the data structure just fine.

This apparent issue is independent of whether the scoped context is in-lined or not.

My service is not (yet) available externally, so I can't provide a link to the JSON-LD.

Framing of @container:@list with embedded reference errors

The code below errors:

  ...
  File "/Library/Python/2.7/site-packages/pyld/jsonld.py", line 3320, in remove_dependents
    embeds[next]['parent']['@id'] == id_):
KeyError: '@id'

When the ref keys are present.
Expected behavior would be to produce the same result as the input document.
Version: 0.7.1

from pyld.jsonld import frame

ctxtjs = {
	"@context": {
		"ns": "http://example.com/ns/",
		"hasList": {"@id": "ns:hasList", "@type": "@id", "@container":"@list"},
		"ref": {"@id": "ns:ref", "@type": "@id"}
	}	
}

doc = {
	"@context": ctxtjs['@context'],
	"@id": "http://top/1",
	"@type": "ns:A",
	"hasList": [
		{
			"@id": "http://child/1", 
		 	"@type": "ns:B", 
		 	"ref": "ns:something"
		},
		{
			"@id": "http://child/2", 
		 	"@type": "ns:B", 
		 	"ref": "ns:something"
		}
	]
}

framejs = {
	"@context": ctxtjs['@context'],
	"@type": "ns:A",
	"hasList": [{
		"@type": "ns:B"
	}]
}

framed = frame(doc, framejs)

(/cc @zimeon, @jpstroop, re IIIF/api#1117)

Building PyLD while PyLD is installed fails

sys.path.append(os.path.join(os.path.dirname(__file__), 'lib'))
from pyld import jsonld

If you have pyld installed already it's version string will be used by setup.py this causes very odd behavior.

README.md - incorrect json example output

The example output of an expanded document is not correct.

According the readme the output is a dictionary:

print(json.dumps(expanded, indent=2))
# Output:
# {
#   "http://schema.org/image": [{"@id": "http://manu.sporny.org/images/manu.png"}],
#   "http://schema.org/name": [{"@value": "Manu Sporny"}],
#   "http://schema.org/url": [{"@id": "http://manu.sporny.org/"}]
# }

I guess this should be changed to:

print(json.dumps(expanded, indent=2))
# Output:
# [ {
#   "http://schema.org/image": [{"@id": "http://manu.sporny.org/images/manu.png"}],
#   "http://schema.org/name": [{"@value": "Manu Sporny"}],
#   "http://schema.org/url": [{"@id": "http://manu.sporny.org/"}]
# } ]

Python keywords used as variables

The source makes use of the following keywords or global functions as variables: input, type, id, dir. They should probably be renamed to avoid clashing.

Doesn't use docstrings

jsonld.py doesn't use docstrings. This makes it impossible to access the documentation interactively.

Unexpected behavior for language maps for data w/o language

Behavior Encountered:

When compacting a context with a language map, when the data does not have languages assigned, the key specified in the context is not used. When the context specifies a default language, this is also not used when the input format is nquads, but IS used when the input is JSON-LD. This should at least be consistent, and preferably the default language assigned.

Behavior Expected:

When compacting data, the default language specified in the context should be used.

Specification Reference:
https://www.w3.org/TR/json-ld/#string-internationalization

Playground Example:
http://tinyurl.com/hr7rwl8

Test Case:
test-langmap.zip

Related Issues:
digitalbazaar/jsonld.js#151

Uploading pyld to PyPI ?

Hi,

It would be great if pyld was on PyPI, Python Package Index, the CPAN of Python.

It could ease it's installation and the dependancy requirements of other projects.

http://guide.python-distribute.org/quickstart.html

Best regards

Difference between behavior of jsonls.js on Playground and pyld

I've run into an issue where I get a different result from pyld that's different from what the JSON-LD playground produces.

On the JSON-LD playground I changed the "Event" example to this:

{
  "@context": {
    "ical": "http://www.w3.org/2002/12/cal/ical#",
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "ical:dtstart": {
      "@type": "xsd:dateTime"
    }
  },
  "ical": "Lady Gaga Concert",
  "ical:location": "New Orleans Arena, New Orleans, Louisiana, USA",
  "ical:dtstart": "2011-04-09T20:00Z"
}

And the compact result is

{
  "@context": {
    "ical": "http://www.w3.org/2002/12/cal/ical#",
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "ical:dtstart": {
      "@type": "xsd:dateTime"
    }
  },
  "ical": "Lady Gaga Concert",
  "ical:dtstart": "2011-04-09T20:00Z",
  "ical:location": "New Orleans Arena, New Orleans, Louisiana, USA"
}

However in Python 2.7.10

from pyld import jsonld
import json

doc =  {  "ical": "Lady Gaga Concert", "ical:dtstart": "2011-04-09T20:00Z",  "ical:location": "New Orleans Arena, New Orleans, Louisiana, USA"}

context = {"ical": "http://www.w3.org/2002/12/cal/ical#",  "xsd": "http://www.w3.org/2001/XMLSchema#",    "ical:dtstart": { "@type": "xsd:dateTime"}}

print json.dumps(jsonld.compact(doc, context))

prints

{
  "@context": {
    "ical:dtstart": {
      "@type": "xsd:dateTime"
    },
    "ical": "http://www.w3.org/2002/12/cal/ical#",
    "xsd": "http://www.w3.org/2001/XMLSchema#"
  },
  "ical:location": "New Orleans Arena, New Orleans, Louisiana, USA",
  "ical:dtstart": "2011-04-09T20:00Z"
}

The "ical": "Lady Gaga Concert", line is now missing.

This may seem pointless, but I am developing "normalized" metadata where I want the synonyms from different vocabularies to be used interchangeably, and I want this to happen at the context level, for example,

doc = dict(start_datetime='2014-01-01T13:00:00', ...)
context_one = dict(start_datetime='http://vocabulary-one.com/start_date', ...)
context_two = dict(start_datetime='http://vocabulary-two.com/coverage_start_datetime', ...)

Then depending on which context is used, different linked data is produced. Is there perhaps an alternative, more correct way of approaching this?

PEP8 compliance: variable naming

The source does not comply with PEP8 coding standards. Some examples

=======         =============
current         suggested fix
=======        =============
usedCtx         used_ctx
parentProperty  parent_property
expandSubjects  expanded_subjects

Exception handling error

Line 4465 of jsonld.py (JsonLdError.str) reads "rval = repr(self.message)". The Exception class doesn't have a 'message' attribute so this throws a second exception. Changing the line to "rval = repr(self)" seems to solve the problem.

I'm having problems installing module

I tried installing pyld on both Mac OS X 10.6.8 and ubuntu 12.04 without success in either case.

On the Mac:

(epd1)Raymond-Yee-Computer:pyld raymondyee$ python setup.py install
/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'playforms'
  warnings.warn(msg)
running install
running bdist_egg
running egg_info
writing pyld.egg-info/PKG-INFO
writing top-level names to pyld.egg-info/top_level.txt
writing dependency_links to pyld.egg-info/dependency_links.txt
file pyld.py (for module pyld) not found
reading manifest file 'pyld.egg-info/SOURCES.txt'
writing manifest file 'pyld.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.5-i386/egg
running install_lib
running build_py
file pyld.py (for module pyld) not found
file pyld.py (for module pyld) not found
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

creating build/bdist.macosx-10.5-i386/egg
creating build/bdist.macosx-10.5-i386/egg/EGG-INFO
copying pyld.egg-info/PKG-INFO -> build/bdist.macosx-10.5-i386/egg/EGG-INFO
copying pyld.egg-info/SOURCES.txt -> build/bdist.macosx-10.5-i386/egg/EGG-INFO
copying pyld.egg-info/dependency_links.txt -> build/bdist.macosx-10.5-i386/egg/EGG-INFO
copying pyld.egg-info/top_level.txt -> build/bdist.macosx-10.5-i386/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/pyld-0.0.1-py2.7.egg' and adding 'build/bdist.macosx-10.5-i386/egg' to it
removing 'build/bdist.macosx-10.5-i386/egg' (and everything under it)
Processing pyld-0.0.1-py2.7.egg
removing '/Users/raymondyee/.virtualenvs/epd1/lib/python2.7/site-packages/pyld-0.0.1-py2.7.egg' (and everything under it)
creating /Users/raymondyee/.virtualenvs/epd1/lib/python2.7/site-packages/pyld-0.0.1-py2.7.egg
Extracting pyld-0.0.1-py2.7.egg to /Users/raymondyee/.virtualenvs/epd1/lib/python2.7/site-packages
pyld 0.0.1 is already the active version in easy-install.pth

Installed /Users/raymondyee/.virtualenvs/epd1/lib/python2.7/site-packages/pyld-0.0.1-py2.7.egg
Processing dependencies for pyld==0.0.1
Finished processing dependencies for pyld==0.0.1

Then:

Python 2.7.3 |CUSTOM| (default, Apr 12 2012, 11:28:34) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyld
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pyld

On Ubuntu, no error messages with

python setup.py install

but I could not import the module either.

It is not always relevant to silently ignore unrecognized properties

Consider the following code:

    CTX={ "foo": "http://example.com/foo"}
    expanded = jsonld.expand({"fooo":"bar"}, {'expandContext': CTX})

The result is an empty list, because "fooo" is not recognized by CTX. The fact that it is silently ignore can be relevant in some situations, but in not in others. In this example, it is probably a typo, which the user would probably like to be detected.

So I think there should be a way to be notified when some JSON attributes were ignored.

The simplest (but not flexible) way to do it would be to have a strict option, defaulting to False. When set to True, an exception would be raised whenever a JSON key could not be converted to a IRI. In my application, I would rather fail than silently drop some data submitted by the user.

A more flexible solution would be to have a droppedKeys option, expecting a set, and populating this set with every key that could not be converted to an IRI.

    CTX={ "foo": "http://example.com/foo"}
    dropped = set()
    expanded = jsonld.expand({"fooo":"bar"}, {'expandContext': CTX, 'droppedKeys': dropped})
    if dropped:
        # print some warning message to the user
    # but still process expended

ValueError: no path specified

When I run the following testcase, I get an exception "ValueError: no path specified". Our codebase is up-to-date as of 12:30p Eastern 3 October. I think you had a similar error about a week ago, but it was fixed.

import pprint
from pyld import jsonld

jsond = {
    "@context": {
        "ns": "http://example.com/core#",
    },
    "ns:associated": [
        { "@id": "#Light", }
    ]
}


contextd = jsond.get("@contextd") or {}

resultd = jsonld.compact(jsond, contextd)
pprint.pprint(resultd)

@type value should be an IRI

According to section 3.3 of JSON-LD (http://json-ld.org/spec/latest/json-ld-syntax/#specifying-the-type) , the value of the "@type" property should always be an IRI. So the following code

  pyld.jsonld.triples({"@type": "http://example.com/Thing"})

should generate

  [{'o': {'@id': 'http://example.com/Thing'}, 'p': '@type', 's': '_:c14n0'}]

but it generates

  [{'o': 'http://example.com/Thing', 'p': '@type', 's': '_:c14n0'}]

"flatten" doesn't play well with @base IRIs with a "/" only path component

Test code

import pprint

from pyld import jsonld

contextd = {
    "@base": "http://example.com/",
    "iot": "http://example.com/core/#",
}

jsond = {
    "iot:associated": [ { "@id": "#Light", } ]
}


compactd = jsonld.compact(jsond, contextd)
flattends = jsonld.flatten(compactd)
pprint.pprint(flattends)

Result

If @base is "http://example.com/", the @id for light is 'http://example.com//#Light' - note the double slash. I would expect it to be a single "/".

If you remove the trailing slash from example.com, no slash appears at all - this is expected.

If you have a path component beyond the single slash, e.g. "http://example.com/bla/" things work as expected also.

PEP8 compliance: comparisons

The code currently implements comparison functions without making use of Python's idioms. This breaches PEP8 and means that the code is less re-usable:

When implementing ordering operations with rich comparisons, it is best to
implement all six operations (__eq__, __ne__, __lt__, __le__, __gt__,
__ge__) rather than relying on other code to only exercise a particular
comparison.

hashlib doesn't accept unicode

I suggest changing

md.update(''.join(nquads))

to

md.update(''.join(nquads).encode('utf-8'))

in line 1919 (jsonld.py).

hashlib doesn't accept unicode as argument and will convert it to str using ascii encoding by default. This will fail if there are non ascii characters in the argument. Adding .encode('utf-8') will prevent using the default encoding.

Round trip to/from RDF fails

Taking the example data from the README:

>>> doc = {
    "http://schema.org/name": "Manu Sporny",
    "http://schema.org/url": {"@id": "http://manu.sporny.org/"},
    "http://schema.org/image": {"@id": "http://manu.sporny.org/images/manu.png"}
}

>>> from pyld import jsonld
>>> open('/tmp/test.nt', 'w+').write(jsonld.to_rdf(doc, options={'format':"application/nquads"}))
>>> jsonld.from_rdf('/tmp/test.nt', {'format':"application/nquads"})
....
   1485                 raise JsonLdError(
   1486                     'Error while parsing N-Quads invalid quad.',
-> 1487                     'jsonld.ParseError', {'line': line_number})
   1488             match = match.groups()
   1489 

JsonLdError: ('Error while parsing N-Quads invalid quad.',)
Type: jsonld.ParseError
Details: {'line': 1}

RDF data look legit:

% cat /tmp/test.nt
_:b0 <http://schema.org/image> <http://manu.sporny.org/images/manu.png> .
_:b0 <http://schema.org/name> "Manu Sporny" .
_:b0 <http://schema.org/url> <http://manu.sporny.org/> .

@list not working?

I'm having trouble either understanding how to use pyld, or how framing works -- or both. In the hopes that it's just the former I was wondering why this doesn't work as I'm expecting:

import json
import pyld

context = {
    "@context": {
        "sc": "http://www.shared-canvas.org/ns/",
        "images": {
            "@type": "@id",
            "@id": "sc:hasImageAnnotations",
            "@container": "@list"
        }
    }
}

description = {
    "@id": "http://example.com/u1",
    "http://www.shared-canvas.org/ns/hasImageAnnotations": [
      {
        "@id": "http://example.com/u3"
      },
      {
        "@id": "http://example.com/u2"
      }
    ]
}

framed_json = pyld.jsonld.frame(description, context)
print json.dumps(framed_json, indent=2)

The result seems to ignore my images property I've defined in the context:

{
  "@context": {
    "sc": "http://www.shared-canvas.org/ns/",
    "images": {
      "@id": "sc:hasImageAnnotations",
      "@type": "@id",
      "@container": "@list"
    }
  },
  "@graph": [
    {
      "sc:hasImageAnnotations": [
        {
          "@id": "http://example.com/u3"
        },
        {
          "@id": "http://example.com/u2"
        }
      ],
      "@id": "http://example.com/u1"
    },
    {
      "@id": "http://example.com/u2"
    },
    {
      "@id": "http://example.com/u3"
    }
  ]
}

However if I comment out the @container in the context:

context = {
    "@context": {
        "sc": "http://www.shared-canvas.org/ns/",
        "images": {
            "@type": "@id",
            "@id": "sc:hasImageAnnotations",
           # "@container": "@list"
        }
    }
}

it seems to work as I expect:

{
  "@context": {
    "sc": "http://www.shared-canvas.org/ns/",
    "images": {
      "@id": "sc:hasImageAnnotations",
      "@type": "@id"
    }
  },
  "@graph": [
    {
      "images": [
        "http://example.com/u3",
        "http://example.com/u2"
      ],
      "@id": "http://example.com/u1"
    },
    {
      "@id": "http://example.com/u2"
    },
    {
      "@id": "http://example.com/u3"
    }
  ]
}

Can anyone help me understand what I'm doing wrong?

typo in compactIRI

Line 128 on jsonld.py reads:
usedCtx[key] = iri
it should be
usedCtx[rval] = iri

Compacting with @vocab does not include already compacted terms

print(jsonld.compact(
    {'name': 'Bob'}, 
    {"@vocab": "http://example.org#"}
))

I would expect this to return

{
  "@context": {
    "@vocab": "http://example.org#"
  },
  "name": "Bob"
}

But instead i got just context:

{
  "@context": {
    "@vocab": "http://example.org#"
  }
}

I checked https://json-ld.org/playground/ - it handles this case correctly.

Compacting failure with @vocab

from pprint import pprint
ctx = {'@vocab': 'http://ex.org/#', 'path': {'@type': '@id'}}
data = {
    'http://ex.org/#maxCount': 1,
    'http://ex.org/#path': 'http://ex.org/#shortname'
}
pprint(jsonld.compact(data, ctx))
# {
#   '@context': {'@vocab': 'http://ex.org/#', 'path': {'@type': '@id'}},
#    # NOTE: maxCount: not declared in context => compacted OK
#   'maxCount': 1
#    # NOTE: path: declared in context => compacting FAILED
#   'http://ex.org/#path': 'http://ex.org/#shortname',
# }

HTTP fetches fail when server responds with gzip-encoded content

I was testing pulling gist content from http://rawgit.com/ while working on developing something, and discovered that pyld.jsonld.load_document() is failing when the server returns a response that is compressed using gzip:

$ curl -I http://rawgit.com/anarchivist/efe97e78a98d2849bfba/raw/collection.json
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 15 May 2014 00:12:54 GMT
Content-Type: application/json; charset=utf-8
Access-Control-Allow-Origin: *
X-Content-Type-Options: nosniff
X-Robots-Tag: none
RawGit-Naughtiness: 0
ETag: "0884db85d95531d7c71f675425a27f32"
Cache-Control: max-age=300
Vary: Accept-Encoding
Content-Encoding: gzip
RawGit-Cache-Status: HIT
Connection: keep-alive

To reproduce:

>>> from pyld import jsonld
>>> j = jsonld.to_rdf("http://rawgit.com/anarchivist/efe97e78a98d2849bfba/raw/collection.json", {"format": "application/nquads"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/pyld/jsonld.py", line 225, in to_rdf
    return JsonLdProcessor().to_rdf(input_, options)
  File "/usr/local/lib/python2.7/site-packages/pyld/jsonld.py", line 946, in to_rdf
    'RDF.', 'jsonld.RdfError', cause=cause)
pyld.jsonld.JsonLdError: 'Could not expand input before serialization to RDF.'
Type: jsonld.RdfError
Cause: 'Could not retrieve a JSON-LD document from the URL.'
Type: jsonld.LoadDocumentError
Code: loading document failed
Cause: 'utf8' codec can't decode byte 0x8b in position 1: invalid start byte  File "/usr/local/lib/python2.7/site-packages/pyld/jsonld.py", line 312, in load_document
    'document': handle.read().decode('utf8')
  File "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
  File "/usr/local/lib/python2.7/site-packages/pyld/jsonld.py", line 942, in to_rdf
    expanded = self.expand(input_, options)
  File "/usr/local/lib/python2.7/site-packages/pyld/jsonld.py", line 618, in expand
    remote_doc = options['documentLoader'](input_)
  File "/usr/local/lib/python2.7/site-packages/pyld/jsonld.py", line 338, in load_document
    cause=cause)

Cannot recognize "defines" property in ontology

I have a context like:

 "@context": 
    {"dbpedia": "http://dbpedia.org/ontology/", 
     "rdfs": "http://www.w3.org/2000/01/rdf-schema#", 
     "owl": "http://www.w3.org/2002/07/owl#", 
     "@base": "<myurl>/astronomy", 
     "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#", 
     "skos": "http://www.w3.org/2004/02/skos/core#", 
     "schema": "https://schema.org/", 
     "defines": {"@reverse": "rdfs:isDefinedBy"}, 
   } 

and then in the same JSON-LD a "defines" to set classes and properties:

 "@id": "",
 "@type": ['http://www.w3.org/2002/07/owl#Ontology']
 "defines" : [{
     "@id": <some_url>
 },...
 ]

I can correctly convert this same JSON-LD in the playground, but there is no way of making PyLD to print out in compacted or expanded mode the very same.

compacted = jsonld.compact(body, context)

returns only @id and @type of the ontology but anything else.

Upload a new version of PyLD on PyPI

Hi,

It would be great if you could upload a new release of PyLD on PyPI.

There has been a lot of changes in PyLD (@reverse for instance) that worth doing a new release.

Best regards

HTTPS SNI Verification fails in python 2

this is related to #57
Running on python 2.7.11 and attempting to load a remote context url hosted on github.io. It seems that the HTTPS handler that was included in this package is circumventing the native HTTPS handling backported into python past 2.7.9.

This is the stacktrace:

Type: jsonld.CompactError
Cause: ('Could not process JSON-LD context.',)
Type: jsonld.ContextError
Cause: ('Dereferencing a URL did not result in a valid JSON-LD context.',)
Type: jsonld.ContextUrlError
Code: loading remote context failed
Details: {'url': <github.io url>}
Cause: ('Could not retrieve a JSON-LD document from the URL.',)
Type: jsonld.LoadDocumentError
Code: loading document failed
Cause: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>  File "/Users/user/venvs/env/lib/python2.7/site-packages/pyld/jsonld.py", line 390, in load_document
    with closing(url_opener.open(url)) as handle:
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/Users/user/venvs/env/lib/python2.7/site-packages/pyld/jsonld.py", line 5135, in https_open
    return self.do_open(self.specialized_conn_class, req)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1197, in do_open
    raise URLError(err)
  File "/Users/user/venvs/env/lib/python2.7/site-packages/pyld/jsonld.py", line 4072, in _retrieve_context_urls
    remote_doc = load_document(url)
  File "/Users/user/venvs/env/lib/python2.7/site-packages/pyld/jsonld.py", line 430, in load_document
    cause=cause)
  File "/Users/user/venvs/env/lib/python2.7/site-packages/pyld/jsonld.py", line 1182, in process_context
    local_ctx, {}, options['documentLoader'], options['base'])
  File "/Users/user/venvs/env/lib/python2.7/site-packages/pyld/jsonld.py", line 4079, in _retrieve_context_urls
    code='loading remote context failed', cause=cause)
  File "/Users/user/venvs/env/lib/python2.7/site-packages/pyld/jsonld.py", line 688, in compact
    active_ctx = self.process_context(active_ctx, ctx, options)
  File "/Users/user/venvs/env/lib/python2.7/site-packages/pyld/jsonld.py", line 1186, in process_context
    'jsonld.ContextError', cause=cause)

Pypi package does not installs - require requests

Hi,

I'm installing PyLD into a (almost empty) docker container, for a django project.

I've added PyLD into a requirement file, and I've the following error when installing the requirements:

...
Collecting PyLD (from -r requirements/common.txt (line 42))
  Downloading PyLD-0.8.0.tar.gz (44kB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-0b391b8m/PyLD/setup.py", line 17, in <module>
        from pyld import jsonld
      File "/tmp/pip-build-0b391b8m/PyLD/lib/pyld/__init__.py", line 2, in <module>
        from . import jsonld
      File "/tmp/pip-build-0b391b8m/PyLD/lib/pyld/jsonld.py", line 19, in <module>
        import requests
    ImportError: No module named 'requests'

Looking in the setup.py file, I can see at line 17:

from pyld import jsonld

where an attempt to load the current installing module, which needs requests, which is not installled because it is a requirement of the current installing module...

I think you shouldn't try to load the currently installing module, the setup.py file is used both to create the package (sdist) and install it.

Thanks for upgrading the pypi package !

No usage instructions

The README doesn't explain how to use the library. Could some documentation please be added about how to use 'pyld'

Dates from schema.org are not compacted correctly.

I've traced this down to https://github.com/digitalbazaar/pyld/blob/master/lib/pyld/jsonld.py#L4132

The original symptoms that I observed were that various Date fields that should be compacted to keys like birthDate were actually including a schema: prefix.

It looks like the core problem is that the default inverse dictionary for @language is not getting populated correctly (perhaps due to a change in the published @context from schema.org?)

Date field specifications include both @id and @type, but not language, and are still expected to be pure strings (according to schema.org's documentation). When the compactor attempts to find the correct term, [_select_term]() returns None.

The inverse dictionary winds up looking something like this:

{
    'http://schema.org/birthDate': {'@none': {'@language': {},
                                              '@type': {'http://schema.org/Date': 'birthDate'}}},
    # etc
}

Instead of containing a {'@none': 'birthDate'} for @language, there's just an empty dict.

Some other non-date fields also seem to exhibit this issue, but I don't know enough about the library or json-ld to know if these symptoms are actually problems, or if they're by design.

Minimum-reproducible sample:

#!/usr/bin/env python

from pyld import jsonld

doc = {
    'http://schema.org/name': 'Buster the Cat',
    'http://schema.org/birthDate': '2012',
    'http://schema.org/deathDate': '2015-02-25'
}

frame = {
    '@context': 'http://schema.org/'
}

framed = jsonld.frame(doc, frame)
contents = framed['@graph'][0]
print(framed)
assert 'name' in contents  # fine
assert 'birthDate' in contents  # not fine, schema:birthDate instead
assert 'deathDate' in contents  # not fine, schema:deathDate instead

My proposal to fix this would be to apply Artory@faaa139, to attempt to set these defaults regardless of the outcome of the conditionals there.

framing with "@embed": "@link" cannot handle circular references

Framing with "@embed": "@link" has been proposed as one way to create an in-memory representation of a graph (as discussed in json-ld/json-ld.org#140 ).

jsonld.js does this in a way that handles circular references, but when I try to use pyld to frame:

{
  "@context": "http://schema.org/",
  "@type": "Person",
  "name": "Jane Doe",
  "jobTitle": "Professor",
  "telephone": "[phone]",
  "@id": "http://www.janedoe.com",
  "knows": {
    "name": "John Smith",
    "@type": "Person",
    "@id": "http://www.johnsmith.me",
    "knows": {
      "@id": "http://www.janedoe.com"
    }
  }
}

with a frame like:

{
  "@context": "http://schema.org",
  "@embed": "@link"
}

... I get a RuntimeError from exceeding the recursion limit.

I also demonstrate this in a gist