iai-group / pkg-api

PKG API

PKG API: A Tool for Personal Knowledge Graph Management

The PKG API is a tool for managing personal knowledge graphs (PKGs). It provides a simple solution for end users and service providers to administrate and interact with the users' PKGs through natural language statements and simple web forms. The representation of a statement inside the PKG is defined by the PKG vocabulary. Within the API, two key modules are present: one for processing natural language statements (NL2PKG), and another for generating and executing SPARQL queries against the PKG (PKG connector).

The PKG API is served as a RESTful API and we provide a user interface, PKG Client, that allows users to manage their PKG online.

PKG API

NL2PKG

This module is responsible for processing natural language statements. The processing is divided into two steps: (1) natural language understanding handled by annotators and (2) entity_linking.

Available annotators and entity linkers:

• StatementAnnotator
  • ThreeStepStatementAnnotator: Annotates statements using a three-step approach: (1) intent recognition, (2) Subject-Predicate-Object triple extraction, and (3) preference extraction.
• EntityLinker
  • RELEntityLinker: Links entities using Radboud Entity Linker API.
  • SpotlightEntityLinker: Links entities using DBpedia Spotlight.

PKG connector

The PKG connector is responsible for executing SPARQL queries against the PKG. Utilities functions are responsible for generating SPARQL queries based on the intent of the user. For example, if a user wants to add a statement to the PKG, a tailored INSERT query is generated.

Server

The backend server is a Flask server. It is responsible for connecting the users and service providers to PKGs.

Starting the server

Before starting the server, make sure that the requirements are installed and that CORS is disabled in your web browser.

To start the server, run the following command:

flask --app pkg_api/server run --debug

Note the --debug flag is optional, but it is recommended to use it during development.

By default, the server will run locally on port 5000. In case you want to run the server on a different port, you can specify the port using the --port flag.

PKG Client

The user interface is a React application that communicates with the server to manage the PKG. More details on how to run PKG Client can be found here.

⚠️ Note that you need to update PKG_API_BASE_URL in the configuration in case the server is not running on the default port.

Demo

PKG_API_demo.mov

The code for the demo is available in this branch.

Conventions

We follow the IAI Python Style Guide.

Publication

If you use the PKG API, please cite the following paper:

@inproceedings{Bernard:2024:WWW,
  author = {Bernard, Nolwenn and Kostric, Ivica and \L{}ajewska, Weronika and Balog, Krisztian and Galus\v{c}\'{a}kov\'{a}, Petra and Setty, Vinay and Skj\ae{}veland, Martin G.},
  title = {{PKG API}: {A} Tool for Personal Knowledge Graph Management},
  booktitle = {Companion Proceedings of the ACM on Web Conference 2024},
  pages = {1051--1054},
  series = {WWW '24},
  year = {2024},
  doi = {10.1145/3589335.3651247},
}

Contributors

PKG API is developed and maintained by the IAI group at the University of Stavanger.

pkg-api's Issues

Add login managment - Backend

Add login managment - Frontend

Decide how to represent preferences in RDF

We want to represent the fact in RDF that a person has some preference (in [-1,1]) about a given entity (movie, actor, genre, etc.)
This is to be used in set_preference(self, who: URI, entity: URI, preference: float)

Service form frontend

Create a form for manipulating the services (add new service) and adding permissions to these services

Create a string to URI method

Example: tea to http://example.org/tea

Implement getters/setters for preferences for a specific class

Depends on #3

Questions/queries for `remove_fact` (PG)

Create an RDF vocabulary about preferences

Questions/queries for `remove_owner_fact` (PG)

Write tests for get/set preferences

Depends on #3

Write tests for get/set fact

(NLtoAPI) Try other LLMs

Try LLMs like Mistral, GPT-3.5, Google's Gemini etc and compare to mini benchmark.

Create remove_fact and remove_owner_fact methods

Assign @kbalog for review when completed

Think of NL questions related to each API calls

Imagine a few natural language questions/queries with different levels of difficulty related to each API calls:

• #45
• #46
• Questions/queries for get_preference (KB)
• Questions/queries for set_preference (KB)
• #74
• #75
• #48
• #47
• Questions/queries for add_owner_fact (VS) #43
• Questions/queries for add_fact (VS) #44
• #49
• #50

For now, we focus on movie related questions/queries.

@ all please pick some of the items and create associated issues.

Add branch protection

Need approval from one lead

Natural language questions for remove_owner_fact API call

Update pyproject.toml for package release

Set packaging configuration.

Update README

Create new React app

(NLtoAPI) Create skeleton module

Identify specific issues for producing SPARQL according to PKG data model (utils.py)

Questions/queries for `set_owner_preference`

• Save The Godfather as my favourite movie (entity: "The Godfather", preference: 1.0)
• I enjoy watching action movies (entity:"action movies", preference: 1.0)
• I went to the movie theatre yesterday and loved Oppenheimer (entity: "Oppenheimer", preference: 1.0)
• Note that I would never watch cheesy romcom (entity: "cheesy romcom", preference: -1.0)
• I don't mind historical drama (entity: "historical drama", preference: 0.0)
• I enjoy watching action movies with Alice (too complex for current version)
• Note that I would never watch cheesy romcom unless with my friends (too complex for current version)

Questions/queries for `get_owner_preferences`

Suggestions:

• Do I like action movies?
• Do I prefer action movies over romantic comedies?
• What kind of movies do I dislike the most?
• What kind of movies are I undecided about, i.e., do I both like and dislike?

Suggest API to reflect PKG architecture

Include functionality to support user-facing (queries from external services) and backend (how queries translate to backend implementations such as a triple store) requests.

Remove code folder

It's not needed

Add support for BerkeleyDB and SPARQLUpdateStore in connector

Add endpoints for populating the PKG

Endpoint that triggers SPARQL queries to add a fact or a preference.

Add basic Flask endpoints

PKG data model

Decide on a data model what supports representing

• facts
• metadata about the facts
• vague and precise facts
• yet not too complex

I suggest the following data model for capturing statements (see below for an example),

The main structure is that :Statement represents the original user statement/fact/utterance.
A :Statement contains all sorts of logging, points to is structured payload using dc:subject, which can point to multiple resources (IRIs or blank nodes). These resources can be precise IRIs (e.g., the IRI representing the actor Tom Cruise), or more vague, which we can represent using a skos:Concept with a textual description.

Additionally, the :Statement can contain some "analysis" of its payload in the form of "intent" (is this a good word?) and weight. For the latter, we can use an existing vocabulary (old, and unknown to me).

• https://pav-ontology.github.io/pav/
• https://smiy.sourceforge.net/wo/spec/weightingontology.html
• https://smiy.sourceforge.net/wi/spec/weightedinterests.html

[] a :Statement ;

  ## LOGGING OF STATEMENT

  pav:authoredBy       # who made the original statement
    :me ;

  pav:createdBy        # who created this representation of the statement
    :me ;

  #pav:curatedBy        # who helped in furter describing analysing the statement
    
  pav:authoredOn       # when the original statement was made
    "2023-12-15T21:12:40"^^xsd:dateTime ;

  pav:createdOn        # when this representation was made
    "2023-12-15T21:12:40"^^xsd:dateTime ;
  pav:createdAt        # where this representation was made
    [
       # some location
    ] ;    

  ## CONTENT OF STATEMENT

  dc:description      # the original textual description
    "Jane dislikes all movies with the actor Tom Cruise"@en ;

  dc:subject           # what is the statement about, subject, topic
    [ a skos:Concept ;
      dc:description "All movies with the actor Tom Cruise" ;
      skos:related xximdb:Movie, xximdb:TomCruise, xx:Jane ;
      # could try to represent this as a query?
    ] ,
    xximdb:Movie, 
    xximdb:TomCruise, 
    xx:Jane ;

  dc:accessRights ???

  ## "ANALYSIS" OF STATEMENT

  # intent??
                       
  wo:weigth            # weight
    [ wo:weigth_value -1.0 ;
      wo:scale :StandardScale ]
        
:StandardScale a wo:Scale ;
  wo:max_weigth  1.0 ;
  wo:min_weigth -1.0 ;
  wo:step_size   0.1 .

Update fact getters setters

who in add_fact and get_objects_from_facts should instead be subject (it doesn't have to be a person).

In the file-level docstring, explain that (1) a fact is an atomic statement that can be expressed (either about the owner of the PKG or about another entity) and (2) a preference is ...

NL query view frontend

Create a view with a simple text input in which the user write theirs NL queries.

Add RDF store and path arguments to PKG.__init__

Update vocabulary once we choose a domain to host the ontology.

          LGTM. Please create a follow-up issue to update the file once we choose a domain to host the ontology.

Originally posted by @NoB0 in #29 (review)

Add endpoint for access control via form #69

Implement endpoints related to adding a service and adding permissions to these services.

Questions/queries for `get_objects_from_facts`

• List what I watched (subject: "I", predicate: "watched")
• What books did my child read? (subject: "my child", predicate: "read")
• Who does Alice follow? (subject: "Alice", predicate: "follows")
• Remind me which places I want to visit (subject: "I", predicate: "wantToVisit")

Too complex for current version:

• How many action movies are stored in my PKG?
• What genre do my older siblings prefer?
• Where is located the movie theatre I went to last Wednesday?
• Which movies were recommended by my friends?

Create module for translating NL to API

• #61
• [NLtoPKG] Update pipeline to match the paper
• [NLtoPKG] Implement simple three-step annotator
• [NLtoPKG] Create folder with prompts #72
• [NLtoPKG] Add prompts using examples defined in #40
• [NLtoPKG] Create method for loading the prompts based on path
• [NLtoPKG] Create method to call llama
• #81
• [NLtoPKG] (Optional) Create method for calling other LLM models
• [NLtoPKG] (Optional) Implement JointBERT model
• [NLtoPKG] Try other LLMs #68
• [NLtoPKG] Create a microbenchmark for comparing models.
• [NLtoPKG] Compare LLama, Mistral and OpenAI models using the mcirobenchmark.
• [NLtoPKG] Implement off-the-shelf entity linker

Add endpoint for PKG visualisation via form

Implement endpoint to process a filtering query and return relevant part of the PKG for visualisation. For example, displaying only the appointments or movies.

Natural language questions for remove_fact API call

Propose a few natural language questions with different levels of difficulty related to the remove_fact API call.

Create examples for add_fact(), set_preference(), get_owner_objects_from_facts(), +unknown

Create 5 annotated examples for the four selected methods.

Questions/queries for `get_owner_preference`

Suggestions:

• Do I like Pulp Fiction?
• Do I like movies directed by Mel Gibson?
• Do I prefer Pulp Fiction over any of the Rambo movies?
• Do I like movies featuring actors that have played Macbeth? Which of these movies do I prefer?

NL questions for add_fact

Scenario: Populating and updating the PKG owned by Alice for a movie recommender. Alice and Bob are married and live together. Alice's mother Emma visits them sometimes. Movie recommender should be able to recommend movies to watch with Bob or when Emma is also vising them.

• Alice and Bob are a couple
• Bob doesn't like romantic comedies
• Alice's mother Emma likes historical dramas
• Alice and Bob prefer to watch action movies on Fridays
• Bob has seen all the Christopher Nolan movies

Create endpoint for NL query

Implement getters/setters for preferences

This issue includes writing the corresponding SPARQL queries as well.

Depends on #2

NL questions for add_owner_fact

Scenario: Populating and updating the PKG owned by Alice for a movie recommender. Alice and Bob are married and live together. Alice's mother Emma visits them sometimes. Movie recommender should be able to recommend movies to watch with Bob or when Emma is also vising them.

• I am married to Bob
• My husband doesn't like romantic comedies
• Emma is my mother
• We usually watch action movies on Fridays
• We prefer to watch historical dramas when my mother is visiting us
• My husband has seen all the Christopher Nolan movies and a big fan

Create vocabulary class

It should map labels to predicates (and add "pkg:" prefix to all)

Use in SPARQL queries e.g., "...<{subject}> <{pkg_vocabulary.entity}> <{entity}>"

Add endpoint for population via form

Implement endpoints related to adding a fact or preference, and removing a fact.

Replace codecov by comment in build workflow

Add endpoints for querying the PKG

Endpoints that trigger a SPARQL query to retrieve facts and preferences.

Population form frontend view

Create a view displaying a form to populate the PKG.

Include services in population form view

Include a section in the form to enable the user to select the access rights of available services.