inventaire / entities-search-engine

Scripts and microservice to feed an ElasticSearch with Wikidata and Inventaire entities, and keep those up-to-date

Inventaire

Libre collaborative resource mapper powered by open-knowledge

This repository hosts Inventaire.io source code. Its a collaborative resources mapper project, while yet only focused on exploring books mapping with wikidata and ISBNs

This repository tracks server-side developments, while the (heavy) client-side can be found here. Client-related technical issues should go in the client repo, while this repo gathers all other technical issues. Non-technical discussions, such as feature requests, should preferably happen in the chat.

Summary

• Installation
  • Dependencies to install manually
  • Project development environment installation
    • server
      • emails
    • client
  • Installation tips
  • Repositories and Branches
    • server
    • client
    • i18n
    • deploy
    • docker
• Stack Map
• Concepts map
• Contribute
• Documentation
• Wiki
• API
• Administration
• License

Installation

This is the installation documentation for a developement environment. For production setup, see: inventaire-deploy

Dependencies to install manually

• git, curl (used in some installation scripts), graphicsmagick (used to resize images), inotify-tools (used in API tests scripts):
• NodeJS (>=10, using the latest LTS is recommended), NVM (allows greater version update flexibility)
• a CouchDB (>=3.1) instance (on port 5984 for default config)
• an Elasticsearch (>=7.10) instance (on port 9200 for default config)

To install all this those dependencies on Ubuntu 20.04:

For packages available in Ubuntu default repositories:

sudo apt-get update
sudo apt-get install git curl wget graphicsmagick inotify-tools

For packages that need a more elaborated installation, see their own documentation:

• Install NodeJS latest LTS via NVM
• Install CouchDB
• Install ElasticSearch

Alternatively, CouchDB and Elasticsearch could be run in Docker, see docker-inventaire

Whatever the way you installed CouchDB and Elasticsearch, you should now be able to get a response from them:

# Verify that CouchDB is up
curl http://localhost:5984
# Verify that Elasticsearch is up
curl http://localhost:9200

Project development environment installation

git clone https://github.com/inventaire/inventaire.git
cd inventaire
npm install --global tsx
npm install

This should have installed:

• the server (this git repository) in the current directory
• the client (inventaire-client) in the client directory
• i18n strings (inventaire-i18n) in the inventaire-i18n directory

server

The installation step above should have triggered the creation of a ./config/local.cjs file, in which you can override all present in ./config/default.cjs: make sure to set db username and password to your CouchDB username and password.

And now you should be all set! You can now start the server (on port 3006 by default)

# Starting the server in watch mode so that it reboots on file changes
npm run watch

emails

To debug emails in the browser:

• Get some username and password at https://ethereal.email/create and set the following values in config: mailer.nodemailer.user and mailer.nodemailer.pass.
• Make an action that triggers the email you would like to work on on the local server (ex: send a friend request)
• Open https://ethereal.email/messages to see the generated email

Note that, while convenient, debugging emails in the browser is quite an approximation, as some email clients are antiquated, and, sadly, modern CSS can't be used.

client

If you want to work on the client, you need to start the webpack watcher and dev server (on port 3005 by default)

# In another terminal
cd inventaire/client
npm run watch

Installation tips

• To use executable that are used by the project (such as mocha), you can either find them in ./node_modules/.bin or install them globally with npm: npm install -g tsx mocha lev2 etc.

Repositories and Branches

server

• main: the stable branch. Unstable work should happen in feature-specific branches and trigger pull requests when ready to be merged in the main branch. See Code Contributor Guidelines.

client

• main: the stable branch. Unstable work should happen in feature-specific branches and trigger pull requests when ready to be merged in the main branch. See Code Contributor Guidelines.

i18n

The repository tracking strings used in the server (for emails, activitypub) and client (for the web UI) in all the supported languages. For helping to translate, see the Inventaire Weblate project

• main: tracking translations fetched from Weblate and build scripts

deploy

tracking installation scripts and documentation to run inventaire in production

• main: the main implementation targeting Ubuntu 16.04. Additional branches can be started to document installation on other environments

docker

• main: tracking docker installation files for development and testing use

Stack Map

This repository correspond to the the "Server" section in the stack map

Concepts map

the app has a few core concepts:

• Users
• Entities : which can be authors (ex: wd:Q353), books (ex: wd:Q393018) and books' specific editions (ex: isbn:9782070389162). The term entities comes from wikidata terminology. See the entities map.
• Items : instances of book entities that a user says they have. It can be an instance of a work or a specific edition of a work.
• Transactions : discussion between two users about a specific item with an open transaction mode (giving, lending, selling). Transactions have effects on items: giving and selling an item make it move from the owner to the requester inventory; lending an item shows it as unavailable.
• Groups: groups of users with one or more admins

Contribute

For code-related contributions, see How to contribute on wiki.inventaire.io.

Documentation

see docs

Wiki

see wiki.inventaire.io You may want to directly go to the technical wiki page

API

see wiki: API

Administration

see Administration

License

Inventaire is an open-sourced project licensed under AGPLv3.

entities-search-engine's Issues

Impoting all of data

how can i insert all of data to elastic search so that i can can wikidata object id by its title

import entities in ES with all the claims

Hi
I am trying to create an ES instance to search for music bands, but I'd like the results to return all claims related to a band. Basically, when searching for "Metallica" I'd like our local ES instance to return the same amount of data as this api query https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&ids=Q15920

I am using this command to do the first import:
cat latest-all.json.gz | gzip -d | wdfilter --keep id,type,labels,descriptions,aliases,claims,sitelinks --claim P31:Q215380 | import_to_elastic music

As far as I can tell, wdfilter is doing its part of the job and extracting the full entities (I tested it on its own and the output seemed more complete than what I could see in the ES response), but then import_to_elastic seems to filter that and remove all the claims before inserting them in the ES index.

Is there a way to achieve the result I am looking for?

Thanks!

BR
Francesco

Docker Image

not actually an issue, rather a question, is there a Docker image available for this entities-search-engine?

Bulk post res failed

I was importing to elasticsearch from the downloaded dump, (ElasticSearch: version 6.3.0)
and got the following error.

****** bulk post res failed ******
{ index:
{ _index: 'wikidata',
_type: 'humans',
_id: 'Q44304637',
status: 400,
error:
{ type: 'illegal_argument_exception',
reason: 'Limit of total fields [1000] in index [wikidata] has been exceeded' } } }

Below is the latest mapping of wikidata index.
https://gist.github.com/junskang/5ebf53a7b540a56ac8bee86b366011c7

import_to_elastic - Content-Type header is Missing

As shown in the documentation, I'm running the command below to filter and index Wikidata.json

cat wikidata-dump.json.gz | gzip -d | wdfilter --claim P31:Q5 --omit type,sitelinks | import_to_elastic humans

However, I'm getting this error:

****** bulk post err body ******
{"error":"Content-Type header [] is not supported","status":406}
-----
12100
12200
12300
12400
12500
12600
12700
12800
12900
****** putting batch... ******
1000
-----
13000
****** bulk post err ******
Error: Not Acceptable
    at handleHttpErrorCode (/Users/hs/BigData/wikidata-subset-search-engine/node_modules/bluereq/lib/promisified_request.js:27:15)
    at Request._callback (/Users/hs/BigData/wikidata-subset-search-engine/node_modules/bluereq/lib/promisified_request.js:20:9)
    at Request.self.callback (/Users/hs/BigData/wikidata-subset-search-engine/node_modules/request/request.js:186:22)
    at Request.emit (events.js:127:13)
    at Request.<anonymous> (/Users/hs/BigData/wikidata-subset-search-engine/node_modules/request/request.js:1163:10)
    at Request.emit (events.js:127:13)
    at Gunzip.<anonymous> (/Users/hs/BigData/wikidata-subset-search-engine/node_modules/request/request.js:1085:12)
    at Object.onceWrapper (events.js:219:13)
    at Gunzip.emit (events.js:132:15)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19)

-----

I assume the header should be JSON, but I'm not sure why it's empty

Please advise

un-index redirected entities

see inventaire/inventaire#92

Error: Cannot find module '../../package.json'

cat latest-all.json | wdfilter --claim P31:Q5 --omit type,claims,sitelinks | import_to_elastic humans

module.js:472
throw err;
^

Error: Cannot find module '../../package.json'
at Function.Module._resolveFilename (module.js:470:15)
at Function.Module._load (module.js:418:25)
at Module.require (module.js:498:17)
at require (internal/module.js:20:19)
at Object. (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/wikidata-filter/lib/program.js:2:13)
at Module._compile (module.js:571:32)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:488:32)
at tryModuleLoad (module.js:447:12)
at Function.Module._load (module.js:439:3)
TypeError: Cannot read property 'bind' of undefined
at Object. (/media/data/server/similarist/wikidata-subset-search-engine/lib/import.coffee:45:19)
at Object. (/media/data/server/similarist/wikidata-subset-search-engine/lib/import.coffee:2:1)
at Module._compile (module.js:571:32)
at Object.exports.run (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/coffee-script/lib/coffee-script/coffee-script.js:173:23)
at compileScript (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/coffee-script/lib/coffee-script/command.js:224:29)
at compilePath (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/coffee-script/lib/coffee-script/command.js:174:14)
at Object.exports.run (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/coffee-script/lib/coffee-script/command.js:98:20)
at Object. (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/coffee-script/bin/coffee:7:41)
at Module._compile (module.js:571:32)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:488:32)
at tryModuleLoad (module.js:447:12)
at Function.Module._load (module.js:439:3)
at Module.runMain (module.js:605:10)
at run (bootstrap_node.js:427:7)
at startup (bootstrap_node.js:151:9)
at bootstrap_node.js:542:3