inventaire / entities-search-engine Goto Github PK

how can i insert all of data to elastic search so that i can can wikidata object id by its title

As shown in the documentation, I'm running the command below to filter and index Wikidata.json

cat wikidata-dump.json.gz | gzip -d | wdfilter --claim P31:Q5 --omit type,sitelinks | import_to_elastic humans

However, I'm getting this error:

****** bulk post err body ******
{"error":"Content-Type header [] is not supported","status":406}
-----
12100
12200
12300
12400
12500
12600
12700
12800
12900
****** putting batch... ******
1000
-----
13000
****** bulk post err ******
Error: Not Acceptable
    at handleHttpErrorCode (/Users/hs/BigData/wikidata-subset-search-engine/node_modules/bluereq/lib/promisified_request.js:27:15)
    at Request._callback (/Users/hs/BigData/wikidata-subset-search-engine/node_modules/bluereq/lib/promisified_request.js:20:9)
    at Request.self.callback (/Users/hs/BigData/wikidata-subset-search-engine/node_modules/request/request.js:186:22)
    at Request.emit (events.js:127:13)
    at Request.<anonymous> (/Users/hs/BigData/wikidata-subset-search-engine/node_modules/request/request.js:1163:10)
    at Request.emit (events.js:127:13)
    at Gunzip.<anonymous> (/Users/hs/BigData/wikidata-subset-search-engine/node_modules/request/request.js:1085:12)
    at Object.onceWrapper (events.js:219:13)
    at Gunzip.emit (events.js:132:15)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19)

-----

I assume the header should be JSON, but I'm not sure why it's empty

Please advise

not actually an issue, rather a question, is there a Docker image available for this entities-search-engine?

cat latest-all.json | wdfilter --claim P31:Q5 --omit type,claims,sitelinks | import_to_elastic humans

module.js:472
throw err;
^

Error: Cannot find module '../../package.json'
at Function.Module._resolveFilename (module.js:470:15)
at Function.Module._load (module.js:418:25)
at Module.require (module.js:498:17)
at require (internal/module.js:20:19)
at Object. (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/wikidata-filter/lib/program.js:2:13)
at Module._compile (module.js:571:32)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:488:32)
at tryModuleLoad (module.js:447:12)
at Function.Module._load (module.js:439:3)
TypeError: Cannot read property 'bind' of undefined
at Object. (/media/data/server/similarist/wikidata-subset-search-engine/lib/import.coffee:45:19)
at Object. (/media/data/server/similarist/wikidata-subset-search-engine/lib/import.coffee:2:1)
at Module._compile (module.js:571:32)
at Object.exports.run (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/coffee-script/lib/coffee-script/coffee-script.js:173:23)
at compileScript (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/coffee-script/lib/coffee-script/command.js:224:29)
at compilePath (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/coffee-script/lib/coffee-script/command.js:174:14)
at Object.exports.run (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/coffee-script/lib/coffee-script/command.js:98:20)
at Object. (/media/data/server/similarist/wikidata-subset-search-engine/node_modules/coffee-script/bin/coffee:7:41)
at Module._compile (module.js:571:32)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:488:32)
at tryModuleLoad (module.js:447:12)
at Function.Module._load (module.js:439:3)
at Module.runMain (module.js:605:10)
at run (bootstrap_node.js:427:7)
at startup (bootstrap_node.js:151:9)
at bootstrap_node.js:542:3

see inventaire/inventaire#92

Hi
I am trying to create an ES instance to search for music bands, but I'd like the results to return all claims related to a band. Basically, when searching for "Metallica" I'd like our local ES instance to return the same amount of data as this api query https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&ids=Q15920

I am using this command to do the first import:
cat latest-all.json.gz | gzip -d | wdfilter --keep id,type,labels,descriptions,aliases,claims,sitelinks --claim P31:Q215380 | import_to_elastic music

As far as I can tell, wdfilter is doing its part of the job and extracting the full entities (I tested it on its own and the output seemed more complete than what I could see in the ES response), but then import_to_elastic seems to filter that and remove all the claims before inserting them in the ES index.

Is there a way to achieve the result I am looking for?

Thanks!

BR
Francesco

I was importing to elasticsearch from the downloaded dump, (ElasticSearch: version 6.3.0)
and got the following error.

****** bulk post res failed ******
{ index:
{ _index: 'wikidata',
_type: 'humans',
_id: 'Q44304637',
status: 400,
error:
{ type: 'illegal_argument_exception',
reason: 'Limit of total fields [1000] in index [wikidata] has been exceeded' } } }

Below is the latest mapping of wikidata index.
https://gist.github.com/junskang/5ebf53a7b540a56ac8bee86b366011c7