Bulk post res failed about entities-search-engine

[junskang]

Am I just supposed to update the threshold to higher number?

[junskang]

As a walk around, I created the index and set the limit to 2000 before beginning to import the data to ES. And it did not raise any error (so far).

curl -XPUT "http://localhost:9200/wikidata"

curl -XPUT "http://localhost:9200/wikidata/_settings" -H 'Content-Type: application/json' -d'
{
  "index.mapping.total_fields.limit": 2000
}'

[junskang]

2000 failed also.

2018-06-29T16:25:37,324][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [R5wjPiw] failed to put mappings on indices [[[wikidata/8zSJEhSJRhaZwLDdKeu5VQ]]], type [humans]
java.lang.IllegalArgumentException: Limit of total fields [2000] in index [wikidata] has been exceeded
at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:463) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:356) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:288) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:313) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:267) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:197) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:132) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.3.0.jar:6.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
[2018-06-29T16:25:37,327][DEBUG][o.e.a.b.TransportShardBulkAction] [wikidata][3] failed to execute bulk item (index) BulkShardRequest [[wikidata][3]] containing [197] requests
java.lang.IllegalArgumentException: Limit of total fields [2000] in index [wikidata] has been exceeded
at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:463) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:356) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:288) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:313) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:267) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:197) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:132) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-6.3.0.jar:6.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

[jum-s]

current ES version has not been updated for a while as the current instance shows :

{
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.4.6",
    "lucene_version" : "5.5.4"
  },
  "tagline" : "You Know, for Search"
}

I would try to downgrade to this version.
you could also take a look at the docker install docs

[junskang]

Thanks. I ended up increasing the threshold to 4000 and it worked. But yeah, it does have potential issues for additional types of documents.