ES5 "org.elasticsearch.index.mapper.MapperParsingException: failed to parse [fields.0]" on receiving end when using anonymous destructuring. about serilog-sinks-elasticsearch

[Rzpeg]

Forgot to add, I have the following packages installed for the test console app.

<packages>
  <package id="Elasticsearch.Net" version="2.4.6" targetFramework="net461" />
  <package id="Serilog" version="2.3.0" targetFramework="net461" />
  <package id="Serilog.Sinks.Elasticsearch" version="4.1.1" targetFramework="net461" />
  <package id="Serilog.Sinks.File" version="3.1.1" targetFramework="net461" />
  <package id="Serilog.Sinks.PeriodicBatching" version="2.1.0" targetFramework="net461" />
  <package id="Serilog.Sinks.RollingFile" version="3.2.0" targetFramework="net461" />
</packages>

[Rzpeg]

I have been toying with Elasticsearch API (via Sense) and if I run the following request without registering any templates, I get the entry created.

POST _bulk
{"index":{"_index":"logstash-2016.11.17","_type":"logevent"}}
{"@timestamp":"2016-11-17T14:23:57.9886572+00:00","level":"Information","messageTemplate":"Here is some stuff {@0}","message":"Here is some stuff { NowIs: 11/17/2016 14:23:57, SomeStuff: \"something\" }","fields":{"0":{"NowIs":"2016-11-17T14:23:57.9561564Z","SomeStuff":"something"}}}

{
   "took": 16,
   "errors": false,
   "items": [
      {
         "index": {
            "_index": "logstash-2016.11.17",
            "_type": "logevent",
            "_id": "AVhy0hLmTiBab5jV7DWH",
            "_version": 1,
            "result": "created",
            "_shards": {
               "total": 2,
               "successful": 1,
               "failed": 0
            },
            "created": true,
            "status": 201
         }
      }
   ]
}

If I run the sink's template, the same _bulk query results in error:

{
   "took": 331,
   "errors": true,
   "items": [
      {
         "index": {
            "_index": "logstash-2016.11.17",
            "_type": "logevent",
            "_id": "AVhy3diRTiBab5jV7DWQ",
            "status": 400,
            "error": {
               "type": "mapper_parsing_exception",
               "reason": "failed to parse [fields.0]",
               "caused_by": {
                  "type": "illegal_state_exception",
                  "reason": "Can't get text on a START_OBJECT at 1:219"
               }
            }
         }
      }
   ]
}

The created template is :

{
  "serilog-events-template": {
    "order": 0,
    "template": "logstash-*",
    "settings": {
      "index": {
        "refresh_interval": "5s"
      }
    },
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "numerics_in_fields": {
              "match_pattern": "regex",
              "path_match": "fields\\.[\\d+]$",
              "mapping": {
                "index": "analyzed",
                "omit_norms": true,
                "type": "string"
              }
            }
          },
          {
            "string_fields": {
              "mapping": {
                "index": "analyzed",
                "omit_norms": true,
                "type": "string",
                "fields": {
                  "raw": {
                    "ignore_above": 256,
                    "index": "not_analyzed",
                    "type": "string"
                  }
                }
              },
              "match_mapping_type": "string",
              "match": "*"
            }
          }
        ],
        "_all": {
          "omit_norms": true,
          "enabled": true
        },
        "properties": {
          "message": {
            "index": "analyzed",
            "type": "string"
          },
          "exceptions": {
            "type": "nested",
            "properties": {
              "ExceptionMessage": {
                "type": "object",
                "properties": {
                  "MemberType": {
                    "type": "integer"
                  }
                }
              },
              "StackTraceString": {
                "index": "analyzed",
                "type": "string"
              },
              "HResult": {
                "type": "integer"
              },
              "RemoteStackTraceString": {
                "index": "analyzed",
                "type": "string"
              },
              "RemoteStackIndex": {
                "type": "integer"
              },
              "Depth": {
                "type": "integer"
              }
            }
          }
        }
      }
    },
    "aliases": {}
  }
}

[ffMathy]

I think this repository might be dead, which is unfortunate. We should maybe fork it and create our own.

[mivano]

The default template mapping tries to force the default 0, 1, etc fields to be strings in order to support old applications where you log for example:

 Log.Information("log line 1", 29292);
 Log.Information("log line 2", "text");

Without the mapping, the fields.0 will become a integer field (as that is the first event that gets consumed) and then ES will fail to parse the string. So we force the 0,1,2, etc to be a string. You try however to fit an object in there which cannot be parsed.

So indeed use another name, add a destructor for these types or change the default template.

[mivano]

The ES mapper warnings are fixed in v5.7