log4net elasticsearch logging with custom parameters about log4net.elasticsearch

[jptoto]

@ayasha88 If the structure of the error document you send is the same for each message, I recommend using templates in Elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html The way this works is that you tell Elasticsearch one time how to create each index of a certain name. If your log names are called "log_yyyymmdd" then you can create a template that tells ES that for each index that is created with a name that starts with "log_*", use a certain mapping schema. The mapping will tell Elasticsearch what type to use for each field. There are some examples in the link above.

The example mapping script https://github.com/jptoto/log4net.ElasticSearch/blob/master/scripts/log-index-spec.json is a good start to create a template from. Then add the custom fields you're using to the mapping in the template, and each time a new daily index is created it will use the types specificed in the template. I think this is the best way to accomplish what you need.

[ayasha88]

But how can I then refer to these properties in c#? The same as now?

[ayasha88]

I have now the following index:

PUT _template/log
{
  "template" : "log-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
      "_all" : {"enabled" : true, "omit_norms" : true},
      "dynamic_templates" : [ {
        "message_field" : {
          "match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "string", "index" : "analyzed", "omit_norms" : true,
            "fielddata" : { "format" : "disabled" }
          }
        }
      }, {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "string", "index" : "analyzed", "omit_norms" : true,
            "fielddata" : { "format" : "disabled" },
            "fields" : {
              "raw" : {"type": "string", "index" : "not_analyzed", "doc_values" : true, "ignore_above" : 256}
            }
          }
        }
      }, {
        "float_fields" : {
          "match" : "*",
          "match_mapping_type" : "float",
          "mapping" : { "type" : "float", "doc_values" : true }
        }
      }, {
        "double_fields" : {
          "match" : "*",
          "match_mapping_type" : "double",
          "mapping" : { "type" : "double", "doc_values" : true }
        }
      }, {
        "byte_fields" : {
          "match" : "*",
          "match_mapping_type" : "byte",
          "mapping" : { "type" : "byte", "doc_values" : true }
        }
      }, {
        "short_fields" : {
          "match" : "*",
          "match_mapping_type" : "short",
          "mapping" : { "type" : "short", "doc_values" : true }
        }
      }, {
        "integer_fields" : {
          "match" : "*",
          "match_mapping_type" : "integer",
          "mapping" : { "type" : "integer", "doc_values" : true }
        }
      }, {
        "long_fields" : {
          "match" : "*",
          "match_mapping_type" : "long",
          "mapping" : { "type" : "long", "doc_values" : true }
        }
      }, {
        "date_fields" : {
          "match" : "*",
          "match_mapping_type" : "date",
          "mapping" : { "type" : "date", "doc_values" : true }
        }
      }, {
        "geo_point_fields" : {
          "match" : "*",
          "match_mapping_type" : "geo_point",
          "mapping" : { "type" : "geo_point", "doc_values" : true }
        }
      } ],
      "properties" : {
        "@timestamp": { "type": "date", "doc_values" : true },
        "@version": { "type": "string", "index": "not_analyzed", "doc_values" : true },
        "geoip"  : {
          "type" : "object",
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip", "doc_values" : true },
            "location" : { "type" : "geo_point", "doc_values" : true },
            "latitude" : { "type" : "float", "doc_values" : true },
            "longitude" : { "type" : "float", "doc_values" : true }
          }
        }
      }
    }
  }
}

i don't know how to access to the property geoip from c# for example..

[jptoto]

@ayasha88 Are you using Nest In your C# code? That makes accessing the fields a lot easier in .NET.

Also I think the example template you're using is much too complicated. You can create one that specifies the fields directly based on the structure of your document. Here is an example template and I added a "location" field of type geo_point below the timeStamp field.

If you're unsure how the structure of your dynamic document looks in ES, just do a GET on one to see how the structure ended up looking and make your template based on that.

{   
    "template" : "log-*",
    "settings": {
        "index": {
            "number_of_shards": 1,
            "number_of_replicas": 0
        }
    },
    "mappings": {
        "LogEvent": {
            "properties": {
                "timeStamp": {
                    "type": "date",
                    "format": "dateOptionalTime"
                },
                "location" : {
                    "type": "geo_point"
                },
                "message": {
                    "type": "string"
                },
                "messageObject": {
                    "type": "object"
                },
                "exception": {
                    "type": "object"
                },
                "loggerName": {
                    "type": "string"
                },
                "domain": {
                    "type": "string"
                },
                "identity": {
                    "type": "string"
                },
                "level": {
                    "type": "string"
                },
                "className": {
                    "type": "string"
                },
                "fileName": {
                    "type": "string"
                },
                "name": {
                    "type": "string"
                },
                "fullInfo": {
                    "type": "string"
                },
                "methodName": {
                    "type": "string"
                },
                "fix": {
                    "type": "string"
                },
                "properties": {
                    "type": "string"
                },
                "userName": {
                    "type": "string"
                },
                "threadName": {
                    "type": "string"
                },
                "hostName": {
                    "type": "string"
                }
            }
        }
    }
}

[ayasha88]

No I am not using Nest.. What is that?
I created a similar index to yours:

{   
    "template" : "psm-*",
    "settings": {
        "index": {
            "number_of_shards": 1,
            "number_of_replicas": 0
        }
    },
    "mappings": {
        "LogEvent": {
            "dynamic_templates": [
                {
                  "string_fields": {
                     "mapping": {
                        "fielddata": {
                           "format": "disabled"
                        },
                        "index": "analyzed",
                        "omit_norms": true,
                        "type": "string",
                        "fields": {
                           "raw": {
                              "ignore_above": 256,
                              "index": "not_analyzed",
                              "type": "string",
                              "doc_values": true
                           }
                        }
                     },
                     "match": "*",
                     "match_mapping_type": "string"
                  }
               }
            ],
            "properties": {
                "properties": {
                "timeStamp": {
                    "type": "date",
                    "format": "dateOptionalTime"
                },
                "location" : {
                    "type": "geo_point"
                },
                "message": {
                    "type": "string"
                },
                "messageObject": {
                    "type": "object"
                },
                "exception": {
                    "type": "object"
                },
                "loggerName": {
                    "type": "string"
                },
                "domain": {
                    "type": "string"
                },
                "identity": {
                    "type": "string"
                },
                "level": {
                    "type": "string"
                },
                "className": {
                    "type": "string"
                },
                "fileName": {
                    "type": "string"
                },
                "name": {
                    "type": "string"
                },
                "fullInfo": {
                    "type": "string"
                },
                "methodName": {
                    "type": "string"
                },
                "fix": {
                    "type": "string"
                },
                "properties": {
                    "type": "string"
                },
                "userName": {
                    "type": "string"
                },
                "threadName": {
                    "type": "string"
                },
                "hostName": {
                    "type": "string"
                }
            }
        }
    }
}
}

Because otherwise it puts all the fields inside the field properties instead I want to have each field separate. But in this way it doens't write any log.. Also I need the the string fields have a raw that is not_analyzed. But in this way nothing is logged..

[jptoto]

@ayasha88 In this case, I think the best thing to do would be to create a custom exception class based on the Exception or ApplicationException base classes. This way you can add in any and all properties you want and they will get passed in as the messageObject when you include that in your exception. If you're not using a strict template or index mapping in ES, it will allow you to take whatever json the log4net.ES lib throws at it.

Here's an example of how to make a custom ApplicationException class https://gist.github.com/jptoto/9be57ce2fe253eb96739

Then, you can see in the _log.Error call I include the basic message and the CustomException object (name that whatever you want). That gets passed in as the "messageObject" and you will see the results in your ES doc (see attached).

In my opinion a custom application exception is the most idiomatic/elegant solution for this case and I think it will suit your needs. Good luck!