GithubHelp home page GithubHelp logo

Comments (3)

TotalGriffLock avatar TotalGriffLock commented on May 26, 2024

Here's the dynamic template generated for this index (and therefore this plugin's messages because nothing else logs to that input)

$ curl -X GET "localhost:9200/_template/gelf-template?pretty=true"

{
  "gelf-template" : {
    "order" : -1,
    "index_patterns" : [
      "gelf_*"
    ],
    "settings" : {
      "index" : {
        "analysis" : {
          "analyzer" : {
            "analyzer_keyword" : {
              "filter" : "lowercase",
              "tokenizer" : "keyword"
            }
          }
        }
      }
    },
    "mappings" : {
      "_source" : {
        "enabled" : true
      },
      "dynamic_templates" : [
        {
          "internal_fields" : {
            "mapping" : {
              "type" : "keyword"
            },
            "match_mapping_type" : "string",
            "match" : "gl2_*"
          }
        },
        {
          "store_generic" : {
            "mapping" : {
              "type" : "keyword"
            },
            "match_mapping_type" : "string"
          }
        }
      ],
      "properties" : {
        "gl2_processing_timestamp" : {
          "format" : "uuuu-MM-dd HH:mm:ss.SSS",
          "type" : "date"
        },
        "gl2_accounted_message_size" : {
          "type" : "long"
        },
        "gl2_receive_timestamp" : {
          "format" : "uuuu-MM-dd HH:mm:ss.SSS",
          "type" : "date"
        },
        "full_message" : {
          "fielddata" : false,
          "analyzer" : "standard",
          "type" : "text"
        },
        "streams" : {
          "type" : "keyword"
        },
        "source" : {
          "fielddata" : true,
          "analyzer" : "analyzer_keyword",
          "type" : "text"
        },
        "message" : {
          "fielddata" : false,
          "analyzer" : "standard",
          "type" : "text"
        },
        "timestamp" : {
          "format" : "uuuu-MM-dd HH:mm:ss.SSS",
          "type" : "date"
        }
      }
    },
    "aliases" : { }
  }
}

The only field which is a [long] is gl2_accounted_message_size. So is this plugin causing that field to sometimes contain the timestamp or a null value?

from graylog-plugin-metrics-reporter.

TotalGriffLock avatar TotalGriffLock commented on May 26, 2024

I have resolved this myself, via https://community.graylog.org/t/graylog-metrics-plugin-feeding-data-via-gelf-to-graylog-causing-parsing-errors/16356/3

Most of the values for metrics are numbers so Graylog/ES correctly decide to store the "value" field as a [long]. However there are 2 metrics (at the time of writing):
org.graylog2.journal.oldest-segment
jvm.threads.deadlocks
where the value is either a string (timestamp) or a collection/array. Obviously this data will not go in a field with the type of long. The graylog community URL above provides a solution but only for 1 specific metric. I've put the GELF metrics input through a pipeline with the following rule, which has resolved the errors for me and should work as new metrics are added which are not numeric:

Rule "Cleanup: Non-numeric metrics value field"
when
has_field("value") AND
not is_long("value")
then
rename_field(
old_field: "value",
new_field: "value_string"
);
end

from graylog-plugin-metrics-reporter.

TotalGriffLock avatar TotalGriffLock commented on May 26, 2024

That didn't appear to be working either, but this does. Can't spend any more time on it right now, but if anyone else is having the same problem this will fix it.

Rule "Cleanup: Non-numeric metrics value field"
when
has_field("name") AND
has_field("value") AND
(to_string($message.name) == "org.graylog2.journal.oldest-segment" OR
to_string($message.name) == "jvm.threads.deadlocks")
then
let value_string = to_string($message.value);
set_field ("value_string",value_string);
remove_field("value");
end

from graylog-plugin-metrics-reporter.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.