sameei / elasticreporter.old Goto Github PK

Cosider: 'Infinity' It's a numeric value in most programing languages but ES is sensitive to parse this values,
for example in a case of sending a Json containing this: "KafkaConsumer.records-lag-max": -Infinity to ES,
it responds with:

{
    "error":{"root_cause":[{"type":"mapper_parsin$_exception","reason":"failed to parse"}],
    "type":"mapper_parsing_exception",
    "reason":"failed to parse","caused_by":{
        "type":"json_pa$se_exception",
        "reason":"Non-standard token '-Infinity': enable JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS to allow\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@721e357a; line: 1, column: 2042]"
    }},"status":400
}

MetricGroups have a set of variables! define a Select base on Variable Availability and Inject variables in report output.
By this you should be able to report metrics per job-names

ES error: Limit of total fields [1000] in index [stageflinkd1-jobmanager-2018-6] has been exceeded

• Configurable name for loggers
• Loggers for FilterBy, GroupBy, and NameBy
• Clear and Document Loggers status and output!

It's good to report all available variables in to elasticsearch: , <task_id>, <operator_id>, , ....

....filter.drop-left-to-scope: developerdefined
host.jobmanger.taskid.operatorid......prod-apps.nathan.forward.log.direct
host.jobmanger.taskid.operatorid......prod-apps.nathan.forward.log.indirect
host.jobmanger.taskid.operatorid......prod-apps.nathan.forward.log.duplicate
host.jobmanger.taskid.operatorid......prod-apps.nathan.forward.log.private
host.jobmanger.taskid.operatorid......prod-apps.nathan.forward.uptime.fedora

.jobmanager.<task_id>.<job_name>.....developerdefined

flink.reporter.Groups
index: "<job_name>---<day_of_month>"
doc: "<operator_id>-<subtask_index>-"
group-by: "<job_name>.<operator_name>.<subtask_index>"
filter-by.required-vars: "<job_name>,<operator_id>,<operator_name>,<subtask_id>"
filter-by.scope: ".jobmanager"

flink.reporter.SingleDoc
index: "<job_name>---<day_of_month>"
doc: "<operator_id>-<subtask_index>-"
group-by: "<job_name>.<operator_name>.<subtask_index>"
filter-by.required-vars: "<job_name>,<operator_id>,<operator_name>,<subtask_id>"
filter-by.scope: ".jobmanager"