Some log sources provide host information in a mixed format. Sometimes as IP address and sometimes as hostname/FQDN. This value is copied as reported into the field source.address. Then this value is copied into the fields source.ip and source.domain and the dns filter plugin should do a reverse lookup on the source.domain field and a normal resolve on source.ip. Hence if source.domain would contain an IP address, it will be replaced with the appropriate hostname and source.ip would be left unchanged; and vice versa.
I confirmed with tcpdump, that not a single DNS query is sent to the DNS server. If a hostname/fqdn is specified instead of an ip address, the dns-filter works as expected and replaced the hostname with an ip address in one field an leaves the other one unchanged.
If the hit cache is disabled, the dns-filter replaces the ip address with the hostname in one field, and leaves the other one unchanged.
input {
generator {
ecs_compatibility => "v8"
count => 1
lines => [
"8.8.8.8"
]}
}
filter {
mutate {
add_field => {
"[source][domain]" => "%{message}"
"[source][ip]" => "%{message}"
}
}
dns {
action => "replace"
resolve => "[source][domain]"
reverse => "[source][ip]"
hit_cache_ttl => 60
hit_cache_size => 10
}
}
output {
stdout { codec => rubydebug }
}
# logstash --log.level debug
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2023-04-22T22:50:48,249][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2023-04-22T22:50:48,258][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.17.9", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.18+10 on 11.0.18+10 +indy +jit [linux-x86_64]"}
[2023-04-22T22:50:48,260][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/]
[2023-04-22T22:50:49,495][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-04-22T22:50:50,545][INFO ][org.reflections.Reflections] Reflections took 52 ms to scan 1 urls, producing 119 keys and 419 values
[2023-04-22T22:50:51,232][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-04-22T22:50:51,257][WARN ][deprecation.logstash.inputs.generator] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-04-22T22:50:51,355][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"dns", :type=>"filter", :class=>LogStash::Filters::DNS}
[2023-04-22T22:50:51,365][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@action = "replace"
[2023-04-22T22:50:51,365][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@hit_cache_size = 10000
[2023-04-22T22:50:51,365][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@id = "32e1cf6783f1e24aad525c2a1db225c8fd292767b1c72c16afd9b51cc43f72c4"
[2023-04-22T22:50:51,365][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@resolve = ["[source][domain]"]
[2023-04-22T22:50:51,365][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@reverse = ["[source][ip]"]
[2023-04-22T22:50:51,365][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@hit_cache_ttl = 60
[2023-04-22T22:50:51,365][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@enable_metric = true
[2023-04-22T22:50:51,365][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@add_tag = []
[2023-04-22T22:50:51,365][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@remove_tag = []
[2023-04-22T22:50:51,365][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@add_field = {}
[2023-04-22T22:50:51,365][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@remove_field = []
[2023-04-22T22:50:51,366][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@periodic_flush = false
[2023-04-22T22:50:51,366][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@timeout = 0.5
[2023-04-22T22:50:51,366][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@max_retries = 2
[2023-04-22T22:50:51,366][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@failed_cache_size = 0
[2023-04-22T22:50:51,366][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@failed_cache_ttl = 5
[2023-04-22T22:50:51,366][DEBUG][logstash.filters.dns ] config LogStash::Filters::DNS/@tag_on_timeout = ["_dnstimeout"]
[2023-04-22T22:50:51,534][DEBUG][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main"}
[2023-04-22T22:50:51,623][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/usr/share/logstash/pipeline/mypipeline.conf"], :thread=>"#<Thread:0x2486058b run>"}
[2023-04-22T22:50:52,261][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.63}
[2023-04-22T22:50:52,316][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-04-22T22:50:52,440][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-04-22T22:50:52,557][DEBUG][logstash.javapipeline ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#<LogStash::WorkerLoopThread:0x7b81e378 run>"}
{
"source" => {
"ip" => "8.8.8.8",
"domain" => "8.8.8.8"
},
"message" => "8.8.8.8",
"sequence" => 0,
"@version" => "1",
"@timestamp" => 2023-04-22T20:50:52.420Z,
"host" => "3493941a8650"
}
[2023-04-22T22:50:52,740][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-04-22T22:50:52,986][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2023-04-22T22:50:53,023][INFO ][logstash.runner ] Logstash shut down.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent