GithubHelp home page GithubHelp logo

logstash-filter-geoip's Introduction

Logstash Plugin

Travis Build Status

This is a plugin for Logstash.

It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

Documentation

Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one central location.

Need Help?

Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.

Developing

1. Plugin Developement and Testing

Code

  • To get started, you'll need JRuby with the Bundler gem installed.

  • Create a new plugin or clone and existing from the GitHub logstash-plugins organization. We also provide example plugins.

  • Install dependencies

bundle install

Test

  • Update your dependencies
bundle install
  • Pull down GeoIP database files
bundle exec rake vendor
  • Install jar dependencies
bundle exec rake install_jars
  • Run tests
bundle exec rspec

2. Running your unpublished Plugin in Logstash

2.1 Run in a local Logstash clone

  • Edit Logstash Gemfile and add the local plugin path, for example:
gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
  • Install plugin
# Logstash 2.3 and higher
bin/logstash-plugin install --no-verify

# Prior to Logstash 2.3
bin/plugin install --no-verify
  • Run Logstash with your plugin
bin/logstash -e 'filter {awesome {}}'

At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.

2.2 Run in an installed Logstash

You can use the same 2.1 method to run your plugin in an installed Logstash by editing its Gemfile and pointing the :path to your local plugin development directory or you can build the gem and install it using:

  • Build your plugin gem
gem build logstash-filter-awesome.gemspec
  • Install the plugin from the Logstash home
# Logstash 2.3 and higher
bin/logstash-plugin install --no-verify

# Prior to Logstash 2.3
bin/plugin install --no-verify
  • Start Logstash and proceed to test the plugin

Contributing

All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.

Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.

It is more important to the community that you are able to contribute.

For more information about contributing, see the CONTRIBUTING file.

logstash-filter-geoip's People

Contributors

andrewvc avatar andsel avatar clintongormley avatar colinsurprenant avatar dedemorton avatar ebuildy avatar edmocosta avatar electrical avatar fhopf avatar garyelephant avatar guyboertje avatar jakelandis avatar jhitze avatar jordansissel avatar jsvd avatar kaisecheng avatar karenzone avatar kares avatar kcm avatar kurtado avatar ph avatar robbavey avatar robin13 avatar sitwon avatar suyograo avatar talevy avatar untergeek avatar volter avatar yaauie avatar ycombinator avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

logstash-filter-geoip's Issues

Update fails - undefined method `ungz'

I'm trying to update all the gems that come with logstash 1.5 beta1 and this gem fails with the error below. How do I skip a gem when I'm trying to update all the gems? Otherwise, I'm going to have to list every single gem I want to update.

NoMethodError: undefined method `ungz' for LogStash::Util::FileTools:Module
process_downloads at /logstash/lib/logstash/util/filetools.rb:180
each at file:/logstash/vendor/jruby/lib/jruby.jar!/jruby/java/java_ext/java.util.rb:7
process_downloads at /logstash/lib/logstash/util/filetools.rb:162
setup_hook at /logstash/lib/logstash/pluginmanager/vendor.rb:13
call at org/jruby/RubyProc.java:271
run_post_install_hooks at /logstash/vendor/jruby/lib/ruby/shared/rubygems/installer.rb:273
each at org/jruby/RubyArray.java:1613
run_post_install_hooks at /logstash/vendor/jruby/lib/ruby/shared/rubygems/installer.rb:272
install at /logstash/vendor/jruby/lib/ruby/shared/rubygems/installer.rb:237
install at /logstash/vendor/jruby/lib/ruby/shared/rubygems/request_set.rb:75
each at org/jruby/RubyArray.java:1613
install at /logstash/vendor/jruby/lib/ruby/shared/rubygems/request_set.rb:62
install at /logstash/vendor/jruby/lib/ruby/shared/rubygems/dependency_installer.rb:363
update_gem at /logstash/lib/logstash/pluginmanager/update.rb:70
execute at /logstash/lib/logstash/pluginmanager/update.rb:36
each at org/jruby/RubyArray.java:1613
execute at /logstash/lib/logstash/pluginmanager/update.rb:36
run at /logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.3/lib/clamp/command.rb:67
execute at /logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.3/lib/clamp/subcommand/execution.rb:11
run at /logstash/lib/logstash/runner.rb:144
call at org/jruby/RubyProc.java:271
run at /logstash/lib/logstash/runner.rb:171
call at org/jruby/RubyProc.java:271
initialize at /logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.18/lib/stud/task.rb:12

Feature idea: calculate geo-distance in Logstash

Rationale: prevent Elasticsearch from doing recurrent calculations by pre-calculating the distance from an IP location to a fixed location (e.g., your city or your servers' location) on Logstash level.

Example of possible config:

  distance { 
       from => "geoip"
       to => [ 48.8534100, 2.3488000 ] # or can be another field's name
       formula => "Haversine"
       unit => "meters"
       target => "DistanceToParisDatacenter"    
   }

Multiple "distance" directives could be present.

It might be part of the geoip filter, or can be a standalone filter.

Possible implementation: use https://github.com/kristianmandrup/geo-distance which does not require any external dependency (SQL DB etc.).

Add support for GeoIP2-ISP database

As written in comment to geoip.rb "The path to the GeoLite2 database file which Logstash should use. Only City database is supported by now".

As for support Legacy database dropped in geoip, for me there is no way to get information from GeoIPASNum.dat (legacy) or GeoIP2-ISP.mmdb (new format) databases to get information about ASNum by IP.

Dropping support for Legacy and not adding support for new database is quite big problem for me.

Failling test on travis

https://travis-ci.org/logstash-plugins/logstash-filter-geoip/builds/121787156

Failures:

  1) LogStash::Filters::GeoIP correct encodings with ASN db "{"ip":"1.1.1.1"}" when processed
     Failure/Error: insist { subject["geoip"]["asn"].encoding } == Encoding::UTF_8
     LogStash::ConfigurationError:
       Something is wrong with your configuration.
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:134:in `config_init'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:417:in `plugin'
     # (eval):7:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:70:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:34:in `pipeline'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:143:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist.rb:47:in `value'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist/comparators.rb:13:in `=='
     # ./spec/filters/geoip_spec.rb:143:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  2) LogStash::Filters::GeoIP correct encodings with ASN db "{"ip":"189.2.0.0"}" when processed
     Failure/Error: insist { subject["geoip"]["asn"].encoding } == Encoding::UTF_8
     LogStash::ConfigurationError:
       Something is wrong with your configuration.
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:134:in `config_init'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:417:in `plugin'
     # (eval):7:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:70:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:34:in `pipeline'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:149:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist.rb:47:in `value'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist/comparators.rb:13:in `=='
     # ./spec/filters/geoip_spec.rb:149:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  3) LogStash::Filters::GeoIP correct encodings with ASN db "{"ip":"187.2.0.0"}" when processed
     Failure/Error: insist { subject["geoip"]["asn"].encoding } == Encoding::UTF_8
     LogStash::ConfigurationError:
       Something is wrong with your configuration.
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:134:in `config_init'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:417:in `plugin'
     # (eval):7:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:70:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:34:in `pipeline'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:146:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist.rb:47:in `value'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist/comparators.rb:13:in `=='
     # ./spec/filters/geoip_spec.rb:146:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  4) LogStash::Filters::GeoIP correct encodings with ASN db "{"ip":"161.24.0.0"}" when processed
     Failure/Error: insist { subject["geoip"]["asn"].encoding } == Encoding::UTF_8
     LogStash::ConfigurationError:
       Something is wrong with your configuration.
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:134:in `config_init'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:417:in `plugin'
     # (eval):7:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:70:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:34:in `pipeline'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:152:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist.rb:47:in `value'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist/comparators.rb:13:in `=='
     # ./spec/filters/geoip_spec.rb:152:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  5) LogStash::Filters::GeoIP an invalid IP should not raise an error "{"ip":"-"}" when processed
     Failure/Error: expect{ subject }.to_not raise_error
       expected no Exception, got #<LogStash::ConfigurationError: Something is wrong with your configuration.> with backtrace:
         # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:134:in `config_init'
         # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
         # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:417:in `plugin'
         # (eval):7:in `initialize'
         # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:70:in `initialize'
         # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:34:in `pipeline'
         # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
         # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
         # ./spec/filters/geoip_spec.rb:198:in `(root)'
         # ./spec/filters/geoip_spec.rb:198:in `(root)'
         # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'
     # ./spec/filters/geoip_spec.rb:198:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  6) LogStash::Filters::GeoIP an invalid IP should not raise an error "{"ip":"~"}" when processed
     Failure/Error: expect{ subject }.to_not raise_error
       expected no Exception, got #<LogStash::ConfigurationError: Something is wrong with your configuration.> with backtrace:
         # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:134:in `config_init'
         # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
         # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:417:in `plugin'
         # (eval):7:in `initialize'
         # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:70:in `initialize'
         # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:34:in `pipeline'
         # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
         # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
         # ./spec/filters/geoip_spec.rb:202:in `(root)'
         # ./spec/filters/geoip_spec.rb:202:in `(root)'
         # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'
     # ./spec/filters/geoip_spec.rb:202:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  7) LogStash::Filters::GeoIP an invalid IP should return the correct source field in the logging message "{"ip":"-"}" when processed
     Failure/Error: subject
       #<Cabin::Channel:0x5ff1c043> received :error with unexpected arguments
         expected: (anything, include {:field => "ip"})
              got: ("Invalid setting for geoip filter plugin:\n\n  filter {\n    geoip {\n      # This setting must be a path\n      # File does not exist or cannot be opened \n      database => \"\"\n      ...\n    }\n  }")
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:374:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:363:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:362:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:277:in `validate'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:133:in `config_init'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:417:in `plugin'
     # (eval):7:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:70:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:34:in `pipeline'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:241:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  8) LogStash::Filters::GeoIP an invalid IP filter method outcomes when the bad IP is N/A should set the target field to an empty hash
     Failure/Error: let(:plugin) { LogStash::Filters::GeoIP.new("source" => "message", "add_tag" => "done", "database" => ASNDB) }
     TypeError:
       can't convert nil into String
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:532:in `validate_value'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:368:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:363:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:362:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:277:in `validate'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:133:in `config_init'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
     # ./spec/filters/geoip_spec.rb:207:in `plugin'
     # ./spec/filters/geoip_spec.rb:211:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  9) LogStash::Filters::GeoIP an invalid IP filter method outcomes when the bad IP is N/A should not have added any tags
     Failure/Error: let(:plugin) { LogStash::Filters::GeoIP.new("source" => "message", "add_tag" => "done", "database" => ASNDB) }
     TypeError:
       can't convert nil into String
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:532:in `validate_value'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:368:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:363:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:362:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:277:in `validate'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:133:in `config_init'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
     # ./spec/filters/geoip_spec.rb:207:in `plugin'
     # ./spec/filters/geoip_spec.rb:211:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  10) LogStash::Filters::GeoIP an invalid IP filter method outcomes when the bad IP is two ip comma separated should set the target field to an empty hash
     Failure/Error: let(:plugin) { LogStash::Filters::GeoIP.new("source" => "message", "add_tag" => "done", "database" => ASNDB) }
     TypeError:
       can't convert nil into String
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:532:in `validate_value'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:368:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:363:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:362:in `validate_check_parameter_values'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:277:in `validate'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:133:in `config_init'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
     # ./spec/filters/geoip_spec.rb:207:in `plugin'
     # ./spec/filters/geoip_spec.rb:211:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  11) LogStash::Filters::GeoIP ASN db "{"ip":"1.1.1.1"}" when processed
     Failure/Error: insist { subject["geoip"]["asn"] } == "Google Inc."
     LogStash::ConfigurationError:
       Something is wrong with your configuration.
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:134:in `config_init'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:417:in `plugin'
     # (eval):7:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:70:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:34:in `pipeline'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:20:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist.rb:47:in `value'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist/comparators.rb:13:in `=='
     # ./spec/filters/geoip_spec.rb:20:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  12) LogStash::Filters::GeoIP ASN db "{"ip":"2a02:8071:aa1:c700:7984:22fc:c8e6:f6ff"}" when processed
     Failure/Error: reject { subject }.include?("geoip")
     LogStash::ConfigurationError:
       Something is wrong with your configuration.
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb:134:in `config_init'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:123:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:417:in `plugin'
     # (eval):7:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:70:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:34:in `pipeline'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:26:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist.rb:47:in `value'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist/enumerables.rb:11:in `include?'
     # ./spec/filters/geoip_spec.rb:26:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  13) LogStash::Filters::GeoIP correct encodings with default db "{"ip":"1.1.1.1"}" when processed
     Failure/Error: next unless subject["geoip"][f]
     TypeError:
       can't convert nil into String
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:112:in `(root)'
     # ./spec/filters/geoip_spec.rb:111:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  14) LogStash::Filters::GeoIP correct encodings with default db "{"ip":"189.2.0.0"}" when processed
     Failure/Error: next unless subject["geoip"][f]
     TypeError:
       can't convert nil into String
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:122:in `(root)'
     # ./spec/filters/geoip_spec.rb:121:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  15) LogStash::Filters::GeoIP location field when latitude field is excluded behaves like an event with a [geoip][location] field should have a location field
     Failure/Error: plugin.register
     TypeError:
       can't convert nil into String
     Shared Example Group: "an event with a [geoip][location] field" called from ./spec/filters/geoip_spec.rb:173
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./spec/filters/geoip_spec.rb:162:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  16) LogStash::Filters::GeoIP location field when both latitude and longitude field are excluded behaves like an event with a [geoip][location] field should have a location field
     Failure/Error: plugin.register
     TypeError:
       can't convert nil into String
     Shared Example Group: "an event with a [geoip][location] field" called from ./spec/filters/geoip_spec.rb:183
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./spec/filters/geoip_spec.rb:162:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  17) LogStash::Filters::GeoIP location field when longitude field is excluded behaves like an event with a [geoip][location] field should have a location field
     Failure/Error: plugin.register
     TypeError:
       can't convert nil into String
     Shared Example Group: "an event with a [geoip][location] field" called from ./spec/filters/geoip_spec.rb:178
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./spec/filters/geoip_spec.rb:162:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  18) LogStash::Filters::GeoIP normal operations when specifying the target "{"ip":"8.8.8.8"}" when processed
     Failure/Error: expect(subject).to include("src_ip")
     TypeError:
       can't convert nil into String
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:73:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  19) LogStash::Filters::GeoIP normal operations when specifying the target "{"ip":"127.0.0.1"}" when processed
     Failure/Error: expect(subject).not_to include("src_ip")
     TypeError:
       can't convert nil into String
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:86:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  20) LogStash::Filters::GeoIP normal operations when specifying add_tag "{"ip":"8.8.8.8"}" when processed
     Failure/Error: expect(subject["tags"]).to include("done")
     TypeError:
       can't convert nil into String
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:92:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  21) LogStash::Filters::GeoIP re-initializing thread current DB should initialize the DB on lookup, regardless of thread state
     Failure/Error: plugin.register
     TypeError:
       can't convert nil into String
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./spec/filters/geoip_spec.rb:292:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  22) LogStash::Filters::GeoIP defaults "{"ip":"8.8.8.8"}" when processed
     Failure/Error: insist { subject }.include?("geoip")
     TypeError:
       can't convert nil into String
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:41:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist.rb:47:in `value'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist/enumerables.rb:11:in `include?'
     # ./spec/filters/geoip_spec.rb:41:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  23) LogStash::Filters::GeoIP defaults "{"ip":"127.0.0.1"}" when processed
     Failure/Error: reject { subject }.include?("geoip")
     TypeError:
       can't convert nil into String
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:45:in `results'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-0.0.19-java/lib/logstash/devutils/rspec/logstash_helpers.rb:58:in `subject'
     # ./spec/filters/geoip_spec.rb:54:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist.rb:47:in `value'
     # ./vendor/bundle/jruby/1.9/gems/insist-1.0.0/lib/insist/enumerables.rb:11:in `include?'
     # ./spec/filters/geoip_spec.rb:54:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

  24) LogStash::Filters::GeoIP returned object identities should dup the objects
     Failure/Error: plugin.register
     TypeError:
       can't convert nil into String
     # ./lib/logstash/filters/geoip.rb:93:in `register'
     # ./spec/filters/geoip_spec.rb:271:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.8/lib/rspec/wait.rb:46:in `(root)'

Finished in 0.817 seconds (files took 1.91 seconds to load)
25 examples, 24 failures

Failed examples:

rspec ./spec/filters/geoip_spec.rb:142 # LogStash::Filters::GeoIP correct encodings with ASN db "{"ip":"1.1.1.1"}" when processed
rspec ./spec/filters/geoip_spec.rb:148 # LogStash::Filters::GeoIP correct encodings with ASN db "{"ip":"189.2.0.0"}" when processed
rspec ./spec/filters/geoip_spec.rb:145 # LogStash::Filters::GeoIP correct encodings with ASN db "{"ip":"187.2.0.0"}" when processed
rspec ./spec/filters/geoip_spec.rb:151 # LogStash::Filters::GeoIP correct encodings with ASN db "{"ip":"161.24.0.0"}" when processed
rspec ./spec/filters/geoip_spec.rb:197 # LogStash::Filters::GeoIP an invalid IP should not raise an error "{"ip":"-"}" when processed
rspec ./spec/filters/geoip_spec.rb:201 # LogStash::Filters::GeoIP an invalid IP should not raise an error "{"ip":"~"}" when processed
rspec ./spec/filters/geoip_spec.rb:239 # LogStash::Filters::GeoIP an invalid IP should return the correct source field in the logging message "{"ip":"-"}" when processed
rspec ./spec/filters/geoip_spec.rb:219 # LogStash::Filters::GeoIP an invalid IP filter method outcomes when the bad IP is N/A should set the target field to an empty hash
rspec ./spec/filters/geoip_spec.rb:223 # LogStash::Filters::GeoIP an invalid IP filter method outcomes when the bad IP is N/A should not have added any tags
rspec ./spec/filters/geoip_spec.rb:232 # LogStash::Filters::GeoIP an invalid IP filter method outcomes when the bad IP is two ip comma separated should set the target field to an empty hash
rspec ./spec/filters/geoip_spec.rb:19 # LogStash::Filters::GeoIP ASN db "{"ip":"1.1.1.1"}" when processed
rspec ./spec/filters/geoip_spec.rb:25 # LogStash::Filters::GeoIP ASN db "{"ip":"2a02:8071:aa1:c700:7984:22fc:c8e6:f6ff"}" when processed
rspec ./spec/filters/geoip_spec.rb:109 # LogStash::Filters::GeoIP correct encodings with default db "{"ip":"1.1.1.1"}" when processed
rspec ./spec/filters/geoip_spec.rb:119 # LogStash::Filters::GeoIP correct encodings with default db "{"ip":"189.2.0.0"}" when processed
rspec ./spec/filters/geoip_spec.rb:166 # LogStash::Filters::GeoIP location field when latitude field is excluded behaves like an event with a [geoip][location] field should have a location field
rspec ./spec/filters/geoip_spec.rb:166 # LogStash::Filters::GeoIP location field when both latitude and longitude field are excluded behaves like an event with a [geoip][location] field should have a location field
rspec ./spec/filters/geoip_spec.rb:166 # LogStash::Filters::GeoIP location field when longitude field is excluded behaves like an event with a [geoip][location] field should have a location field
rspec ./spec/filters/geoip_spec.rb:72 # LogStash::Filters::GeoIP normal operations when specifying the target "{"ip":"8.8.8.8"}" when processed
rspec ./spec/filters/geoip_spec.rb:84 # LogStash::Filters::GeoIP normal operations when specifying the target "{"ip":"127.0.0.1"}" when processed
rspec ./spec/filters/geoip_spec.rb:91 # LogStash::Filters::GeoIP normal operations when specifying add_tag "{"ip":"8.8.8.8"}" when processed
rspec ./spec/filters/geoip_spec.rb:295 # LogStash::Filters::GeoIP re-initializing thread current DB should initialize the DB on lookup, regardless of thread state
rspec ./spec/filters/geoip_spec.rb:40 # LogStash::Filters::GeoIP defaults "{"ip":"8.8.8.8"}" when processed
rspec ./spec/filters/geoip_spec.rb:52 # LogStash::Filters::GeoIP defaults "{"ip":"127.0.0.1"}" when processed
rspec ./spec/filters/geoip_spec.rb:274 # LogStash::Filters::GeoIP returned object identities should dup the objects

Randomized with seed 39399


travis_time:end:12f57b08:start=1460144825155326385,finish=1460144833218561802,duration=8063235417
๏ฟฝ[0K
๏ฟฝ[31;1mThe command "bundle exec rspec spec" exited with 1.๏ฟฝ[0m
travis_fold:start:cache.2
๏ฟฝ[0Kstore build cache
$ bundle clean
Resolving dependencies...........
travis_time:start:0204b9eb
๏ฟฝ[0K
travis_time:end:0204b9eb:start=1460144836320788531,finish=1460144836324151191,duration=3362660
๏ฟฝ[0Ktravis_time:start:0a611b2d
๏ฟฝ[0K๏ฟฝ[32;1mchanges detected, packing new archive๏ฟฝ[0m
๏ฟฝ[32;1muploading archive๏ฟฝ[0m

travis_time:end:0a611b2d:start=1460144836327891230,finish=1460144838118791387,duration=1790900157
๏ฟฝ[0Ktravis_fold:end:cache.2
๏ฟฝ[0K
Done. Your build exited with 1.

geoip contry_code3 value is equal to contry_code2

When doing lookup in Geoip2-City I see that returned values for country_code3 are invalid and equal to country_code2 value.

As I see now GeoIP2 database doesn't have country_code2/3 values, but just one iso_code field http://dev.maxmind.com/geoip/geoip2/javascript/

Version

/opt/logstash/bin/logstash --version
logstash 2.3.4
/opt/logstash/bin/logstash-plugin list --verbose logstash-filter-geoip
logstash-filter-geoip (3.0.1)

Operating System

lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.5 (jessie)
Release:    8.5
Codename:   jessie

Config File

geoip {
        source => "client_ip"
        target => "geoip"
        database => "/etc/geoip/GeoIP2-City.mmdb"
     }

Sample data - IP 193.29.204.7

"geoip" => {
             "city_name" => "Kiev",
        "continent_code" => "EU",
         "country_code2" => "UA",
         "country_code3" => "UA",
          "country_name" => "Ukraine",
              "dma_code" => nil,
                    "ip" => "193.29.204.7",
              "latitude" => 50.4333,
             "longitude" => 30.5167,
           "postal_code" => nil,
           "region_name" => "Kyiv City",
           "region_code" => "30",
              "timezone" => "Europe/Kiev",
              "location" => [
            [0] 30.5167,
            [1] 50.4333
        ]
    }

GeoLite2 support

(This issue was originally filed by @coder3000 at elastic/logstash#1709)


Will logstash support geolite2 in future? I am using geolite2 on server-side and there is some discrepancy caused by logstash. Specifying geolite2 db in geoip filter spits out errors.

ASN Lookup on RFC1918 IPs fails

Found that doing an GeoIP lookup on a private IP against the ASN database causes Logstash to hang, requiring a SIGKILL.

# cat /etc/centos-release
CentOS Linux release 7.1.1503 (Core)

# /opt/logstash/bin/logstash -V
logstash 1.5.0

GeoIP database is latest as of this filing
# md5sum /usr/local/share/GeoIP/GeoIPASNum.dat
d5672dde5cd81286aaaf87ab7defc690 /usr/local/share/GeoIP/GeoIPASNum.dat

Configuration (distilled down from production configuration)
https://gist.github.com/philhagen/5f6d1df7fb32a958d523

Input samples (only differences between these are the actual SRC and DST IP addresses - embedded ICMP message fields are not parsed):
Jun 8 11:23:34 fw_ids kernel: FW DENY_OUTPUT: IN= OUT=eth0 SRC=10.3.16.99 DST=10.3.59.52 LEN=265 TOS=0x00 PREC=0xC0 TTL=64 ID=41078 PROTO=ICMP TYPE=3 CODE=1 [SRC=10.3.59.52 DST=10.3.58.204 LEN=237 TOS=0x00 PREC=0x00 TTL=127 ID=180 DF PROTO=UDP SPT=52091 DPT=514 LEN=217 ]

Jun 8 11:23:34 fw_ids kernel: FW DENY_OUTPUT: IN= OUT=eth0 SRC=192.168.15.109 DST=8.8.8.8 LEN=265 TOS=0x00 PREC=0xC0 TTL=64 ID=41078 PROTO=ICMP TYPE=3 CODE=1 [SRC=10.3.59.52 DST=10.3.58.204 LEN=237 TOS=0x00 PREC=0x00 TTL=127 ID=180 DF PROTO=UDP SPT=52091 DPT=514 LEN=217 ]

Jun 8 11:23:34 fw_ids kernel: FW DENY_OUTPUT: IN= OUT=eth0 SRC=1.2.3.4 DST=172.17.51.10 LEN=265 TOS=0x00 PREC=0xC0 TTL=64 ID=41078 PROTO=ICMP TYPE=3 CODE=1 [SRC=10.3.59.52 DST=10.3.58.204 LEN=237 TOS=0x00 PREC=0x00 TTL=127 ID=180 DF PROTO=UDP SPT=52091 DPT=514 LEN=217 ]

Jun 8 11:23:34 fw_ids kernel: FW DENY_OUTPUT: IN= OUT=eth0 SRC=1.2.3.4 DST=8.8.8.8 LEN=265 TOS=0x00 PREC=0xC0 TTL=64 ID=41078 PROTO=ICMP TYPE=3 CODE=1 [SRC=10.3.59.52 DST=10.3.58.204 LEN=237 TOS=0x00 PREC=0x00 TTL=127 ID=180 DF PROTO=UDP SPT=52091 DPT=514 LEN=217 ]

Results:
Samples 1-3 result in the following:

Exception in filterworker {"exception"=>#<NoMethodError: undefined method `to_hash' for "*AS18144 Energia Communications,Inc.":String>, "backtrace"=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-0.1.9/lib/logstash/filters/geoip.rb:119:in `filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/filters/base.rb:162:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/filters/base.rb:159:in `multi_filter'", "(eval):1468:in `cond_func_65'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):1465:in `cond_func_65'", "(eval):482:in `filter_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:221:in `filterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:157:in `start_filters'"], :level=>:error}

When the filter fails, Logstash cannot be stopped with a SIGTERM - only a SIGKILL.

Sample 4 (which contains two non-RFC1918 IPs) parses fine:

{
        "message" => "Jun  8 11:23:34 fw_ids kernel: FW DENY_OUTPUT: IN= OUT=eth0 SRC=1.2.3.4 DST=8.8.8.8 LEN=265 TOS=0x00 PREC=0xC0 TTL=64 ID=41078 PROTO=ICMP TYPE=3 CODE=1 [SRC=10.3.59.52 DST=10.3.58.204 LEN=237 TOS=0x00 PREC=0x00 TTL=127 ID=180 DF PROTO=UDP SPT=52091 DPT=514 LEN=217 ]",
       "@version" => "1",
     "@timestamp" => "2015-06-12T20:40:21.165Z",
           "type" => "archive-syslog",
           "host" => "logstash2.identityvector.com",
            "ips" => [
        [0] "1.2.3.4",
        [1] "8.8.8.8"
    ],
      "ipt_label" => "Jun  8 11:23:34 fw_ids kernel: FW DENY_OUTPUT",
     "ipt_outdev" => "eth0",
      "ipt_srcip" => "1.2.3.4",
      "ipt_dstip" => "8.8.8.8",
        "ipt_ttl" => "127",
      "ipt_proto" => "UDP",
    "ipt_srcport" => "52091",
    "ipt_dstport" => "514",
           "tags" => [
        [0] "got_iptables",
        [1] "parse_done"
    ],
     "ipt_srcgeo" => {
                  "number" => "AS15169",
                     "asn" => "Google Inc.",
           "country_code2" => "US",
           "country_code3" => "USA",
            "country_name" => "United States",
          "continent_code" => "NA",
             "region_name" => "WA",
               "city_name" => "Mukilteo",
             "postal_code" => "98275",
                "latitude" => 47.91300000000001,
               "longitude" => -122.30420000000001,
                "dma_code" => 819,
               "area_code" => 425,
                "timezone" => "America/Los_Angeles",
        "real_region_name" => "Washington",
                "location" => [
            [0] -122.30420000000001,
            [1] 47.91300000000001
        ]
    },
     "ipt_dstgeo" => {
                  "number" => "AS15169",
                     "asn" => "Google Inc.",
           "country_code2" => "US",
           "country_code3" => "USA",
            "country_name" => "United States",
          "continent_code" => "NA",
             "region_name" => "CA",
               "city_name" => "Mountain View",
             "postal_code" => "94040",
                "latitude" => 37.385999999999996,
               "longitude" => -122.0838,
                "dma_code" => 807,
               "area_code" => 650,
                "timezone" => "America/Los_Angeles",
        "real_region_name" => "California",
                "location" => [
            [0] -122.0838,
            [1] 37.385999999999996
        ]
    }
}

It should also be noted that the same IP lookups work fine against the latest GeoLiteCity database - only the ASNs fail.

Exception in Logstash 1.5.0

While reloading data (which worked ok on 1.4.2), this occured
Exception in filterworker {"exception"=>#, "backtrace"=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-0.1.9/lib/logstash/filters/geoip.rb:119:in filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/filters/base.rb:162:inmulti_filter'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/filters/base.rb:159:inmulti_filter'", "(eval):569:in cond_func_7'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):561:in cond_func_7'", "(eval):412:infilter_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:219:in filterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:156:instart_filters'"], :level=>:error}

Invalid attempt to open a GeoIP2-Country database using the city method

I am getting the "Invalid attempt to open a GeoIP2-Country database using the city method" error message while using country database with logstash 5.1.2 and logstash-filter-geoip-4.0.3-java.

The filter configuration I am using:
geoip { add_tag => [ "GeoIP" ] database => "/opt/logstash/vendor/GeoCity/GeoIP2-Country.mmdb" source => "clientip" }

The database is opened without any issue, but once the logs come in, I get the following error messages and logstash stops working.

[2016-12-11T18:43:47,063][ERROR][logstash.filters.geoip   ] Unknown error while looking up GeoIP data {:exception=>java.lang.UnsupportedOperationException: Invalid attempt to open a GeoIP2-Country database using the city method, :field=>"clientip", :event=>2016-12-09T17:30:00.000Z 192.168.99.10 192.168.17.9 - - [10/Dec/2016:04:30:00 +1100] "GET /Javascript/TOL_Javascript.js HTTP/1.1" 200 7036 "-" "libwww-perl/6.02" "-" (user.toltest.om.net -> 192.168.17.31:80) 0.001 0.001 }
[2016-12-11T18:43:47,066][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>java.lang.UnsupportedOperationException: Invalid attempt to open a GeoIP2-Country database using the city method, "backtrace"=>["com.maxmind.geoip2.DatabaseReader.get(com/maxmind/geoip2/DatabaseReader.java:150)", "com.maxmind.geoip2.DatabaseReader.city(com/maxmind/geoip2/DatabaseReader.java:217)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "RUBY.filter(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.0.3-java/lib/logstash/filters/geoip.rb:160)", "RUBY.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:156)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:153)", "RUBY.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:41)", "RUBY.initialize((eval):9001)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.initialize((eval):8995)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)", "RUBY.filter_func((eval):1794)", "RUBY.filter_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:260)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)", "RUBY.each(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:186)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1342)", "RUBY.each(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:185)", "RUBY.filter_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:258)", "RUBY.worker_loop(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:225)", "java.lang.Thread.run(java/lang/Thread.java:745)"]}

Looking into the code, the codes(which set geoip_type accordingly) to detect which database edition is in use was removed in commit a0ef0d3

Geoip configuration problems

Hi All

I've problems with mi geoip ip configuration, i've kibana 4.1.1 with logstash 1.5.3 and apache 2.4.

This is my personalized geoip file configuration on logstash, i called 12-geoip.conf :

filter {
if [type] == "apache_access" {
grok {
match => { "message" => "%{COMMONAPACHELOG}" }

}
geoip {
source => "clientip"
target => "geoip.location"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip.location][coordinates]", "%{[geoip.location][longitude]}" ]
add_field => [ "[geoip.location][coordinates]", "%{[geoip.location][latitude]}" ]
}
mutate {
convert => [ "[geoip.location][coordinates]", "float"]
}
}
}

Mi apache configuration on other file
filter {
if [type] == "apache" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
add_field => [ "received_from", "%{host}" ]
}
}
}

My problem is that Kibana don't appears the "geoip" field, but clientip, hostname, etc.... appears.

Part of my logstash-forwarder configuration

{
  "paths": [
    "/var/log/apache2/*error.log",
    "/var/log/apache2/*access-ssl.log"
   ],
  "fields": { "type": "apache" }
},
{
  "paths": [
    "/var/log/apache2/*access.log"
   ],
  "fields": { "type": "apache_access" }
 }

]
}

And my logstash-forwarder registered events:

2015/09/04 09:07:57.807119 Registrar: processing 5 events
2015/09/04 09:08:45.244783 Registrar: processing 2 events
2015/09/04 09:08:50.238178 Registrar: processing 2 events
2015/09/04 09:09:02.744967 Registrar: processing 1 events

Thax so much.

Problem with legacy databases - rename the plugin to logstash-filter-geoip-city ?

Well, the problem is simple, the new version of geoip supports only GeoIP2 City database, while this is understandable while the support for other databases is developed, the problem is that if you install new version you cannot use the old legacy databases within one logstash instance. So basically the new version brakes backward compatibility.

We for example use 3 databases in our filter definition.

So for this to work, we could develop our own version of a plugin (fine for us, but I bet there are more people using this), use two versions of a plugin, one of which is renamed basically, or something can be done with the plugin to allow use of other formats simultaneously with mmdb once.

An option in config section :

config :database_type, :validate => :string, :default => 'mmdb'

accepting two values :

...
geoip {
  ...
  database_type => "dat"
  database_type => "mmdb"
  ...
}

plus some conditional code (basically copy paste from old version of a plugin) which uses this config setting.

Just a thought.

allow the inclusion/exclusion of the "extra" location field

The location field is dynamically added to the geoip structure when both latitude and longitude fields are present https://github.com/logstash-plugins/logstash-filter-geoip/blob/master/lib/logstash/filters/geoip.rb#L141-L144 to provide a GeoJSON compatible fields structure.

The fields config https://github.com/logstash-plugins/logstash-filter-geoip/blob/master/lib/logstash/filters/geoip.rb#L48 should also support including/excluding this location field to avoid unnecessary duplicate info.

Natively support IPv6 address lookups

The geoip filter works great for IPv4 lookups, but there's a couple of IPv6 addresses on the internet, and we'd like to be able to look those up, too. ๐Ÿ‘

Maxmind provides IPv6 databases, and the filter provides a database option, so while this can be done manually by dropping the IPv6 database onto a machine and adding logic to use the correct database, it would certainly be nice to at least bundle it along with v4 database in the plugin so that one could change the database setting to the IPv6 one explicitly after doing some regex catching:

if [ip] =~ /:/ {
    geoip {
        database => "GeoLiteCityv6.dat"
        ...
    }
} else {
    geoip {
        ...
    }
}

or, the filter could handle it transparently if it's feeling ambitious and detect the address type within the plugin itself and make life even better for everyone.

Multiple geoip filters causes Logstash to freeze

I've run across an issue that causes logstash to freeze while trying to use two geoip filters consecutively with two different Maxmind databases. If only one filter is used, no issue occurs. The logstash process will not respond to shutdown messages and must be killed manually.

This is on CentOS 7 x64 with Logstash 1.5.0 and no external plugins installed. I read a previous bug report that was marked as fixed for version 1.4.0, so I'm trying to determine if this is indeed a regression or if I am doing something incorrectly in my config.

Here is the offending portion of my config:

  if [prog] =~ /^filterlog$/ {  
    mutate {
      add_tag => ["firewall"]
      remove_field => [ "msg", "datetime" ]      
    }
    grok {
      patterns_dir => "/etc/logstash/patterns/*"
      match => [ "message", "%{LOG_DATA}%{IP_SPECIFIC_DATA}%{IP_DATA}%{PROTOCOL_DATA}" ]
    }
    mutate {
      lowercase => [ 'proto' ]
    }
    # do GeoIP lookup for the Country, State and City.
    geoip {
      source => "src_ip"
    }
    # do GeoIP lookup for the ASN/ISP information.
    geoip {
      database => "/etc/logstash/geoip/GeoIPASNum.dat"
      source => "src_ip"
    }
   }

Any help is greatly appreciated!

v3.0.0-beta Sometimes returns nil value for location field

Testing new version with Maxmind geoIp2 Java, for some IP adresses:

  • 113.208.89.219
  • 67.209.98.194
  • 217.15.165.185

The plugins returns a nil values for all fields, which is OK because elasticsearch will cast nil as empty string, but in the case of location field, if you setup geo mapping, ES will raise a mapping_exception error, hence document is not indexed.

I know a solution is to use https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html. But I think the plugin should not include location field if it contains nil values.

GeoIP and ISP/ASN Support Plan

Overview

  • We have had many requests to bring back ISP and ASN lookups for the GeoIP filter, which was a regression when we upgraded to using the GeoIP2 DB.
  • In the GeoIP filter v2.0.7 and prior, we shipped the plugin with the legacy GeoIP Maxmind DB which came inclusive of ISP and ASN data. These plugin versions only work for Logstash 2.x and prior.
  • In the GeoIP filter v4.0.0 and above, we implemented the new GeoIP2 DB which, although has the Country and City info, can no longer lookup ISP and ASN data. Maxmind now offers a separately licensed ISP Database for ISP and ASN. These plugin versions only work for Logstash 2.4 and above.

Plans

  • Enable ISP and ASN lookups with the current GeoIP2 implementation. Users will have the option of plugging in the GeoIP2 ISP Database (which must be purchased separately from Maxmind) to lookup ISP and ASN data beyond the current Country and City info that is available already. This is a high priority feature and can be expected in an upcoming release.
  • Support the GeoIP2 ASN Database (free)
  • Create a new plugin (logstash-filter-geoip_legacy) which will enable legacy GeoIP lookup on latest versions of Logstash. This is a lower priority and there may be a lag on when this is available based on user demand.

Target object gets over-written in v5.0

Fields which, before the event is handled by the geoip filter, were part of the object used as "target" in geoip are missing afterward. In the case below, "ip".

Config:

input { stdin { } }

filter {
  csv {
    columns => ["[source][ip]", "[target][ip]"]
     separator => ","
  }
  #-------------
  # geo lookup
  #-------------
  if [source][ip] and [source][ip] != "" {
    geoip {
      source => "[source][ip]"
      fields => [ "city_name", "region_name", "country_name", "country_code2", "location" ]
      target => "source"
    }
  }
  if [target][ip] and [target][ip] != "" {
    geoip {
      source => "[target][ip]"
      fields => [ "city_name", "region_name", "country_name", "country_code2", "location" ]
      target => "target"
    }
  }
}
output { stdout {codec => rubydebug {metadata => true}}}

Data:

173.9.34.108,173.9.34.107

Output from 2.4:

{
       "message" => "173.9.34.108,173.9.34.107",
      "@version" => "1",
    "@timestamp" => "2016-11-09T00:54:21.342Z",
          "host" => "crowley",
        "source" => {
                   "ip" => "173.9.34.108",
        "country_code2" => "US",
         "country_name" => "United States",
          "region_name" => "MA",
            "city_name" => "Newton Upper Falls",
             "location" => [
            [0] -71.2187,
            [1] 42.31270000000001
        ]
    },
        "target" => {
                   "ip" => "173.9.34.107",
        "country_code2" => "US",
         "country_name" => "United States",
          "region_name" => "MA",
            "city_name" => "Newton Upper Falls",
             "location" => [
            [0] -71.2187,
            [1] 42.31270000000001
        ]
    }
}

Output from 5.0:

{
    "@timestamp" => 2016-11-09T00:56:03.029Z,
      "@version" => "1",
          "host" => "crowley",
        "source" => {
            "city_name" => "Mendon",
         "country_name" => "United States",
        "country_code2" => "US",
          "region_name" => "Massachusetts",
             "location" => [
            [0] -71.5439,
            [1] 42.103
        ]
    },
       "message" => "173.9.34.108,173.9.34.107",
        "target" => {
            "city_name" => "Mendon",
         "country_name" => "United States",
        "country_code2" => "US",
          "region_name" => "Massachusetts",
             "location" => [
            [0] -71.5439,
            [1] 42.103
        ]
    }
}

Logstash 5.x no longer supports .dat format Geo Databases

Testing on Logstash 5.0-alpha4, noticing that .dat version DBs are no longer supported, and now the .mmdb format is required.

@suyograo This issue is meant to track updates to breaking changes & docs.

--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Invalid setting for geoip filter plugin:

  filter {
    geoip {
      # This setting must be a path
      # File does not exist or cannot be opened /tmp/GeoLiteCity.dat
      database => ["/tmp/GeoLiteCity.dat"]
      ...
    }
  } {:level=>:error}
fetched an invalid config {:config=>"input {\n\tstdin {}\n}\nfilter {\n        geoip {\n            database => [\"/tmp/GeoLiteCity.dat\"]\n\t    source => \"message\"\n        }\n}\noutput {\n \tstdout {}\n}\n\n", :reason=>"Something is wrong with your configuration.", :level=>:error}

If source is an array iterate through array

Currently the geoip filter takes either only one 1 IP or if its an array takes the first element into the array:

# The field containing the IP address or hostname to map via geoip. If

  # The field containing the IP address or hostname to map via geoip. If
  # this field is an array, only the first value will be used.
  config :source, :validate => :string, :required => true

It would be useful if it encounters an array that it iterates through the array and creates something like:

geoip[0]
geoip[1]
etc..

for each IP address. Or whatever is defined in target.

Pull Geo-IP for a country please?

Enhancement please.

I understand that this is some thing which might be very usefull for people. A business use case here. A user has a profile and a country is assigned to user profile.

When the log is logging the informaiton it takes into account the profile and logs the country name, So all i now need is a way of saying I just pass my country name i need the co-ordinates for it.

I have a full detail on Stack overflow

http://stackoverflow.com/questions/40628896/reason-something-is-wrong-with-your-configuration-geoip-dat-mutate-logstash

If i can do that that means we should be able to show the stats per country. I could pass in a 3 digit or 2 digit ISO code if needed.

geoip filter is not parsing ip_adress

Hi guys,
i'm trying to parsing ip_adress with geoip filter , my data comes in form of "172.24.12.250:19552", so before passing the geopi filter, i use the mutate filter to get off the port
mutate {
gsub=>[

        "IP_PORT","(.*\..*\..*\..*):.*","\1"
    ]   
}   

geoip{
source=> 'IP_PORT'
target=> 'geoip'
add_tag=>['geoip']
}

but after the gsub, geoip seems could not parsing the modified de ip adress.
my output is :
{
"message" => [
[0] "01_W40,4,28.09.2015 02:10:10,Authentication Process,47152093228,Accepted,MRS0400CM,0119AA,4,CMSU0219,HOSTED,0,28.09.2015 02:10:02,Local,,,6F9976CAD5CCDB48CCA679783938FBAC77872776,MAC,CM,A/MRS/T/1/TRN/1,GS,1ASIGDC04,1A/CMG/G-CM,172.25.57.244,YKUISWVHSHUO0O6COWAGSG4"
],
"@Version" => "1",
"@timestamp" => "2015-10-26T16:15:25.371Z",
"host" => "NCEVC-03441",
"path" => "/home/admin/testcuthistory_short.csv",
"P_PERIOD" => "01_W40",
"CONTEXT" => "4",
"TIME_STAMP" => "28.09.2015 02:10:10",
"FUNCTION_IDENTIFIER" => "Authentication Process",
"LINE_ID" => "47152093228",
"ACTION" => "Accepted",
"OFC" => "MRS",
"SIGN" => "0119AA",
"ORGANIZATION" => "4",
"USER_ID" => "CMSU0219",
"USER_TYlPE" => "HOSTED",
"UNSUCCESSFUL_ATTEMPTS" => "0",
"LAST_SIGNIN_DATE" => "28.09.2015 02:10:02",
"DATA_PROPERTY1" => "Local",
"DATA_PROPERTY2" => nil,
"SHORT_DETAIL" => nil,
"LOCATION_VALUE" => "6F9976CAD5CCDB48CCA679783938FBAC77872776",
"LOCATION_TYPE" => "MAC",
"APPLICATION" => "CM",
"FULL_LOCATION" => "A/MRS/T/1/TRN/1",
"DUTY_CODE" => "GS",
"TARGET_SAP" => "1ASIGDC04",
"CLAPP" => "1A/CMG/G-CM",
"IP_PORT" => "172.25.57.244",
"DCXID" => "YKUISWVHSHUO0O6COWAGSG4"
}
PS: the input file is csv file just incase you wonder.

could someone tell me the reason?
thanks

skip the lookup and tag_on_failure when ip is private

since version 3.0.0 this plugin tags a failed lookup with tag_on_failure.

However this tagging does not differentiate between a failed lookup for a private ip (which will always fail and it's OK), and a public ip (which may require further action).

we could add an option called skip_private_ips => false, which would skip the lookup for any ip in private network ranges, and not tag as a failure.

Maxmind doesnt now have GeoIP ASN DB as new format

Now, Maxmind does not have GeoIP ASN DB as a new format called GeoIPLite2.

Geoip ASN DB is still legacy format. Becouse of usage geoip legacy format, we should choose which geo ip format: GeoIP or GeoIP2.

Could you bring a new parameter that choosing geoip format?

Need more information extracted from GeoIP database not only lat or lng

We have to extract province, city information from MaxMind GeoIP2 database when using logstash filter.

But it seems that logstash default geoip plugin can just provide "latitude" and "longitude" info.
Actually we write a plugin for info extraction, but we strongly recommend official plugin maintainer can update this plugin. Can future version add more fields that can be extracted from GeoIP2 databases?

Thanks in advance and waiting for your reply.

Windows test failure

11) LogStash::Filters::GeoIP correct encodings with default db "{"ip":"1.1.1.1"}" when processed
     Failure/Error: pipeline.instance_eval { @filters.each(&:register) }
     TypeError:
       can't convert nil into String
     # C:\Users\jls\Documents\GitHub\logstash\lib\logstash\runner.rb:57:in `run'
     # C:\Users\jls\Documents\GitHub\logstash\lib\logstash\runner.rb:112:in `run'
     # C:\Users\jls\Documents\GitHub\logstash\lib\logstash\runner.rb:170:in `run'

Lots of test failures, actually.

See elastic/logstash#2487 for the full run output

Formalize the update process of the geoip2 database

@peterskim12 brought this article to our attention http://arstechnica.com/tech-policy/2016/08/kansas-couple-sues-ip-mapping-firm-for-turning-their-life-into-a-digital-hell/

We have no process in place to frequently update the database of the geop2 filter, we usually update it when we make a change to the plugin and the test fails because of a sha1 mismatch.

Even if users can easily update their IP database, we might want to have an automatic process to update it with maxmind's update.

Reopen database on errors

When the database is hosted on a NFS share it is possible for the handle to become stale and the database to need to be re-opened.

Currently this just results in an avelanche of errors

:message=>"Unknown error while looking up GeoIP data", :exception=>#<IOError: Stale NFS file handle>

At a minimum the Geoip database should be closed (to be re-opened) in case of error.

Cache has no eviction policy - when it fills up, it never changes

It seems that geoip plugin has inconsistent cache size option:

IPv6 ASN lookup throws exception: #<NoMethodError: undefined method `to_hash' for ...

(Migrated from elastic/logstash#3235; filed by @jannae)


I'm at a bit of a loss, so any help is appreciated. I'm getting a few errors that seem to have brought my ELK instance to it's knees. It stops dead on these, each time I attempt to restart or update the config in any way. Not sure what to make of it, as all my research has led me to previously corrected issues.

It seems related to this:

https://logstash.jira.com/browse/LOGSTASH-1394

Though this is indicated as having been corrected. :(

Also, if you need to review my configs they are here: https://github.com/jannae/logstash-configs

This may or may not be related to my recent upgrade to 1.5.0 rc4, it's tough to tell, except that this is a brand new problem for me, so, seems plausible.

{:timestamp=>"2015-05-12T22:11:44.302000-0400",
    :message=>"Exception in filterworker",
        "exception"=>#<NoMethodError: undefined method `to_hash' for "*AS18144 Energia Communications,Inc.":String>,
        "backtrace"=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-0.1.9/lib/logstash/filters/geoip.rb:119:in `filter'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/filters/base.rb:162:in `multi_filter'",
        "org/jruby/RubyArray.java:1613:in `each'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/filters/base.rb:159:in `multi_filter'",
        "(eval):10100:in `cond_func_383'",
        "org/jruby/RubyArray.java:1613:in `each'",
        "(eval):10094:in `cond_func_383'",
        "(eval):3085:in `filter_func'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/pipeline.rb:219:in `filterworker'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/pipeline.rb:156:in `start_filters'"],
    :level=>:error}
{:timestamp=>"2015-05-12T22:11:44.349000-0400",
    :message=>"Exception in filterworker",
        "exception"=>#<NoMethodError: undefined method `to_hash' for "*AS18144 Energia Communications,Inc.":String>,
        "backtrace"=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-0.1.9/lib/logstash/filters/geoip.rb:119:in `filter'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/filters/base.rb:162:in `multi_filter'",
        "org/jruby/RubyArray.java:1613:in `each'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/filters/base.rb:159:in `multi_filter'",
        "(eval):10100:in `cond_func_383'",
        "org/jruby/RubyArray.java:1613:in `each'",
        "(eval):10094:in `cond_func_383'",
        "(eval):3085:in `filter_func'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/pipeline.rb:219:in `filterworker'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/pipeline.rb:156:in `start_filters'"],
    :level=>:error}
{:timestamp=>"2015-05-12T22:11:44.427000-0400",
    :message=>"Exception in filterworker",
        "exception"=>#<NoMethodError: undefined method `to_hash' for "*AS18144 Energia Communications,Inc.":String>,
        "backtrace"=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-0.1.9/lib/logstash/filters/geoip.rb:119:in `filter'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/filters/base.rb:162:in `multi_filter'",
        "org/jruby/RubyArray.java:1613:in `each'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/filters/base.rb:159:in `multi_filter'",
        "(eval):10100:in `cond_func_383'",
        "org/jruby/RubyArray.java:1613:in `each'",
        "(eval):10094:in `cond_func_383'",
        "(eval):3085:in `filter_func'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/pipeline.rb:219:in `filterworker'",
        "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc4-java/lib/logstash/pipeline.rb:156:in `start_filters'"],
    :level=>:error}

(Migrated from elastic/logstash#3235; filed by @jannae)

Running from source w/o GeoLite DB yields a cryptic error

If you run this filter without using the packaged logstash distro w/o the GeoLiteCity db included you get a mysterious error:

The error reported is:
  can't convert nil into String

If you add in --debug you get the full trace:

The error reported is:
  can't convert nil into String
org/jruby/RubyFileTest.java:122:in `exist?'
/Users/andrewvc/projects/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-1.0.0/lib/logstash/filters/geoip.rb:66:in `register'
org/jruby/RubyArray.java:1613:in `each'
/Users/andrewvc/projects/logstash/lib/logstash/pipeline.rb:154:in `start_filters'
/Users/andrewvc/projects/logstash/lib/logstash/pipeline.rb:81:in `run'
/Users/andrewvc/projects/logstash/lib/logstash/agent.rb:150:in `execute'
/Users/andrewvc/projects/logstash/lib/logstash/runner.rb:91:in `run'
org/jruby/RubyProc.java:271:in `call'
/Users/andrewvc/projects/logstash/lib/logstash/runner.rb:96:in `run'
org/jruby/RubyProc.java:271:in `call'
/Users/andrewvc/projects/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/task.rb:12:in `initialize'

It seems there's a missing nil check in the code there.

Ruby exception occurred: undefined method `hex' for [\"0x00\", \"0x08\"]:Array

I don't know exactly what the issue is but it seems related to #50 where the IP field on which geoip is done is invalid. In this case, it contains 2 IP addresses separated by a comma:

{:timestamp=>"2015-11-11T20:55:49.179000-0700", :message=>"Ruby exception occurred: undefined method `hex' for [\"0x00\", \"0x08\"]:Array", :level=>:error}
{:timestamp=>"2015-11-11T20:55:49.184000-0700", :message=>"Ruby exception occurred: undefined method `hex' for [\"0x00\", \"0x20\"]:Array", :level=>:error}
{:timestamp=>"2015-11-11T20:55:49.264000-0700", :message=>"IP Field contained invalid IP address or hostname", :field=>"dstip", :event=>#<LogStash::Event:0x22fb345f @metadata_accessors=#<LogStash::Util::Accessors:0x683ea598 @store={"path"=>"/var/log/kern.log"}, @lut={"[path]"=>[{"path"=>"/var/log/kern.log"}, "path"]}>, @cancelled=false, @data={"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, @metadata={"path"=>"/var/log/kern.log"}, @accessors=#<LogStash::Util::Accessors:0x1f6d6540 @store={"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, @lut={"host"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "host"], "path"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "path"], "type"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "type"], "[type]"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "type"], "message"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "message"], "timestamp_noyear"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "timestamp_noyear"], "[tags]"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "tags"], "year"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "year"], "[iptables_TOS]"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_TOS"], "iptables_TOS"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_TOS"], "tags"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "tags"], "[iptables_PREC]"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_PREC"], "iptables_PREC"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_PREC"], "timestamp_year"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "timestamp_year"], "iptables_SRC"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_SRC"], "srcip"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "srcip"], "iptables_DST"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_DST"], "dstip"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "dstip"], "iptables_PROTO"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_PROTO"], "proto"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "proto"], "[iptables_SPT]"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_SPT"], "iptables_SPT"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_SPT"], "srcport"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "srcport"], "[iptables_DPT]"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_DPT"], "iptables_DPT"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_DPT"], "dstport"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "dstport"], "iptables_ID"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_ID"], "iptables_TTL"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "iptables_TTL"], "@timestamp"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "@timestamp"], "[srcip]"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "srcip"], "geoip_src"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "geoip_src"], "[geoip_src][longitude]"=>[{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}, "longitude"], "[geoip_src][coordinates]"=>[{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}, "coordinates"], "[geoip_src][latitude]"=>[{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}, "latitude"], "[dstip]"=>[{"message"=>"IN=eth0 OUT= MAC=00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00 SRC=202.97.52.177 DST=123.45.67.89 LEN=96 TOS=0x00 PREC=0x00 TTL=248 ID=64950 PROTO=ICMP TYPE=11 CODE=0 [SRC=123.45.67.89 DST=61.160.232.222 LEN=40 TOS=0x08 PREC=0x20 TTL=1 ID=58853 PROTO=TCP SPT=60124 DPT=7000 WINDOW=0 RES=0x00 CWR ECE SYN URGP=1280 ] ", "@version"=>"1", "@timestamp"=>"2015-11-12T03:55:49.000Z", "host"=>"siem", "path"=>"/var/log/kern.log", "type"=>"iptables-dropped", "iptables_IN"=>"eth0", "iptables_MAC"=>"00:01:23:45:67:89:00:1d:71:cc:cc:00:08:00", "iptables_SRC"=>"202.97.52.177", "iptables_DST"=>["123.45.67.89", "61.160.232.222"], "iptables_LEN"=>["96", "40"], "iptables_TOS"=>[0, 0], "iptables_PREC"=>[0, 0], "iptables_TTL"=>[248, 1], "iptables_ID"=>[64950, 58853], "iptables_PROTO"=>["ICMP", "TCP"], "iptables_TYPE"=>"11", "iptables_CODE"=>"0", "iptables_[SRC"=>"123.45.67.89", "iptables_SPT"=>60124, "iptables_DPT"=>7000, "iptables_WINDOW"=>"0", "iptables_RES"=>"0x00", "iptables_URGP"=>"1280", "tags"=>["_rubyexception"], "srcip"=>"202.97.52.177", "dstip"=>"123.45.67.89,61.160.232.222", "proto"=>["ICMP", "TCP"], "srcport"=>60124, "dstport"=>7000, "geoip_src"=>{"ip"=>"202.97.52.177", "country_code2"=>"CN", "country_code3"=>"CHN", "country_name"=>"China", "continent_code"=>"AS", "region_name"=>"22", "city_name"=>"Beijing", "latitude"=>39.9289, "longitude"=>116.38830000000002, "timezone"=>"Asia/Harbin", "real_region_name"=>"Beijing", "location"=>[116.38830000000002, 39.9289], "coordinates"=>[116.38830000000002, 39.9289]}}, "dstip"]}>>, :level=>:error}

Investigate caching the entire database into memory.

The Geoip's gem look like to have an option to preload the files into memory, lets investigate what would be the memory and performance impact of loading the entire set.

This could be used in the conjunction with the LRU cache #40 to speed up even more the IP lookups.
The compressed database is around ~20MB.

ref: #40 (comment)

Not able to lookup IPv6 in GEOIP database of maxmind

geoip {
source => "remoteip"
target => "geoip"
database => "/etc/logstash/mmcity6.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
date {
locale => "en"
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss +0530" ]
target => "@timestamp"
}
kv {
field_split => "&?"
}
}

FORWARDEDipV6 [CLIENTIPV6] HOSTNAME [25/Mar/2016:19:47:13 +0530] HIT "GET URL HTTP/1.1" 200 5 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" 0.000 -

"forwarded_ip": "ipv6",
"remoteip": "IPV6",
"loghost": "HOSTN,E",
"timestamp": "25/Mar/2016:19:47:13 +0530",
"cache": "HIT",
"httpmethod": "GET",
"request": "URL",
"httpversion": "1.1",
"response": "200",
"bytes": 5,
"agent": ""Apache-HttpClient/UNAVAILABLE (java 1.4)"",
"request_time": 0

bUT I AM NOT GETTING GEO LOCATION

Error message doesn't work right.

Version

/opt/logstash/bin/logstash --version
logstash 1.5.0

conf:

input {
  stdin {}
}

filter{ 
  geoip { source => "message" }
}

output {
  stdout { codec => rubydebug }
}

console (Send a dash "-"):

/opt/logstash/bin/logstash agent -f ~/test.conf -vvv
...{ Whole bunch of stuff I removed for brevity. }...
-
filter received {:event=>{"message"=>"-", "@version"=>"1", "@timestamp"=>"2015-06-05T01:44:49.078Z", "host"=>"vagrant-ubuntu-trusty-64"}, :level=>:debug, :file=>"(eval)", :line=>"40", :method=>"filter_func"}
IP Field contained invalid IP address or hostname {:field=>nil, :event=>#<LogStash::Event:0x794bc2a6 @metadata={}, @accessors=#<LogStash::Util::Accessors:0x5cd85e80 @store={"message"=>"-", "@version"=>"1", "@timestamp"=>"2015-06-05T01:44:49.078Z", "host"=>"vagrant-ubuntu-trusty-64"}, @lut={"host"=>[{"message"=>"-", "@version"=>"1", "@timestamp"=>"2015-06-05T01:44:49.078Z", "host"=>"vagrant-ubuntu-trusty-64"}, "host"], "message"=>[{"message"=>"-", "@version"=>"1", "@timestamp"=>"2015-06-05T01:44:49.078Z", "host"=>"vagrant-ubuntu-trusty-64"}, "message"]}>, @data={"message"=>"-", "@version"=>"1", "@timestamp"=>"2015-06-05T01:44:49.078Z", "host"=>"vagrant-ubuntu-trusty-64"}, @metadata_accessors=#<LogStash::Util::Accessors:0x5b02de9a @store={}, @lut={}>, @cancelled=false>, :level=>:error, :file=>"logstash/filters/geoip.rb", :line=>"112", :method=>"filter"}
output received {:event=>{"message"=>"-", "@version"=>"1", "@timestamp"=>"2015-06-05T01:44:49.078Z", "host"=>"vagrant-ubuntu-trusty-64"}, :level=>:debug, :file=>"(eval)", :line=>"46", :method=>"output_func"}
{
       "message" => "-",
      "@version" => "1",
    "@timestamp" => "2015-06-05T01:44:49.078Z",
          "host" => "vagrant-ubuntu-trusty-64"
}

Problem:

IP Field contained invalid IP address or hostname {:field=>nil, 

It is returning the fieldas nil. Looking at the source code it always will. I don't see anywhere where @field is instantiated. Would it make more sense to have @source be the field it is using for the error message?

I ran into this because my IIS logs would sometimes put a - into the field I am doing the GeoIP filter on. When I looked in my logs, I had no idea what was going on, since :field was nil.

how to set the "source" while the json format like this {"params":{"ip":"xxx.xxx.xxx.xxx"}}

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For all general issues, please provide the following details for fast resolution:

  • Version:
  • Operating System:
  • Config File (if you have sensitive info, please remove it):
  • Sample Data:
  • Steps to Reproduce:

Option to turn off logging for invalid IP message

What do you all think of having the "IP Field contained invalid IP address or hostname" message be able to turn off?

Like, I know it's going to happen in my IIS logs, as it will put a - field in there if the event doesn't have an IP.

The reason I ask, is because my logstash logs are at 25gb in just a week, mostly because of this log entry.

grep "IP Field contained invalid IP address or hostname" * | wc -l
375898

And that count doesn't even count the number of log lines associated with each of those entries! :(

I'm thinking that it should default to true, so that anyone trying it out for the first time would get the error message like today. Then allow it to be turned off in the config.

geoip {
  source => "ip"
  invalid_ip_logging => false
}

"fields" doesn't work as expected for "location"

What is happening:

If you don't specify "longitude" and "latitude" in "fields", "location" is not created, even if you requested it in "fields".

Why is it happening:

"location" is constructed from "longitude" and "latitude" property of the event.

Expected behaviour:

Construct "location" regardless whether "longitude" and/or "latitude" are specified in "fields".

IPv6 causes crash

Afaik there was a similar crash, but it had been fixed. I'm running the latest plugin (Logstash 1.5.2 and ./bin/plugin update logstash-filter-geoip says it's up to date).

This is the config I'm trying to use to get around not having native ipv6 support:

if [clientip] =~ /:/ {
    geoip {
        database => "/usr/share/GeoIP/GeoLiteCityv6.dat"
        source => "clientip"
    }
} else {
    geoip {
        database => "/usr/share/GeoIP/GeoLiteCity.dat"
        source => "clientip"
    }
}

And this is the error it spits out:

{:timestamp=>"2015-07-09T22:41:27.710000-0500", :message=>"The error reported is: \n  uninitialized constant LogStash::Filters::GeoIP::RuntimeException"}

If I remove the first geoip then it works fine (no ipv6 lookups).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.