Investigate caching the entire database into memory. about logstash-filter-geoip

[hp197]

This would be nice and cool to have, but I doubt it'll give you much more speed when you have multiple workers.

In my personal tests, the limiting factor with 8 workers was the mutex and not the disk IO.
But this might have improved with #40

Looking at the preload:

    if options[:preload] || !IO.respond_to?(:pread)
      @mutex = Mutex.new
    end

and on read of the database:

  def atomic_read(length, pos)
    if @mutex
      @mutex.synchronize { atomic_read_unguarded(length, pos) }
    else
      atomic_read_unguarded(length, pos)
    end
  end

  def atomic_read_unguarded(length, pos)
    if @use_pread
      IO.pread(@file.fileno, length, pos)
    else
      io = @contents || @file
      io.seek(pos)
      io.read(length)
    end
  end

(preload is pumped into @contents as StringIO).

So preloading isn't going to help much if you want to counter the mutex.
Secondly, I dont think that the database will be re-readed into memory every now end then, so if you preload, you'll need to restart logstash if the geo database is updated.

[hp197]

I have to test a bit more on this, but first runs with preloading didn't show any major improvements.

That I didnt see any major improvement might be explainable by the fact that GeoIP databases are rather small and fit into OS cache easily, so most of the time they will be cached by the OS anyway.

[jordansissel]

I wouldn't expect preloading to really improve much since the file generally lives happily within filesystem cache.

[suyograo]

We've added a LRU cache for recent lookups, so closing this.