jflatow / ego

Erlang (data-based) geo-services

~jflatow

To bootstrap on a machine w/ my Dropbox:

cd ~/Dropbox/Code/stencil/users/jflatow

Otherwise, if not using Dropbox:

cd /usr/local/src/etc/users/jflatow

Then:

git clone [email protected]:jflatow/jflatow.git
cd jflatow
./install

To make a change to emacs prefs:

emacs .emacs
./install

Notes to future self

On the occasion of my next reboot, there should be no ambiguity about how to recover my session.

All my instructions to myself for bootstrapping packages should live in this repo. Notes for emacs are in .emacs itself.

TODO: macbook from scratch: /usr/local/{src,etc} /usr/local/src/lang/{C,JavaScript,Python,Plato,...} /usr/local/src/scratch/{emacs,web,...} latest emacs, Emacs.app Python3 matplotlib ipython + Jupyter + notebeook setup? Chrome canary + React dev tools?

ego's Issues

offset about data section value

hi jflatow:
the code in maxmind.erl line 70-71 about offset when recordval > nodecount:

 node(#db{node_count=N, tree_size=T} = G, K) when K > N ->
 {stop, value(G, K - N + T)};

here T is tree_size but not datas_off, why?
the maxmind (http://maxmind.github.io/MaxMind-DB/) doc about the offset is:

$offset_in_file = ( $record_value - $node_count ) + $search_tree_size_in_bytes + 16

Crashes with some ips

Some ips that crash are:
"5.50.20.5"
"190.100.190.50"
"5.110.190.20"
"50.55.190.31"
"190.110.110.31"
"5.110.110.31"
"190.100.110.31"
"31.25.110.31"
"190.20.110.31"

For example:

maxminddb:find(maxminddb:open(), "5.220.250.250").
** exception error: no function clause matching maxminddb:decode({db,<<0,0,1,0,43,224,195,0,0,2,0,43,224,195,0,0,3,0,42,89,229,0,0,4,0,43,...>>,
[{<<"record_size">>,28},
{<<"node_count">>,2875587},
{<<"languages">>,
[<<"de">>,<<"en">>,<<"es">>,<<"fr">>,<<"ja">>,<<"pt-BR">>,
<<"ru">>,<<"zh-CN">>]},
{<<"ip_version">>,6},
{<<"description">>,
[{<<"en">>,<<"GeoLite2 City database">>}]},
{<<"database_type">>,<<"GeoLite2-City">>},
{<<"build_epoch">>,1386635716},
{<<"binary_format_minor_version">>,0}, {<<"binary_format_major_version">>,2}], 6,2875587,28,4,24,20129109,20129125}, <<100,111,32,73,114,195,163,66,114,117,72,208,152,209, 128,208,176,208,189,32,134,88,228,188,138,230,156,...>>)
(src/maxminddb.erl, line 76)
in function maxminddb:decode/3 (src/maxminddb.erl, line 149)
in call from maxminddb:decode/3 (src/maxminddb.erl, line 150)
in call from maxminddb:value/2 (src/maxminddb.erl, line 165)
in call from maxminddb:node/2 (src/maxminddb.erl, line 71)
in call from maxminddb:find/3 (src/maxminddb.erl, line 66)

Hang in maxminddb:find/2

I've discovered that the following code hangs:

$ erl -pa ebin 
Erlang/OTP 17 [erts-6.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Eshell V6.0  (abort with ^G)
1> DB = maxminddb:open("/etc/geoip/GeoLite2-City.mmdb").
{db,<<0,0,1,0,44,159,160,0,0,2,0,44,159,160,0,0,3,0,42,
      254,108,0,0,4,0,44,159,...>>,
    [{<<"record_size">>,28},
     {<<"node_count">>,2924448},
     {<<"languages">>,
      [<<"de">>,<<"en">>,<<"es">>,<<"fr">>,<<"ja">>,<<"pt-BR">>,
       <<"ru">>,<<"zh-CN">>]},
     {<<"ip_version">>,6},
     {<<"description">>,
      [{<<"en">>,<<"GeoLite2 City database">>}]},
     {<<"database_type">>,<<"GeoLite2-City">>},
     {<<"build_epoch">>,1401912337},
     {<<"binary_format_minor_version">>,0},
     {<<"binary_format_major_version">>,2}],
    6,2924448,28,4,24,20471136,20471152}
2> maxminddb:find(DB, "178.171.148.35").

The call to maxminddb:find/2 never returns and Erlang pegs my CPU until I kill it.

The database is Maxmind's free "GeoLite2 City" database, available here.

So far, I've only encountered the problem while looking up "178.171.148.35".

This is a MaxMind DB reader, it's not GeoIP2 specific

In our official APIs, we've made an effort to separate the MaxMind DB implementation from GeoIP2. The former is totally agnostic as to what's in the database, whereas the latter expects a certain data structure for each IP address.

We hope that others might adopt this database format for their own use, so it makes sense to separate these.

Your code is really just implementing the MaxMind DB format. It'd be great to also build a separate GeoIP2 package on top of this, but ideally they'd be two separate things.

If you look at our public repos (https://github.com/maxmind) you'll see many examples of this. For PHP, Perl, Python, and Java, we've implemented both a MaxMind-DB-Reader-$lang library and a separate GeoIP2-$lang library.

I think in your case you might just need to rename geoip.erl to maxminddb.erl or something like that. If you do go ahead and implement something GeoIP2 specific, please include the "2" in the name. Otherwise people might get confused and think your code handles the Legacy GeoIP database format, but it won't unless you code that separately.