maxmind / maxmind-db-reader-xs

Fast XS implementation of MaxMind DB reader

NAME

MaxMind::DB::Reader::XS - Fast XS implementation of MaxMind DB reader

VERSION

version 1.000009

SYNOPSIS

my $reader = MaxMind::DB::Reader->new( file => 'path/to/database.mmdb' );

my $record = $reader->record_for_address('1.2.3.4');

DESCRIPTION

Simply installing this module causes MaxMind::DB::Reader to use the XS implementation, which is much faster than the Perl implementation.

The XS implementation links against the libmaxminddb library.

See MaxMind::DB::Reader for API details.

VERSIONING POLICY

This module uses semantic versioning as described by http://semver.org/. Version numbers can be read as X.YYYZZZ, where X is the major number, YYY is the minor number, and ZZZ is the patch number.

MAC OS X SUPPORT

If you're running into install errors under Mac OS X, you may need to force a build of the 64 bit binary. For example, if you're installing via cpanm:

ARCHFLAGS="-arch x86_64" cpanm MaxMind::DB::Reader::XS

UBUNTU SUPPORT

The version of libmaxminddb that is available by default with Ubuntu may be too old for this level of MaxMind::DB::Reader::XS. However, we do maintain a Launchpad PPA for all supported levels of Ubuntu.

https://launchpad.net/~maxmind/+archive/ubuntu/ppa

Please visit the PPA page for more information, or, to configure your system, run as root:

# apt-add-repository ppa:maxmind/ppa
# apt-get update

The PPA is now configured, and you may install (or upgrade) the libmaxminddb library via the usual apt commands.

SUPPORT

This module is deprecated and will only receive fixes for major bugs and security vulnerabilities. New features and functionality will not be added.

Please report all issues with this code using the GitHub issue tracker at https://github.com/maxmind/MaxMind-DB-Reader-XS/issues.

Bugs may be submitted through https://github.com/maxmind/MaxMind-DB-Reader-XS/issues.

AUTHORS

• Boris Zentner [email protected]
• Dave Rolsky [email protected]
• Ran Eilam [email protected]

CONTRIBUTORS

• Andy Jack [email protected]
• Chris Weyl [email protected]
• Florian Ragwitz [email protected]
• Greg Oschwald [email protected]
• Hidenori Sugiyama [email protected]
• Mark Fowler [email protected]
• Nick Logan [email protected]
• Olaf Alders [email protected]

COPYRIGHT AND LICENSE

This software is Copyright (c) 2013 - 2022 by MaxMind, Inc.

This is free software, licensed under:

The Artistic License 2.0 (GPL Compatible)

maxmind-db-reader-xs's Issues

Failed test '$reader->iterate_search_tree()

#   Failed test '$reader->iterate_search_tree() finds all the networks in the database'
#   at t/MaxMind/DB/Reader.t line 119.
#     Structures begin differing at:
#          $got->[5] = '::1.1.1.32/128'
#     $expected->[5] = '::1.1.1.63/128'
# Looks like you failed 1 test of 317.

MaxMind-DB-Reader-XS-0.060003

problem installing on macOS Mojave

Installing MaxMind::DB::Reader::XS from cpan on Mojave
halts with the message that maxminddb_config.h is not provided.
It's right there next to maxminddb.h.

Wrong version detection in Reader-broken-databases.t

I encountered the following error during $ cpanm MaxMind::DB::Reader::XS.

t/00-report-prereqs.t ...................... ok

#   Failed test 'expected exception with unknown file type'
#   at t/MaxMind/DB/Reader-broken-databases.t line 84.
#                   'MaxMind::DB::Reader::XS - Error opening database file "/tmp/9AJiT3aWv9/garbage": The MaxMind DB file contains invalid metadata at /root/.cpanm/work/1519046793.22989/MaxMind-DB-Reader-XS-1.000004/blib/lib/MaxMind/DB/Reader/XS.pm line 67.
# '
#     doesn't match '(?^:Error opening database file "\/tmp\/9AJiT3aWv9\/garbage": The lookup path does not match the data .+)'
# Looks like you failed 1 test of 5.
t/MaxMind/DB/Reader-broken-databases.t .....
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/5 subtests
t/MaxMind/DB/Reader-decoder.t .............. ok
t/MaxMind/DB/Reader-no-ipv4-search-tree.t .. ok
t/MaxMind/DB/Reader.t ...................... ok
t/MaxMind/DB/Reader/NoMoose.t .............. ok
# libmaxminddb version is 1.3.2
t/libmaxminddb-version.t ................... ok

My libmaxminddb version is 1.3.2 (latest release version).

Reader-broken-databases.t uses the patch level to distinguish whether it is an older version or not.
Version 1.3.2 is probably mistaken as an older version.

MaxMind-DB-Reader-XS/t/MaxMind/DB/Reader-broken-databases.t

Lines 70 to 77 in 527804f

	if ( $minor >= 1 && $patch >= 3 ) {
	$expect
	= qr/Error opening database file "\Q$file\E": The MaxMind DB file contains invalid metadata .+/;
	}
	elsif ( $minor >= 1 && $patch >= 2 ) {
	$expect
	= qr/Error opening database file "\Q$file\E": The lookup path does not match the data .+/;
	}

tests fail becasue the XS.so module can not be loaded

After building MaxMind-DB-Reader-XS I run the Build test but get errors.

# === Configure Requires ===
# 
#     Module        Want   Have
#     ------------- ---- ------
#     Module::Build 0.28 0.4224
# 
# === Build Requires ===
# 
#     Module             Want   Have
#     ------------------ ---- ------
#     ExtUtils::CBuilder  any   0.27
#     Module::Build      0.28 0.4224
# 
# === Test Requires ===
# 
#     Module                          Want     Have
#     ------------------------------- ---- --------
#     ExtUtils::MakeMaker              any     7.30
#     File::Spec                       any     3.30
#     Module::Implementation           any     0.09
#     Net::Works::Network             0.21     0.22
#     Path::Class                     0.27     0.37
#     Test::Fatal                      any    0.008
#     Test::MaxMind::DB::Common::Util  any 0.040001
#     Test::More                      0.96 1.302086
#     Test::Number::Delta              any     1.06
#     Test::Requires                   any     0.10
#     autodie                          any  2.06_01
#     lib                              any     0.62
#     utf8                             any     1.07
# 
# === Test Recommends ===
# 
#     Module         Want     Have
#     ---------- -------- --------
#     CPAN::Meta 2.120900 2.150010
# 
# === Runtime Requires ===
# 
#     Module                                     Want     Have
#     -------------------------------------- -------- --------
#     Math::Int128                                any     0.22
#     MaxMind::DB::Metadata                  0.040001 0.040001
#     MaxMind::DB::Reader                    1.000012 1.000013
#     MaxMind::DB::Reader::Role::HasMetadata      any 1.000013
#     MaxMind::DB::Reader::Role::Reader           any 1.000013
#     MaxMind::DB::Types                          any 0.040001
#     Moo                                         any 2.003002
#     XSLoader                                    any     0.10
#     namespace::autoclean                        any     0.28
#     strict                                      any     1.04
#     warnings                                    any     1.06

t/00-report-prereqs.t ...................... ok
t/MaxMind/DB/Reader-broken-databases.t ..... Can't load '/usr/local/uvredhat/BUILD/MaxMind-DB-Reader-XS-1.000004/blib/arch/auto/MaxMind/DB/Reader/XS/XS.so' for module MaxMind::DB::Reader::XS: /usr/local/uvredhat/BUILD/MaxMind-DB-Reader-XS-1.000004/blib/arch/auto/MaxMind/DB/Reader/XS/XS.so: undefined symbol: DPPP_my_sv_setpvf_mg_nocontext at /usr/lib64/perl5/DynaLoader.pm line 200.
at t/lib/Test/MaxMind/DB/Reader.pm line 6.

XS reader is broken with the latest libmaxminddb

The XS reader calculates and sets offsets that it passes to libmaxminddb. This depends on private behavior of libmaxminddb and it broke with the latest release.

See maxmind/libmaxminddb#110 for the description of the change we want to make to libmaxminddb

A/C:

• Use new libmaxminddb MMDB_read_node functionality
• Will only install if libmaxminddb supports new functionality

error message saying that version 1.56 is less than 1.5.

[thvv ~]$ perl --version

This is perl 5, version 30, subversion 3 (v5.30.3) built for darwin-thread-multi-2level
[snip]
[thvv ~]$ ipcity
Could not find a suitable MaxMind::DB::Reader implementation: List::Util version 1.56 required--this is only version 1.5 at /opt/local/lib/perl5/site_perl/5.30/List/AllUtils.pm line 8.
BEGIN failed--compilation aborted at /opt/local/lib/perl5/site_perl/5.30/List/AllUtils.pm line 8.
Compilation failed in require at /opt/local/lib/perl5/site_perl/5.30/MaxMind/DB/Types.pm line 10.
BEGIN failed--compilation aborted at /opt/local/lib/perl5/site_perl/5.30/MaxMind/DB/Types.pm line 10.
Compilation failed in require at /opt/local/lib/perl5/site_perl/5.30/MaxMind/DB/Metadata.pm line 10.
BEGIN failed--compilation aborted at /opt/local/lib/perl5/site_perl/5.30/MaxMind/DB/Metadata.pm line 10.
Compilation failed in require at /opt/local/lib/perl5/site_perl/5.30/darwin-thread-multi-2level/MaxMind/DB/Reader/XS.pm line 15.
BEGIN failed--compilation aborted at /opt/local/lib/perl5/site_perl/5.30/darwin-thread-multi-2level/MaxMind/DB/Reader/XS.pm line 15.
Compilation failed in require at /opt/local/lib/perl5/site_perl/5.30/Module/Runtime.pm line 314.
Attempt to reload MaxMind/DB/Types.pm aborted.
Compilation failed in require at /opt/local/lib/perl5/site_perl/5.30/MaxMind/DB/Reader/PP.pm line 12.
BEGIN failed--compilation aborted at /opt/local/lib/perl5/site_perl/5.30/MaxMind/DB/Reader/PP.pm line 12.
Compilation failed in require at /opt/local/lib/perl5/site_perl/5.30/Module/Runtime.pm line 314.
at /opt/local/lib/perl5/site_perl/5.30/MaxMind/DB/Reader.pm line 22.
Compilation failed in require at /opt/local/lib/perl5/site_perl/5.30/GeoIP2/Database/Reader.pm line 23.
BEGIN failed--compilation aborted at /opt/local/lib/perl5/site_perl/5.30/GeoIP2/Database/Reader.pm line 23.
Compilation failed in require at /Users/thvv/bin/thvvutil3.pm line 587.
BEGIN failed--compilation aborted at /Users/thvv/bin/thvvutil3.pm line 587.
Compilation failed in require at /Users/thvv/bin/ipcity line 8.
BEGIN failed--compilation aborted at /Users/thvv/bin/ipcity line 8.
[snip]

well shucks. My version of libmaxminddb dates back to 2018. Other issues on this list e.g. #22 suggest
that i should upgrade it. So I dowloaded the tarball.

The README.md file says I should execute
$ ./configure
$ make
$ make check
$ sudo make install
$ sudo ldconfig

but there is no file "configure". double shucks.

there is a file "configure.ac" that says "process this with autoconf to produce a configure script"
so i did

[thvv@ libmaxminddb-main]$ autoconf configure.ac
Usage: DynaLoader::dl_find_symbol(libhandle, symbolname) at /opt/local/lib/perl5/5.30/XSLoader.pm line 86.
Compilation failed in require at /opt/local/lib/perl5/5.30/File/stat.pm line 36.
BEGIN failed--compilation aborted at /opt/local/lib/perl5/5.30/File/stat.pm line 36.
Compilation failed in require at /opt/local/share/autoconf/Autom4te/FileUtils.pm line 42.
BEGIN failed--compilation aborted at /opt/local/share/autoconf/Autom4te/FileUtils.pm line 42.
Compilation failed in require at /opt/local/bin/autom4te line 46.
BEGIN failed--compilation aborted at /opt/local/bin/autom4te line 46.

{I am using macOS Mojave 10.14.6, installed Perl 5.30 with MacPorts, tried to install CPAN libraries,
had to force install several modules.}

uninstallable on Fedora32, errors: "/usr/bin/ld: unrecognized option '-Wl,-z,relro'" & "Unsupported compile language "C"" ?

On

	grep PRETTY  /etc/os-release
		PRETTY_NAME="Fedora 32 (Server Edition)"

I've

	perl -v
		This is perl 5, version 30, subversion 3 (v5.30.3) built for x86_64-linux-thread-multi
		(with 75 registered patches, see perl -V for more detail)

	perl -V | egrep " gccversion=| cc="
		gccversion='10.1.1 20200507 (Red Hat 10.1.1-1)'
		cc='gcc'

	gcc --version
		gcc (GCC) 10.2.1 20200723 (Red Hat 10.2.1-1)

	ld -v
		GNU ld version 2.34-4.fc32

	rpm -qa libmaxminddb
		libmaxminddb-1.4.2-1.fc32.x86_64

perl mod "MaxMind::DB::Reader::XS" is not currently available in distro pkgs.

attempting to install via cpanp

	cpanp -i MaxMind::DB::Reader::XS

fails

	Installing MaxMind::DB::Reader::XS (1.000008)
	[ERROR] Build.PL failed: /usr/bin/ld: unrecognized option '-Wl,-z,relro'
	/usr/bin/ld: use the --help option for usage information
	Warning: ExtUtils::CBuilder not installed or no compiler detected
	Proceeding with configuration, but compilation may fail during Build

	Checking for MMDB_lookup_string in -lmaxminddb... Checking for cc... /usr/bin/ld: unrecognized option '-Wl,-z,relro'
	/usr/bin/ld: use the --help option for usage information

	Unsupported compile language "C" at /root/.cpanplus/5.30.3/build/0WD_GYmY9N/MaxMind-DB-Reader-XS-1.000008/Build.PL line 109.


	[ERROR] Unable to create a new distribution object for 'MaxMind::DB::Reader::XS' -- cannot continue

	*** Install log written to:
	  /root/.cpanplus/install-logs/MaxMind-DB-Reader-XS-1.000008-1596687514.log

	Error installing 'MaxMind::DB::Reader::XS'
	Problem installing one or more modules

checking, distro perl's built with relro enabled in ldflags,

	perl -V | grep -i " ldflags"
	    ldflags ='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -fstack-protector-strong -L/usr/local/lib'

sanity checking with 'cpanm'

	cpanm MaxMind::DB::Reader::XS
		--> Working on MaxMind::DB::Reader::XS
		Fetching http://www.cpan.org/authors/id/M/MA/MAXMIND/MaxMind-DB-Reader-XS-1.000008.tar.gz ... OK
		Configuring MaxMind-DB-Reader-XS-1.000008 ... N/A
		! Configure failed for MaxMind-DB-Reader-XS-1.000008. See /root/.cpanm/work/1596689538.36881/build.log for details.

	/root/.cpanm/work/1596689538.36881/build.log
		...
		Running Build.PL
		/usr/bin/ld: unrecognized option '-Wl,-z,relro'
		/usr/bin/ld: use the --help option for usage information
		Warning: ExtUtils::CBuilder not installed or no compiler detected
		Proceeding with configuration, but compilation may fail during Build

		Checking for MMDB_lookup_string in -lmaxminddb... Checking for cc... /usr/bin/ld: unrecognized option '-Wl,-z,relro'
		/usr/bin/ld: use the --help option for usage information
		Unsupported compile language "C" at Build.PL line 109.

		-> N/A
		-> FAIL Configure failed for MaxMind-DB-Reader-XS-1.000008. See /root/.cpanm/work/1596689538.36881/build.log for details.

fails similarly

Reader-broken-database.t fails one test (Debian)

Installation of the module fails one test (running cpanm on Debian, libmaxminddb 1.1.4 installed from https://github.com/maxmind/libmaxminddb/releases).

Failed test 'expected exception with unknown file type' at t/MaxMind/DB/Reader-broken-databases.t line 76.
'MaxMind::DB::Reader::XS - Error opening database file "/tmp/FTtnzaB3pm/garbage": The MaxMind DB file contains invalid metadata at /home/[removed]/.cpanm/work/1/MaxMind-DB-Reader-XS-1.000001/blib/lib/MaxMind/DB/Reader/XS.pm line 68.'
doesn't match '(?^:Error opening database file "/tmp/FTtnzaB3pm/garbage": The lookup path does not match the data .+)'
Looks like you failed 1 test of 5.

Missing modules Math::Int64 and Math::Int128

MaxMind::DB::Reader started using MaxMind::DB::Reader:PP instead of the XS implementation in our CentOS 7 based Docker container yesterday. Here's how we set up the module in our Dockerfile:

RUN	curl -L https://github.com/maxmind/libmaxminddb/releases/download/1.3.2/libmaxminddb-1.3.2.tar.gz > /tmp/libmaxminddb-1.3.2.tar.gz && \
	cd /tmp && \
	tar -zxf libmaxminddb-1.3.2.tar.gz && \
	cd libmaxminddb-1.3.2 && \
	./configure && \
	make && \
	make install && \
	echo /usr/local/lib >> /etc/ld.so.conf.d/local.conf && \
	ldconfig && \
	cd .. && \
	rm -rf libmaxminddb-1.3.2*

RUN	cpanm -n -q MaxMind::DB::Reader::XS

When I tried loading it manually using perl -MMaxMind::DB::Reader::XS it complained that it couldn't locate Math::Int64 and Math::Int128. After installing those two it worked again.

cpanm -n -q Math::Int64 Math::Int128

I suspect something changed might have changed in version 1.000006 or maybe there are some missing dependencies?

Weird Travis failures

There are multiple dependency issues and bizarre failures. For instance, Net::Works is failing to install.

https://travis-ci.org/maxmind/MaxMind-DB-Reader-XS/builds/102476130

Greg says "I did look into them. Net::Works is failing to build for some, and the metadata object is getting created incorrectly on the Perl, not XS side for others. For the metadata ones, the correct hash is being passed to the MaxMind::DB::Metadata constructor but the object isn't getting constructed properly. See the debugging output from:"

https://travis-ci.org/maxmind/MaxMind-DB-Reader-XS/jobs/118084574#L873

Build.PL double-dash parameters

Prior to 1.000001 Build.PL used to accept parameters with double-dash (--), but such parameters not recognized anymore and the module fails to install (on Ubuntu at least).

This is how we build the module:

$ ./Build.PL --installdirs site --extra_compiler_flags='...' --extra_linker_flags='...'
Unknown option: installdirs
Unknown option: extra_compiler_flags
Unknown option: extra_linker_flags
Too early to specify a build action 'site'.  Do 'Build site' instead.

We can pass, extra_compiler_flags/extra_linker_flags without dashes, but --installdirs site portion is appended by CPAN itself and may vary by OS. This is what Ubuntu has:

$ grep mbuildpl_arg ~/.cpan/CPAN/MyConfig.pm
  'mbuildpl_arg' => q[--installdirs site],

It seems like Build.PL need to be fixed to accept double-dash parameters.

Reader stops working until service is restarted

After running for a while (hours, days, maybe weeks) we start getting these errors, where the reader stops working:

MaxMind::DB::Reader::XS - error decoding unknown type number 222 at [deleted]/lib/perl5/x86_64-linux/MaxMind/DB/Reader/XS.pm line 56

Sometimes we also get errors like

MaxMind::DB::Reader::XS - Entry data error looking up "106.197.196.58": The MaxMind DB file's data section contains bad data (unknown data type or corrupt data) at [deleted]/lib/perl5/x86_64-linux/MaxMind/DB/Reader/XS.pm line 56

Restarting the service fixes the issue.

new release wanted

Because of e862bd2 module fails to install with current 1.3.2 version of libmaxminddb