zeek / pysubnettree

A Python Module for CIDR Lookups

The Zeek Network Security Monitor

A powerful framework for network traffic analysis and security monitoring.

Key Features — Documentation — Getting Started — Development — License

Follow us on Twitter at @zeekurity.

Key Features

• In-depth Analysis Zeek ships with analyzers for many protocols, enabling high-level semantic analysis at the application layer.

• Adaptable and Flexible Zeek's domain-specific scripting language enables site-specific monitoring policies and means that it is not restricted to any particular detection approach.

• Efficient Zeek targets high-performance networks and is used operationally at a variety of large sites.

• Highly Stateful Zeek keeps extensive application-layer state about the network it monitors and provides a high-level archive of a network's activity.

Getting Started

The best place to find information about getting started with Zeek is our web site www.zeek.org, specifically the documentation section there. On the web site you can also find downloads for stable releases, tutorials on getting Zeek set up, and many other useful resources.

You can find release notes in NEWS, and a complete record of all changes in CHANGES.

To work with the most recent code from the development branch of Zeek, clone the master git repository:

git clone --recursive https://github.com/zeek/zeek

With all dependencies in place, build and install:

./configure && make && sudo make install

Write your first Zeek script:

# File "hello.zeek"

event zeek_init()
    {
    print "Hello World!";
    }

And run it:

zeek hello.zeek

For learning more about the Zeek scripting language, try.zeek.org is a great resource.

Development

Zeek is developed on GitHub by its community. We welcome contributions. Working on an open source project like Zeek can be an incredibly rewarding experience and, packet by packet, makes the Internet a little safer. Today, as a result of countless contributions, Zeek is used operationally around the world by major companies and educational and scientific institutions alike for securing their cyber infrastructure.

If you're interested in getting involved, we collect feature requests and issues on GitHub here and you might find these to be a good place to get started. More information on Zeek's development can be found here, and information about its community and mailing lists (which are fairly active) can be found here.

License

Zeek comes with a BSD license, allowing for free use with virtually no restrictions. You can find it here.

Tooling

We use the following tooling to help discover issues to fix, amongst a number of others.

• Clang-Tidy
• Coverity
• PVS-Studio - static analyzer for C, C++, C#, and Java code.

pysubnettree's Issues

Error installing Newlines are not allowed

pip3 install pysubnettree yields Newlines are not allowed error because \n is contained in the description.
Full error:

Collecting pysubnettree
  Downloading https://files.pythonhosted.org/packages/3d/32/ffa0e8150c2a455bb202e1a1ca384ff3e6ea746968d92f5a53bfd6e4b368/pysubnettree-0.35.tar.gz (51kB)
    100% |████████████████████████████████| 51kB 1.4MB/s 
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/pysubnettree.egg-info
    writing pip-egg-info/pysubnettree.egg-info/PKG-INFO
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-61uigbkv/pysubnettree/setup.py", line 22, in <module>
        setup(name="pysubnettree",
      File "/home/unknown/.local/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/unknown/.local/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 292, in run
        writer(self, ep.name, os.path.join(self.egg_info, ep.name))
      File "/home/unknown/.local/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 656, in write_pkg_info
        metadata.write_pkg_info(cmd.egg_info)
      File "/usr/lib/python3.8/distutils/dist.py", line 1117, in write_pkg_info
        self.write_pkg_file(pkg_info)
      File "/home/unknown/.local/lib/python3.8/site-packages/setuptools/dist.py", line 167, in write_pkg_file
        write_field('Summary', single_line(self.get_description()))
      File "/home/unknown/.local/lib/python3.8/site-packages/setuptools/dist.py", line 151, in single_line
        raise ValueError('Newlines are not allowed')
    ValueError: Newlines are not allowed
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-61uigbkv/pysubnettree/

Over scribing ASN to same net but distinct mask

Apparently, when you put the longest prefix it over scribe all less specific.

t["10.1.0.0/22"] = "as10"
t["10.1.0.0/24"] = "as20"
print(t["10.1.0.0/22"], t["10.1.0.0/24"])
as20 as20

Segmentation fault checking "large string" in tree

Hi,

using python 3.7.3rc1 on debian I can trigger a Segfault:

In [1]: import SubnetTree
In [2]: s=SubnetTree.SubnetTree()
In [3]: 'x' * 1000000  in s
   ...: 
Segmentation fault

Using python 2.7.16 however it works...

In [1]: import SubnetTree

In [2]: s=SubnetTree.SubnetTree()

In [3]: 'x' * 100000  in s
Out[3]: False

I have installed pysubnettree==0.28 for python 2 and 3.

ii  python-subnettree                         0.28-1                                  amd64        Python Module for CIDR Lookups
ii  python3-subnettree                        0.28-1                                  amd64        Python 3 Module for CIDR Lookups

UPDATE: sorry, seems to be fixed in 0.29 which is available via pip.

Issue When Installing

I'm on Windows 10.
I get an error when doing either pip or python setup.py install: " fatal error C1083: Cannot open include file: 'netinet/in.h': No such file or directory "

issue when installing with pip

Hi!

I'm tryig to install subnettree (python 2.7.3 64bit) using pip but I receive this message

Could not find a version that satisfies the requirement subnettree (from versions: )
No matching distribution found for subnettree

Any suggestion in order to solve this?
Roberto

Segmentation fault in 0.29 - due to bogus cidr pointer?

I am sanitizing larger amounts of bgpdumps and check prefixes against bogon lists.
As a result I keep larger amounts of data (multiple GB) in various data structures, mainly sets().

I am using python 3.5.3 and SubnetTree 0.29 from pip.
It seems the segfault does not occur using python 2.7 - at least not with the amount of data I put in. Will report back here once I observe a segfault.

After initial load the SubnetTree is used readonly using the "in" operator.

For some bgpdumps I can reproduce a Segmentation fault when checking if a certain IP-adress is in the SubnetTree. It is alway the same position in the file, so I guess the amount of data stored on the heap or stack is relevant.

A Segfault does not happen if I do not store data in the python sets.

I was able to backtrace this using gdb up to SubnetTree_wrap.cc

Program received signal SIGSEGV, Segmentation fault.
__strchr_sse2 () at ../sysdeps/x86_64/multiarch/../strchr.S:32
32      ../sysdeps/x86_64/multiarch/../strchr.S: No such file or directory.
(gdb) bt
#0  __strchr_sse2 () at ../sysdeps/x86_64/multiarch/../strchr.S:32
#1  0x00007f8ab61330a6 in strchr (__c=47, __s=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>) at /usr/include/string.h:226
#2  parse_cidr (mask=<synthetic pointer>, subnet=0x7ffd0252efe0, family=<synthetic pointer>, cidr=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>) at SubnetTree.cc:59
#3  SubnetTree::lookup (this=this@entry=0x56249ae10310, cidr=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>, size=<optimized out>) at SubnetTree.cc:257
#4  0x00007f8ab6137288 in SubnetTree___contains____SWIG_0 (size=<optimized out>, cidr=<optimized out>, self=0x56249ae10310) at SubnetTree_wrap.cc:3422
#5  _wrap_SubnetTree___contains____SWIG_0 (args=<optimized out>) at SubnetTree_wrap.cc:4585
#6  _wrap_SubnetTree___contains__ (self=<optimized out>, args=<optimized out>) at SubnetTree_wrap.cc:4664
#7  0x00005624989bf6df in PyCFunction_Call ()
#8  0x0000562498981ab2 in PyEval_EvalFrameEx ()
#9  0x00005624989850df in PyEval_EvalCodeEx ()
#10 0x00005624989c15d3 in ?? ()
#11 0x0000562498a08647 in PyObject_Call ()
#12 0x000056249892894e in ?? ()
#13 0x0000562498a08647 in PyObject_Call ()
#14 0x00005624989c9a15 in ?? ()
#15 0x000056249897fa6e in PyEval_EvalFrameEx ()
#16 0x000056249897f93f in PyEval_EvalFrameEx ()
#17 0x0000562498984286 in ?? ()
#18 0x0000562498984f9f in PyEval_EvalCode ()
#19 0x0000562498a528f2 in ?? ()
#20 0x0000562498a54e1d in PyRun_FileExFlags ()
#21 0x0000562498a555be in PyRun_SimpleFileExFlags ()
#22 0x0000562498a834d7 in Py_Main ()
#23 0x0000562498913c01 in main ()


(gdb) up
#1  0x00007f8ab61330a6 in strchr (__c=47, __s=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>) at /usr/include/string.h:226
226       return __builtin_strchr (__s, __c);
(gdb) 
#2  parse_cidr (mask=<synthetic pointer>, subnet=0x7ffd0252efe0, family=<synthetic pointer>, cidr=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>) at SubnetTree.cc:59
59      SubnetTree.cc: No such file or directory.
(gdb) print subnet
$1 = (inx_addr *) 0x7ffd0252efe0
(gdb) print *subnet
$2 = {sin = {s_addr = 3057479128}, sin6 = {__in6_u = {__u6_addr8 = "\330m=\266\212\177\000\000\005\000\000\000\000\000\000", __u6_addr16 = {28120, 46653, 32650, 0, 5, 0, 0, 0}, __u6_addr32 = {3057479128, 32650, 5, 0}}}}
(gdb) print *cidr
Cannot access memory at address 0x7f8ab4073050
(gdb) up
#3  SubnetTree::lookup (this=this@entry=0x56249ae10310, cidr=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>, size=<optimized out>) at SubnetTree.cc:257
257     in SubnetTree.cc
(gdb) 
#4  0x00007f8ab6137288 in SubnetTree___contains____SWIG_0 (size=<optimized out>, cidr=<optimized out>, self=0x56249ae10310) at SubnetTree_wrap.cc:3422
3422    SubnetTree_wrap.cc: No such file or directory.
(gdb) 
#5  _wrap_SubnetTree___contains____SWIG_0 (args=<optimized out>) at SubnetTree_wrap.cc:4585
4585    in SubnetTree_wrap.cc
(gdb) 
#6  _wrap_SubnetTree___contains__ (self=<optimized out>, args=<optimized out>) at SubnetTree_wrap.cc:4664
4664    in SubnetTree_wrap.cc
(gdb)  

signal 11 during ip lookup

our celery task is crashing occasionally with signal 11 in subnetree get_item. Ubuntu 20.04

if self.iptree[str(lookup_ip)]: (Python) File "/opt/ns/python-apps/nslogcollector/app-env/lib/python3.8/site-packages/SubnetTree.py", line 157, in __getitem__ return _SubnetTree.SubnetTree___getitem__(self, cidr)

pysubnettree==0.33
celery==5.2.3

Semantics of lookup of CIDR

I just found out that you can try to look up a CIDR in the SubnetTree. But it seems to just take the network part, complete it with zeros and look up that address. Is their some (documented or undocumented) semantics attached to this? It was a bit confusing for me to discover:

import SubnetTree
tree = SubnetTree.SubnetTree()
tree['10.0.0.0/16'] = 'Test net'
print t['10.0.0.0/8']
-> Test net

Add type annotations?

Type annotations are a useful feature for consumers of any library, so it would be nice if this library had it as well.

segmentation fault on bad cidr

the following is causing a segmentation fault on OSX:

In [1]: import SubnetTree
In [2]: t = SubnetTree.SubnetTree()
In [3]: t.insert('8.8.8.8/56')
Segmentation fault: 11

a bit more info:

  * frame #0: 0x000000010497b128 _SubnetTree.so`patricia_lookup + 200
    frame #1: 0x00000001049799bc _SubnetTree.so`SubnetTree::insert(int, _inx_addr, unsigned short, _object*) + 204
    frame #2: 0x0000000104979767 _SubnetTree.so`SubnetTree::insert(char const*, _object*) + 71
    frame #3: 0x000000010497d1f2 _SubnetTree.so`_wrap_SubnetTree_insert(_object*, _object*) + 370

Automatic pypi upload via GitHub actions

The current Makefile has twine -u zeek .. and that's not working anymore due to two-factor authentication for user zeek. Needs twine -u __token__ instead, but maybe should move to GitHub action.

Prefix lookups possible?

Hi there,

I was just wondering, whether it's also possible to directly lookup prefixes (instead of IPs) in the SubnetTree.
I tried the following:

from SubnetTree import SubnetTree

subnet_tree = SubnetTree()

subnet_tree['10.0.0.0/24'] = "/24"
subnet_tree['10.0.0.0/25'] = "/25"

print(subnet_tree['10.0.0.0/24'])

As output I received: "/25"

I am wondering what happens here, as I expected "/24" as result.
First idea that came to my mind, was, that the lookup (last line) just cuts-off the trailing "/24" and uses the prefix as IP, which would match the /25 (longest prefix). Otherwise, may this be a bug?

EDIT: I am using version 0.33 with Python 3.8.1 on macOS

swig can't find header file in new location

In Debian we rebuild SubnetTree_wrap.cc with swig every time we build the package, just to make sure we can with the swig version we have. With 0.32, this fails because SubnetTree.i wasn't updated for the new include directory. Please see the patch below that fixes the issue for me.

Scott K

--- pysubnettree-0.32.orig/SubnetTree.i
+++ pysubnettree-0.32/SubnetTree.i
@@ -1,8 +1,8 @@

 %module SubnetTree
 %{
-  #include "SubnetTree.h"
+  #include "include/SubnetTree.h"
 %}

-%include "SubnetTree.h"
+%include "include/SubnetTree.h"

IPv6 buggs

I noticed that IPv6 in general is not really matched correctly. This is the complete reserved IP space currently known. When doing this you get all kind of weird ass matches on the mapped IPv4 space etc.

RESERVED_IP = (
    # IPv4 reserved space
    '0.0.0.0/8',               # Used for broadcast messages to the current ("this") (Disabled because of IPv6 bug, reported)
    '10.0.0.0/8',               # Used for local communications within a private network
    '100.64.0.0/10',            # Used for communications between a service provider and its subscribers when using a Carrier-grade NAT
    '127.0.0.0/8',              # Used for loopback addresses to the local host
    '169.254.0.0/16',           # Used for link-local addresses between two hosts on a single link when no IP address is otherwise specified
    '172.16.0.0/12',            # Used for local communications within a private network
    '192.0.0.0/24',             # Used for the DS-Lite transition mechanism
    '192.0.2.0/24',             # Assigned as "TEST-NET"
    '192.88.99.0/24',           # Used by 6to4 anycast relay
    '192.168.0.0/16',           # Used for local communications within a private network
    '198.15.0.0/15',            # Used for testing of inter-network communications between two separate subnets
    '198.51.100.0/24',          # Assigned as "TEST-NET-2"
    '203.0.113.0/24',           # Assigned as "TEST-NET-3"
    '224.0.0.0/4',              # Reserved for multicast assignments, 233.252.0.0/24 is assigned as "MCAST-TEST-NET"
    '240.0.0.0/4',              # Reserved for future use
    '255.255.255.255/32'        # Reserved for the "limited broadcast" destination address
    # IPv6 reserved space
    '::/128'                    # Unspecified address
    '::1/128'                   # loopback address to the local host.
    #'::ffff:0:0/96'            # IPv4 mapped addresses (Disabled and redone reserved IPv4 in a mapped IPv4 space)
    '100::/64'                  # Discard Prefix RFC 6666
    '64:ff9b::/96'              # IPv4/IPv6 translation (RFC 6052)
    '2001::/32'                 # Teredo tunneling
    '2001:10::/28'              # Deprecated (previously ORCHID)
    '2001:20::/28'              # ORCHIDv2
    '2001:db8::/32'             # Addresses used in documentation
    '2002::/16'                 # 6to4
    'fc00::/7'                  # Unique local address
    'fe80::/10'                 # Link-local address
    'ff00::/8'                  # Multicast
    # Inside mapped IPv4 space this is still reserved (BUGGY AS HELL ATM)
    '::ffff:0.0.0.0/8',         # Used for broadcast messages to the current ("this") (Disabled because of IPv6 bug, reported)
    '::ffff:10.0.0.0/8',        # Used for local communications within a private network
    '::ffff:100.64.0.0/10',     # Used for communications between a service provider and its subscribers when using a Carrier-grade NAT
    '::ffff:127.0.0.0/8',       # Used for loopback addresses to the local host
    '::ffff:169.254.0.0/16',    # Used for link-local addresses between two hosts on a single link when no IP address is otherwise specified
    '::ffff:172.16.0.0/12',     # Used for local communications within a private network
    '::ffff:192.0.0.0/24',      # Used for the DS-Lite transition mechanism
    '::ffff:192.0.2.0/24',      # Assigned as "TEST-NET"
    '::ffff:192.88.99.0/24',    # Used by 6to4 anycast relay
    '::ffff:192.168.0.0/16',    # Used for local communications within a private network
    '::ffff:198.15.0.0/15',     # Used for testing of inter-network communications between two separate subnets
    '::ffff:198.51.100.0/24',   # Assigned as "TEST-NET-2"
    '::ffff:203.0.113.0/24',    # Assigned as "TEST-NET-3"
    '::ffff:224.0.0.0/4',       # Reserved for multicast assignments, 233.252.0.0/24 is assigned as "MCAST-TEST-NET"
    '::ffff:240.0.0.0/4',       # Reserved for future use
    '::ffff:255.255.255.255/32' # Reserved for the "limited broadcast" destination address
)

IPv6 matching on IPv4

Strange enough, some IPv6 addresses are matching on IPv4 CIDR notations, not sure if this is intended or not (but I think it should not be the case)

import SubnetTree
t1 = SubnetTree.SubnetTree()
t1.insert("0.0.0.0/8")
print t1["127.0.0.1"] # KeyError {CORRECT}
print t1["0.1.2.3"] # <dummy string> {CORRECT}
print t1["1:2:3:4::5"] # <dummy string> {WRONG}
print t1["1:2:3:4::5"] # <dummy string> {WRONG}
print t1["::ffff:197.44.42.34"] # <dummy string> {WRONG}

This behaviour is only with the 0.0.0.0/X range as far as I could test.

Thread safety

Is PySubnetTree thread-safe?

Install error on Debian

run python2.7 setup.py install
got error: c++: error: unrecognized command line option '-fstack-protector-strong'

error: command 'c++' failed with exit status 1

i'm not familiar with C++,so can you give some adivce?

g++ version
g++ (Debian 4.7.2-5) 4.7.2

Is this package suitable for Windows?

Sorry if this is a dumb question but when I run
pip install pysubnettree
the installation on my Win10 fails because of missing netinet/in.h which is to my knowledge Linux/Unix library. So is my assumption correct that pysubnettree is not suitable for windows at the moment? And if so are there possibly any plans to port it?

C:\Users\xxx\AppData\Local\Temp\pip-install-llhms6k7\pysubnettree\include\patricia.h(65): fatal error C1083: Cannot open include file: 'netinet/in.h': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x86\\cl.exe' failed with exit status 2

PySubnetTree on pypi.python.org

Why don't upload PySubnetTree to pypi.python.org?

Very high memory usage with large sets

Hello, while importing a 115MB list of CIDRs into a pysubnettree set, I noticed that the final memory structure was ~10x the size of the list of strings.

This is counterintuitive: I'd have expected it to be radically smaller. What am I missing?

SWIG-Wrapper not updated after #14

Our Segfault from #19 is back. This time being reproducible, I found the likely culprit.

The fix from #14 was only applied to the SWIG input file. The generated SWIG wrappers are unchanged in the repo and also in the Source-Tarballs in PyPI. Looking at SubnetTree_wrap.cc shows the reference count of ascii being decremented before calling the lookup functions in the contains-Wrappers.

Just installing the package with python3 setup,py install does not recreate the wrappers and just installs the pre #14 version susceptible to the Segfault. Removing the wrappers and recreating them with

make clean
make
python3 setup.py install

fixes the Segfault for us.

Segfault with python 3.6.10 and subnettree 0.33

We experience segmentation faults with 0.33 that look very similar to #15. We have a daemon that manages a couple hundred SubnetTrees in a dict. Each tree contains a good handful of networks. The daemon processes files from a directory. With each file the trees are reinitialized by emptying the dictionary and setting it back up by inserting the current networks returned by an API. After that the trees aren't modified and the IP addresses found in the files are looked up against all the trees. For the next file this process repeats: reinitialize the trees, lookup all IPs.

From time to time the daemon dies with a segfault. Unfortunately not always choking on the same file, so restarting it and processing the same file again succeeds sometimes, suggesting it is connected to the history of files processed.

I have a traceback from a core dump. The actual IP address substituted below by 'aaa.bbb.ccc.ddd'.

Program terminated with signal SIGSEGV, Segmentation fault.                                                                                                                
#0  __strchr_sse2 () at ../sysdeps/x86_64/multiarch/../strchr.S:32                                                                                                         
32      ../sysdeps/x86_64/multiarch/../strchr.S: Datei oder Verzeichnis nicht gefunden.                                                                                    
[Current thread is 1 (Thread 0x7f6f7ff58540 (LWP 1454))]                                                                                                                   
(gdb) bt
#0  __strchr_sse2 () at ../sysdeps/x86_64/multiarch/../strchr.S:32                                                                                                         
#1  0x00007f6f756cea96 in SubnetTree::lookup(char const*, int) const () from /usr/lib64/python3.6/site-packages/_SubnetTree.cpython-36m-x86_64-linux-gnu.so                
#2  0x00007f6f756d2da3 in _wrap_SubnetTree___contains__ () from /usr/lib64/python3.6/site-packages/_SubnetTree.cpython-36m-x86_64-linux-gnu.so                             
#3  0x00007f6f7f91461f in PyCFunction_Call (func=func@entry=<built-in method SubnetTree___contains__ of module object at remote 0x7f6f758e95e8>, args=args@entry=          
    (<SubnetTree(this=<SwigPyObject at remote 0x7f6f71b5a090>) at remote 0x7f6f720091d0>, 'aaa.bbb.ccc.ddd'), kwds=kwds@entry=0x0) at Objects/methodobject.c:114           
#4  0x00007f6f7f965a41 in do_call_core (kwdict=0x0, callargs=(<SubnetTree(this=<SwigPyObject at remote 0x7f6f71b5a090>) at remote 0x7f6f720091d0>, '193.254.135.252'),     
    func=<built-in method SubnetTree___contains__ of module object at remote 0x7f6f758e95e8>) at Python/ceval.c:5116                                                       
#5  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3404                                                                         
#6  0x00007f6f7f969474 in _PyEval_EvalCodeWithName (qualname='SubnetTree.__contains__', name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2,     
    kwcount=<optimized out>, kwargs=<optimized out>, kwnames=<optimized out>, argcount=<optimized out>, args=0x0, locals=0x0, globals=<optimized out>,                     
    _co=<code at remote 0x7f6f758ec5d0>) at Python/ceval.c:4166                                                                                                            
#7  _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7f6f758f4950>, args=args@entry=0x7ffc76dd8940, nargs=<optimized out>, kwargs=kwargs@entry=0x0)         
    at Python/ceval.c:5084                                                                                                                                                 
#8  0x00007f6f7f8ddc3e in _PyObject_FastCallDict (func=<function at remote 0x7f6f758f4950>, args=0x7ffc76dd8940, nargs=<optimized out>, kwargs=0x0)                        
    at Objects/abstract.c:2310                                                                                                                                             
#9  0x00007f6f7f8de4e1 in _PyObject_Call_Prepend (func=<function at remote 0x7f6f758f4950>, obj=<optimized out>, args=('193.254.135.252',), kwargs=0x0)                    
    at Objects/abstract.c:2373                                                                                                                                             
#10 0x00007f6f7f8ddb6c in _PyObject_FastCallDict (func=func@entry=<method at remote 0x7f6f72d55888>, args=args@entry=0x7ffc76dd8a28, nargs=nargs@entry=1,                  
    kwargs=kwargs@entry=0x0) at Objects/abstract.c:2331                                                                                                                    
#11 0x00007f6f7f9274a6 in slot_sq_contains (self=<SubnetTree(this=<SwigPyObject at remote 0x7f6f71b5a090>) at remote 0x7f6f720091d0>, value=<optimized out>)               
    at Objects/typeobject.c:5957                                                                                                                                           
#12 0x00007f6f7f9646c1 in cmp_outcome (w=<SubnetTree(this=<SwigPyObject at remote 0x7f6f71b5a090>) at remote 0x7f6f720091d0>, v='aaa.bbb.ccc.ddd', op=<optimized out>)     
    at Python/ceval.c:5186                                                                                                                                                 
#13 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2883                                                                         
#14 0x00007f6f7f967e2b in _PyEval_EvalCodeWithName (qualname='RPCaller.lookup', name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=<optimized out>,     
    defs=0x7f6f750186d8, kwstep=1, kwcount=3, kwargs=<optimized out>, kwnames=0x7f6f749635b8, argcount=<optimized out>, args=<optimized out>, locals=0x0,                  
    globals=<optimized out>, _co=<code at remote 0x7f6f75019300>) at Python/ceval.c:4166                                                                                   
#15 fast_function (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:4992
[...several more levels in our code...]

The location of the segfault seems to be similar to #15. I cannot get the local variables from the frames above 0, but looking into the code at frame 0

Dump of assembler code for function __strchr_sse2:
   0x00007f6f7f2a4290 <+0>:     movd   %esi,%xmm1
   0x00007f6f7f2a4294 <+4>:     mov    %edi,%eax
   0x00007f6f7f2a4296 <+6>:     and    $0xfff,%eax
   0x00007f6f7f2a429b <+11>:    punpcklbw %xmm1,%xmm1
   0x00007f6f7f2a429f <+15>:    cmp    $0xfc0,%eax
   0x00007f6f7f2a42a4 <+20>:    punpcklwd %xmm1,%xmm1
   0x00007f6f7f2a42a8 <+24>:    pshufd $0x0,%xmm1,%xmm1
   0x00007f6f7f2a42ad <+29>:    jg     0x7f6f7f2a4410 <__strchr_sse2+384>
=> 0x00007f6f7f2a42b3 <+35>:    movdqu (%rdi),%xmm0
   0x00007f6f7f2a42b7 <+39>:    pxor   %xmm3,%xmm3
   0x00007f6f7f2a42bb <+43>:    movdqa %xmm0,%xmm4

As far as I read it, rdi should contain the CIDR here and trying to access it shows

(gdb) x $rdi                                                                                                                                                               
0x7f6f70a40050: Cannot access memory at address 0x7f6f70a40050

We are trying the workaround from #15 to see if it helps in our case as well.

I know this is not much to work on. Anything else I can try to extract from the core dump? Or things we could try to log or capture if it segfaults again? As I said, we weren't able to reproduce it with a test case, therefore it's the regular process that runs until it eventually dies.

Swig 4 issue

In Debian we switched from swig 3 to swig 4 and it did not go well. The specific error we get after SubnetTree_wrap.cc is regenerated with swig 4 is:

import SubnetTree
t = SubnetTree.SubnetTree()
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3/dist-packages/SubnetTree.py", line 82, in init
_SubnetTree.SubnetTree_swiginit(self, _SubnetTree.new_SubnetTree(binary_lookup_mode))
AttributeError: module '_SubnetTree' has no attribute 'SubnetTree_swiginit'

Any suggestions you might have on how to get this working with swig 4 would be greatly appreciated.