anrid / ipcheck

Check IPs against a large set of IP ranges. Pass in a file or a URL. Checks against known datacenter IP ranges by default.

IP Checker

• Check IP addresses against a large set of IP address ranges.
• Pass in a file or URL, e.g. an access log or a DB export containing one or more IP addresses per line.
• Checks IPs against known datacenter IP ranges by default, using data from this project.
• Optionally checks IPs against all FireHOL blocklists.
• Uses an Interval Tree for fast matching. Able to check >500k IPs against >33k IP ranges in a couple of seconds (on a MacBook Pro).

Docker Image

Image available on Docker hub: https://hub.docker.com/repository/docker/anrid/ipcheck/general

Input File

An input file with some test IPs can be found in inside the Docker image at /test-ips.txt:

$ docker run --rm --entrypoint bash anrid/ipcheck -c 'cat /test-ips.txt'

34.64.161.255
15.177.101.100
aws: ---- ip 3.2.35.193
azure(20.209.46.151)=xxx
2023-03-11.10:10:10.23020 access_log -- ip:4.4.4.4 | ip:8.8.8.8

• Each line in the input file may contain text besides IP addresses.
• The program uses a regexp to find all IPs on each line in the input file (can match one or more IPs per line).

Basic Usage

To check an input file with IP addresses against known datacenter IP ranges:

$ docker run --rm anrid/ipcheck -i /test-ips.txt

34.64.161.255  <==  GCP   | 34.64.160.0 - 34.64.191.255
aws: ---- ip 3.2.35.193  <==  AWS   | 3.2.35.192 - 3.2.35.255
azure(20.209.46.151)=xxx  <==  Azure | 20.209.0.0 - 20.209.255.255

Found 3 matches | Checked 6 IPs against 33279 ranges and 0 blocked or flagged IPs (0 dupes)

• Scanned the input file and found 6 IPs.
• Checked 6 IPs against 33,365 IP ranges (these by default) and found 3 matches.

Check a large number of IPs

To check ~600,000 IPs stored in a file locally, e.g. exported from access logs stored in Bigquery:

$ docker run --rm -v $(pwd)/../bigquery-exports:/data anrid/ipcheck -i /data/bq-results-20230311.csv

.. lots of omitted ..

54.249.174.66   <==  AWS    | 54.248.0.0 - 54.249.255.255
13.231.129.239  <==  AWS    | 13.230.0.0 - 13.231.255.255
52.197.13.28    <==  AWS    | 52.196.0.0 - 52.199.255.255

Found 882 matches | Checked 588933 IPs against 33365 ranges and 0 blocked or flagged IPs (0 dupes)

• Checked 588,933 IPs against 33,365 IP ranges and found 882 matches.
• Runtime was ~2.2 sec on a MacBook Pro.

Supply your own IP ranges

To check against other IP ranges, create a CSV file in this format:

$ cat data/test-ranges.csv

"cidr","hostmin","hostmax","vendor"
"3.2.35.192/26","x","x","AWS"
"20.209.46.0/23","x","x","Azure"
"34.64.160.0/19","x","x","GCP"

• Only the cidr and vendor columns need to be filled in.

Then pass in your CSV file using the --ip-ranges flag:

$ docker run --rm -v $(pwd)/data:/data anrid/ipcheck -i /data/test-ips.txt --ip-ranges /data/test-ranges.csv

Test against FireHOL blocklists

Being by downloading the lastest FireHOL blocklists to a local Docker volume:

# Create a new local Docker volume named `fire`
$ docker volume create fire
fire

# Download latest FireHOL blocklists into local Docker volume
$ docker run -v file:/data anrid/ipcheck --download /data/fire

Found 1337 IP sets
Loaded IP set: alienvault_reputation (0 CIDRs, 609 IPs)
Loaded IP set: asprox_c2 (0 CIDRs, 0 IPs)
Loaded IP set: bambenek_banjori (0 CIDRs, 136 IPs)

.. lots of lines omitted ..

Loaded IP set: xroxy_30d (0 CIDRs, 24 IPs)
Loaded IP set: xroxy_7d (0 CIDRs, 24 IPs)
Loaded IP set: yoyo_adservers (0 CIDRs, 9942 IPs)

Imported FireHOL 318 blocklists (120571 ranges, 3611584 blocked / flagged IPs, 4113069 dupes)

You should now have the following files in your Docker volume:

$ docker run -v file:/data --entrypoint bash anrid/ipcheck -c 'ls -l /data/fire'

total 85696
drwxrwxrwx    6 root     root         20480 Mar 10 01:05 blocklist-ipsets-master
-rw-r--r--    1 root     root      53595811 Mar 10 01:05 firehol.ips
-rw-r--r--    1 root     root      34136038 Mar 10 01:05 master.zip

• firehol.ips now contains 120,047 IP ranges and 3,604,185 blocked / flagged IPs.

To check an input file with IPs against both the FireHOL blocklists and the default datacenter ranges:

# Note that we're passing the `--verbose` to see more of what's going on.
$ docker run -v file:/data anrid/ipcheck -i /test-ips.txt --firehol-file /data/fire/firehol.ips --verbose

Reading IP ranges from https://raw.githubusercontent.com/jhassine/server-ip-addresses/master/data/datacenters.csv ..
Loaded 33279 IP ranges into interval tree
Loaded 0 IPs into hash map
Loading FireHOL data from /data/fire/firehol.ips (this takes a while) ..
Loaded 153850 IP ranges into interval tree
Loaded 3611584 IPs into hash map

34.64.161.255  <==  pushing_inertia_blocklist | Pushing Inertia | https://github.com/pushinginertia/ip-blacklist (1307 CIDRs, 2 IPs) | 34.64.0.0 - 34.127.255.255
aws---- ip3.2.35.193  <==  AWS   | 3.2.35.192 - 3.2.35.255
aaazureee 20.209.46.151xxx  <==  Azure | 20.209.0.0 - 20.209.255.255
2023-03-11.10:10:10.23020 access_log--ip:4.4.4.4 aaa 8.8.8.8  <==  iblocklist_org_joost | iBlocklist.com | https://www.iblocklist.com/ (4 CIDRs, 0 IPs) | 4.0.0.0 - 4.255.255.255

Found 4 matches | Checked 6 IPs against 153850 ranges and 3611584 blocked or flagged IPs (0 dupes)

• Note that loading the firehol.ips file into memory takes some time (~15 sec on a MacBook Pro).

Output to CSV file

# Note that the `--to-csv-file` flag takes a path to an output file.
# In this case we output to a mounted local dir.
$ docker run -v file:/data -v $(pwd)/..:/out anrid/ipcheck -i /test-ips.txt --firehol-file /data/fire/firehol.ips --to-csv-file /out/blocked-ips.csv

Found 4 matches | Checked 6 IPs against 153850 ranges and 3611584 blocked or flagged IPs (0 dupes)
Wrote /out/blocked-ips.csv

# We now have a file named `blocked-ips.csv` in $(pwd)/..
$ cat ../blocked-ips.csv

IP,Info
34.64.161.255,"pushing_inertia_blocklist | Pushing Inertia | https://github.com/pushinginertia/ip-blacklist (1307 CIDRs, 2 IPs) | 34.64.0.0 - 34.127.255.255"
3.2.35.193,AWS | 3.2.35.192 - 3.2.35.255
20.209.46.151,Azure | 20.209.0.0 - 20.209.255.255
4.4.4.4,"iblocklist_org_joost | iBlocklist.com | https://www.iblocklist.com/ (4 CIDRs, 0 IPs) | 4.0.0.0 - 4.255.255.255"