GithubHelp home page GithubHelp logo

projectdiscovery / mapcidr Goto Github PK

View Code? Open in Web Editor NEW
938.0 26.0 93.0 1.03 MB

Utility program to perform multiple operations for a given subnet/CIDR ranges.

Home Page: https://projectdiscovery.io

License: MIT License

Dockerfile 0.32% Go 98.85% Makefile 0.33% Shell 0.51%
cidr subnetting subnetwork cidr-ranges

mapcidr's Introduction

mapCIDR

A utility program to perform multiple operations for a given subnet/cidr ranges.

FeaturesInstallUsageLibraryJoin Discord


mapCIDR is developed to ease load distribution for mass scanning operations, it can be used both as a library and as independent CLI tool.

Features

mapCIDR

  • CIDR expansion support (default)
  • CIDR slicing support (sbh, sbc)
  • CIDR/IP aggregation support (a, aa)
  • CIDR/IP matcher support (match-ip)
  • CIDR/IP filter support (filter-ip)
  • CIDR/IP sorting support (s, sr)
  • CIDR host count support (count)
  • Multiple IP Format support (ip-format)
  • IP/PORT shuffling support (si, sp)
  • IPv4/IPv6 Conversation support (t4, t6)
  • CIDR STDIN (pipe) input support

Installation

go install -v github.com/projectdiscovery/mapcidr/cmd/mapcidr@latest

Usage

mapcidr -h

This will display help for the tool. Here are all the switches it supports.

INPUT:
   -cl, -cidr string[]  CIDR/IP/File containing list of CIDR/IP to process

PROCESS:
   -sbc int                  Slice CIDRs by given CIDR count
   -sbh int                  Slice CIDRs by given HOST count
   -a, -aggregate            Aggregate IPs/CIDRs into minimum subnet
   -aa, -aggregate-approx    Aggregate sparse IPs/CIDRs into minimum approximated subnet
   -c, -count                Count number of IPs in given CIDR
   -t4, -to-ipv4             Convert IPs to IPv4 format
   -t6, -to-ipv6             Convert IPs to IPv6 format
   -ip-format, -if string[]  IP formats (0,1,2,3,4,5,6,7,8,9,10,11)
   -zpn, -zero-pad-n int     number of padded zero to use (default 3)
   -zpp, -zero-pad-permute   enable permutations from 0 to zero-pad-n for each octets

FILTER:
   -f4, -filter-ipv4         Filter IPv4 IPs from input
   -f6, -filter-ipv6         Filter IPv6 IPs from input
   -skip-base                Skip base IPs (ending in .0) in output
   -skip-broadcast           Skip broadcast IPs (ending in .255) in output
   -mi, -match-ip string[]   IP/CIDR/FILE containing list of IP/CIDR to match (comma-separated, file input)
   -fi, -filter-ip string[]  IP/CIDR/FILE containing list of IP/CIDR to filter (comma-separated, file input)

MISCELLANEOUS:
   -s, -sort                  Sort input IPs/CIDRs in ascending order
   -sr, -sort-reverse         Sort input IPs/CIDRs in descending order
   -si, -shuffle-ip           Shuffle Input IPs in random order
   -sp, -shuffle-port string  Shuffle Input IP:Port in random order

UPDATE:
   -up, -update                 update mapcidr to latest version
   -duc, -disable-update-check  disable automatic mapcidr update check
   
OUTPUT:
   -verbose            Verbose mode
   -o, -output string  File to write output to
   -silent             Silent mode
   -version            Show version of the project

Running mapCIDR

In order to get list of IPs for a give CIDR, use the following command.

CIDR expansion

mapcidr -cidr 173.0.84.0/24
                   ____________  ___    
  __ _  ___ ____  / ___/  _/ _ \/ _ \
 /  ' \/ _ '/ _ \/ /___/ // // / , _/   
/_/_/_/\_,_/ .__/\___/___/____/_/|_| v0.5
          /_/                                                     	 

		projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions
[WRN] Developers assume no liability and are not responsible for any misuse or damage.

173.0.84.0
173.0.84.1
173.0.84.2
173.0.84.3
173.0.84.4
173.0.84.5
173.0.84.13
173.0.84.14
173.0.84.15
173.0.84.16

It is also possible to get list of IP's for a given IP range, use the following command

$ echo "192.168.0.0-192.168.0.5" | mapcidr
192.168.0.0
192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.4
192.168.0.5

CIDR Slicing by CIDR Count

In order to slice given CIDR or list of CIDR by CIDR count or slice into multiple and equal smaller subnets, use the following command.

mapcidr -cidr 173.0.84.0/24 -sbc 10 -silent
173.0.84.0/27
173.0.84.32/27
173.0.84.64/27
173.0.84.96/27
173.0.84.128/27
173.0.84.160/27
173.0.84.208/28
173.0.84.192/28
173.0.84.240/28
173.0.84.224/28

CIDR slicing by HOST Count

In order to slice given CIDR for equal number of host count in each CIDR, use the following command.

mapcidr -cidr 173.0.84.0/16 -sbh 20000 -silent
173.0.0.0/18
173.0.64.0/18
173.0.128.0/18
173.0.192.0/18

Note: it's possible to obtain a perfect split only when the desired amount of slices or hosts per subnet is a powers of two. Otherwise, the tool will attempt to automatically find the best split strategy to obtain the desired outcome.

CIDR/IP Aggregation

In order to merge multiple CIDR ranges into smaller subnet block, use the following command.

$ mapcidr -cl cidrs.txt -aggregate

In order to list CIDR blocks for given list of IPs, use the following command.

$ mapcidr -il ips.txt -aggregate

It's also possible to perform approximated aggregations for sparse ips groups (only version 4). The final interval will contain contiguous ips not belonging to the input:

$ cat ips.txt 

1.1.1.1
1.1.1.16
1.1.1.31
$ cat ips.txt | mapcidr -aggregate-approx

1.1.1.0/27

In order to list CIDR blocks for given IP Range (IPv4 | IPv6), use the following command.

 $ mapcidr  -cl 192.168.0.1-192.168.0.255 -aggregate
 OR
 $ echo 192.168.0.1-192.168.0.255 | mapcidr -aggregate
192.168.0.1/32
192.168.0.2/31
192.168.0.4/30
192.168.0.8/29
192.168.0.16/28
192.168.0.32/27
192.168.0.64/26
192.168.0.128/25

Match / Filter IP's from CIDR

In order to match IPs from the given list of CIDR ranges, use the following command.

$ mapcidr -cidr 192.168.1.0/24 -mi 192.168.1.253,192.168.1.252
$ mapcidr -cidr 192.168.1.0/24 -mi ip_list_to_match.txt

In order to match IPs from the given list of CIDR ranges, use the following command.

$ mapcidr -cidr 192.168.1.224/28 -fi 192.168.1.233,192.168.1.234
$ mapcidr -cidr 192.168.1.224/28 -fi ip_list_to_filter.txt

IP Formats

In order to represent given IP into multiple formats, -if 0 flag can be used to display all the supported format values, and specific type of format can be displayed using specific index number as listed here, currently 10 unique formats are supported.

$ echo 127.0.1.0 | mapcidr -if 0 -silent

127.0.1.0
127.1
0177.0.01.0
0x7f.0x0.0x1.0x0
0x7f000100
0xabfa659dfa7f000100
281472812450048
111111111111111101111111000000000000000100000000
0x7f.0.01.0x0
::ffff:7f00:0100
%31%32%37%2E%30%2E%31%2E%30
127.000.001.000

IP Conversion

IPv4 | IPv6 addresses can be converted from either the v6 to v4 notation or IPv4-mapped notation into IPv4 addresses using -t4 and -t6 to IPv4 and IPv6 respectively.

$ cat ips.txt 

1.1.1.1
2.2.2.2
$ mapcidr -cl ips.txt -t6

00:00:00:00:00:ffff:0101:0101
00:00:00:00:00:ffff:0202:0202

Note:

Not all IPv6 address can be converted to IPv4. You can only convert valid IPv4 represented IPv6 addresses.

CIDR Host Counting

In order to count number of hosts for a given CIDR or list of CIDR, use the following command.

$ echo 173.0.84.0/16 | mapcidr -count -silent

65536

ASN Input

In order to get the IP address of ASN number, use the following command

echo AS15133 | mapcidr -silent

5.104.64.0
5.104.64.1
5.104.64.2
5.104.64.3
5.104.64.4

Use mapCIDR as a library

It's possible to use the library directly in your go programs. The following code snippets outline how to divide a cidr into subnets, and how to divide the same into subnets containing a certain number of hosts

package main

import (
	"fmt"

	"github.com/projectdiscovery/mapcidr"
)

func main() {
	// Divide the CIDR into two subnets
	subnets1 := mapcidr.SplitN("192.168.1.0/24", 2)
	for _, subnet := range subnets1 {
		fmt.Println(subnet)
	}
	// Divide the CIDR into two subnets containing 128 hosts each
	subnets2 := mapcidr.SplitByNumber("192.168.1.0/24", 128)
	for _, subnet := range subnets2 {
		fmt.Println(subnet)
	}

	// List all ips in the CIDR
	ips, _ := mapcidr.IPAddresses("192.168.1.0/24")
	for _, ip := range ips {
		fmt.Println(ip)
	}
}

mapCDIR is made with 🖤 by the projectdiscovery team.

mapcidr's People

Contributors

actions-user avatar dependabot[bot] avatar dogancanbakir avatar edoardottt avatar ehrishirajsharma avatar ehsandeep avatar forgedhallpass avatar hexagr avatar ice3man543 avatar luitelsamikshya avatar m09ic avatar mzack9999 avatar ramanareddy0m avatar shubhamrasal avatar tarunkoyalwar avatar xm1k3 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mapcidr's Issues

All pre-build assets have gone from Release-tag (latest) on github

All pre-build assets have gone starting from v1.1.4 (pushed 19 hours ago). Results in multiple distro scripts failing because of missing assets when trying to pull 'latest' tag.

My gut feeling is the CI/CD script did not execute correctly on GitHub. Please check & thanks for the great work. 🤗

[Issue] using mapcidr as a lib in readme points to a wrong function name

Hi guys,
Found that your lib offers a mapcidr.IPAddresses function, but your Readme doc has a mapcidr.Ips reference:

// List all ips in the CIDR
	ips, _ := mapcidr.Ips("192.168.1.0/24")
	for _, ip := range ips {
		fmt.Println(ip)
	}

it should be just replaced by

ips, _ := mapcidr.IPAddresses("192.168.1.0/24")

Ty!

ASN via URL or FIle as RIPE query is limited

1st - Describe the solution you'd like

I have my own RIPE database of ASNs that belong on my server that gets daily updated as using RIPE can only query for a limited amount
https://www.ripe.net/manage-ips-and-asns/db/support/querying-the-ripe-database

Something like this instead of the default?

./mapcidr -asnurl http://myripedatabaseserver/?get=AS206067 -aggregate >> Firewall

Also might come useful for Tor, Using -aggregate should make them small or /32 if single use.

./mapcidr -asnurl https://check.torproject.org/torbulkexitlist -aggregate > Tor

2nd Describe the solution you'd like

I would like to see an option for file as Google Cloud and Amazon AWS have theses in a json file e.g

https://ip-ranges.amazonaws.com/ip-ranges.json

So i would like to see something like

./mapcidr -asnfile ip-ranges.json -aggregate > AmazonAWS

Using -aggregate should make them smallest cidr,

There is way around for AWS using jq to get the CDIRs

jq -r '.prefixes | .[].ip_prefix' < ip-ranges.json > AmazonIPv4
jq -r '.ipv6_prefixes | .[].ipv6_prefix' < ip-ranges.json > AmazonIPv6

Adding support for IPs aggregation

With the addition of #8 mapCIDR supports aggregating multiple CIDRs into a smaller CIDR block when available and only support CIDR input.

For example, here is cirds.txt

137.155.0.0/16
137.155.2.0/24
137.155.11.0/24
137.155.12.0/24
137.155.121.0/24
137.155.122.0/24
137.155.128.0/22
137.155.156.0/24
137.155.192.0/24
137.155.193.0/24
137.155.194.0/24
137.155.195.0/24
137.155.196.0/24
137.155.197.0/24
137.155.198.0/24
137.155.199.0/24
137.155.200.0/24
137.155.240.0/24
137.155.241.0/24
137.155.242.0/24
137.155.249.0/24
137.155.250.0/24
137.155.253.0/24
137.155.254.0/24
137.155.255.0/24
cat cirds.txt | ./mapcidr -silent -aggregate
137.155.0.0/16

We can expand this support for IPs as well, where IPs along with CIDR can be used as input to aggregate and output CIDR blocks.

where input will look like:-

200.200.200.200
192.168.2.2
192.168.2.99
192.168.70.33
192.168.1.240
192.168.240.1
10.0.0.22
10.0.1.33
10.0.90.9
172.16.45.1
172.16.43.1
172.16.47.1
cat ips.txt | ./mapcidr -silent -aggregate

or

./mapcidr -ips ips.txt -silent -aggregate

Multi Format IP conversion support

Is your feature request related to a problem? Please describe.

IP obfuscation can be used to bypass multiple waf/regex based protection which supposed to block specific IP format, multi format support could come handy to test those protections, hence adding IP conversation would be useful addition into mapcidr.

Describe the solution you'd like

   -if, -ip-format  Covert IPs in given format (0,1,2,3,4,5,6,7,8)
$ echo 127.0.0.1 | mapcidr -ip-format 0

127.1
0177.0.0.01
0x7f.0x0.0x0.0x1
2130706433
01111111000000000000000000000001
127.0x1
00:00:00:00:00:ffff:7f00:0001
%31%32%37%2E%30%2E%30%2E%31

More example inputs:

$ echo 192.168.1.0/24 | mapcidr -ip-format 1
$ cat cidrs.txt | mapcidr -ip-format 1

Format to support and index to map with types:

Format Index Example
All 0  all below one
0 Optimized dotted decimal 1 127.1
Octal 2 0177.0.0.01
Hexadecimal 3 0x7f.0x0.0x0.0x1
Dword 4 2130706433
Binary 5 01111111000000000000000000000001
Mixed 6 127.0x1
IPv6 7 00:00:00:00:00:ffff:7f00:0001
URL encoded 8 %31%32%37%2E%30%2E%30%2E%31

Reference: https://www.hacksparrow.com/networking/many-faces-of-ip-address.html

[Issue] This project has too many dependencies.

Describe the bug

golang.org/x/net/html/atom
golang.org/x/net/internal/iana
golang.org/x/exp/maps
github.com/projectdiscovery/gologger/levels
github.com/dsnet/compress/internal
github.com/dsnet/compress/bzip2/internal/sais
github.com/ulikunitz/xz/internal/hash
github.com/tidwall/match
github.com/tidwall/tinyqueue
github.com/projectdiscovery/blackrock
github.com/tidwall/rtred/base
github.com/VividCortex/ewma
github.com/tidwall/rtred
github.com/rivo/uniseg
github.com/tidwall/btree
github.com/cnf/structhash
github.com/pkg/errors
github.com/aymerick/douceur/css
golang.org/x/net/html
github.com/gorilla/css/scanner
github.com/asaskevich/govalidator
github.com/aymerick/douceur/parser
github.com/saintfish/chardet
github.com/microcosm-cc/bluemonday/css
gopkg.in/yaml.v3
golang.org/x/net/bpf
golang.org/x/sys/unix
github.com/microcosm-cc/bluemonday - a fast golang HTML sanitizer (inspired by the OWASP Java HTML Sanitizer) to scrub user generated content of XSS 
github.com/projectdiscovery/utils/slice
github.com/modern-go/concurrent
github.com/modern-go/reflect2
github.com/projectdiscovery/utils/strings
github.com/logrusorgru/aurora
github.com/projectdiscovery/utils/file
golang.org/x/net/internal/socket
github.com/dsnet/compress/internal/errors
github.com/json-iterator/go
github.com/dsnet/compress
github.com/golang/snappy
github.com/dsnet/compress/internal/prefix
github.com/nwaples/rardecode
golang.org/x/net/ipv4
golang.org/x/net/ipv6
github.com/dsnet/compress/bzip2
github.com/pierrec/lz4/internal/xxh32
github.com/miekg/dns
github.com/ulikunitz/xz/internal/xlog
github.com/pierrec/lz4
github.com/ulikunitz/xz/lzma
github.com/xi2/xz
github.com/projectdiscovery/gologger/formatter
gopkg.in/djherbis/times.v1
github.com/projectdiscovery/hmap/store/cache
github.com/akrylysov/pogreb/internal/errors
github.com/akrylysov/pogreb/internal/hash
github.com/akrylysov/pogreb/fs
github.com/ulikunitz/xz
github.com/akrylysov/pogreb - Embedded key-value store for read-heavy workloads written in Go
github.com/syndtr/goleveldb/leveldb/util
github.com/syndtr/goleveldb/leveldb/cache
github.com/syndtr/goleveldb/leveldb/comparer
github.com/mholt/archiver
github.com/syndtr/goleveldb/leveldb/storage
github.com/syndtr/goleveldb/leveldb/filter
github.com/syndtr/goleveldb/leveldb/opt
github.com/tidwall/pretty
github.com/tidwall/gjson
github.com/projectdiscovery/gologger/writer
github.com/syndtr/goleveldb/leveldb/errors
github.com/projectdiscovery/gologger
github.com/syndtr/goleveldb/leveldb/iterator
github.com/syndtr/goleveldb/leveldb/journal
go.etcd.io/bbolt
github.com/syndtr/goleveldb/leveldb/memdb
github.com/syndtr/goleveldb/leveldb/table
github.com/projectdiscovery/utils/maps
github.com/tidwall/grect
github.com/tidwall/buntdb - BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support 
github.com/projectdiscovery/goflags
github.com/syndtr/goleveldb/leveldb
github.com/projectdiscovery/mapcidr
github.com/projectdiscovery/stringsutil
go.uber.org/atomic
github.com/yl2chen/cidranger/net
github.com/Mzack9999/go-http-digest-auth-client
github.com/yl2chen/cidranger
go.uber.org/multierr
github.com/projectdiscovery/utils/reader
github.com/projectdiscovery/iputil
github.com/projectdiscovery/utils/errors
github.com/projectdiscovery/utils/ip
github.com/projectdiscovery/networkpolicy
github.com/projectdiscovery/utils/url
golang.org/x/text/transform
golang.org/x/text/unicode/bidi
golang.org/x/net/http2/hpack
golang.org/x/text/unicode/norm
github.com/projectdiscovery/retryabledns/hostsfile
golang.org/x/net/internal/socks
golang.org/x/text/secure/bidirule
github.com/projectdiscovery/hmap/store/disk
github.com/Masterminds/semver/v3
golang.org/x/net/proxy
github.com/mattn/go-runewidth
github.com/dlclark/regexp2/syntax
golang.org/x/net/idna
github.com/muesli/reflow/ansi
github.com/muesli/reflow/indent
github.com/muesli/reflow/padding
github.com/projectdiscovery/hmap/store/hybrid
github.com/muesli/reflow/wordwrap
github.com/aymanbagabas/go-osc52/v2
github.com/lucasb-eyer/go-colorful
golang.org/x/net/http/httpguts
golang.org/x/net/http2
github.com/projectdiscovery/ipranger
github.com/mattn/go-isatty
github.com/muesli/termenv
github.com/dlclark/regexp2
github.com/olekukonko/tablewriter
github.com/yuin/goldmark-emoji/definition
github.com/yuin/goldmark/util
github.com/alecthomas/chroma
github.com/projectdiscovery/retryablehttp-go - Package retryablehttp provides a familiar HTTP client interface with automatic retries and exponential backoff 
github.com/alecthomas/chroma/formatters/html
github.com/projectdiscovery/retryabledns/doh
github.com/alecthomas/chroma/formatters/svg
github.com/projectdiscovery/retryabledns
github.com/alecthomas/chroma/formatters
github.com/projectdiscovery/asnmap/libs
github.com/projectdiscovery/mapcidr/asn
github.com/alecthomas/chroma/styles
github.com/alecthomas/chroma/lexers/internal
github.com/alecthomas/chroma/lexers/b
github.com/alecthomas/chroma/lexers/a
github.com/alecthomas/chroma/lexers/p
github.com/alecthomas/chroma/lexers/j
github.com/alecthomas/chroma/lexers/d
github.com/alecthomas/chroma/lexers/e
github.com/alecthomas/chroma/lexers/c
github.com/alecthomas/chroma/lexers/f
github.com/alecthomas/chroma/lexers/i
github.com/alecthomas/chroma/lexers/k
github.com/alecthomas/chroma/lexers/h
github.com/alecthomas/chroma/lexers/l
github.com/alecthomas/chroma/lexers/n
github.com/alecthomas/chroma/lexers/o
github.com/alecthomas/chroma/lexers/q
github.com/alecthomas/chroma/lexers/r
github.com/alecthomas/chroma/lexers/t
github.com/alecthomas/chroma/lexers/circular
github.com/alecthomas/chroma/lexers/g
github.com/alecthomas/chroma/lexers/m
github.com/alecthomas/chroma/lexers/s
github.com/alecthomas/chroma/lexers/v
github.com/alecthomas/chroma/lexers/w
github.com/alecthomas/chroma/lexers/x
github.com/alecthomas/chroma/lexers/y
github.com/alecthomas/chroma/lexers/z
github.com/cheggaaa/pb/v3/termutil
github.com/mattn/go-colorable
github.com/google/go-querystring/query
github.com/fatih/color
golang.org/x/crypto/openpgp/errors
github.com/cheggaaa/pb/v3
golang.org/x/crypto/openpgp/armor
golang.org/x/crypto/cast5
golang.org/x/crypto/openpgp/elgamal
golang.org/x/crypto/openpgp/s2k
golang.org/x/crypto/openpgp/packet
github.com/alecthomas/chroma/lexers
github.com/alecthomas/chroma/quick
golang.org/x/sys/cpu
golang.org/x/crypto/pbkdf2
golang.org/x/crypto/scrypt
golang.org/x/crypto/blake2b
github.com/minio/selfupdate/internal/binarydist
github.com/minio/selfupdate/internal/osext
aead.dev/minisign
golang.org/x/oauth2/internal
github.com/minio/selfupdate
golang.org/x/oauth2
golang.org/x/crypto/openpgp
github.com/google/go-github/v30/github
github.com/yuin/goldmark/text
github.com/yuin/goldmark/ast
github.com/yuin/goldmark-emoji/ast
github.com/yuin/goldmark/extension/ast
github.com/yuin/goldmark/renderer
github.com/yuin/goldmark/parser
github.com/yuin/goldmark/renderer/html
github.com/charmbracelet/glamour/ansi
github.com/yuin/goldmark - A markdown parser written in Go. Easy to extend, standard(CommonMark) compliant, well structured. 
github.com/yuin/goldmark-emoji - An emoji extension for the goldmark markdown parser
github.com/yuin/goldmark/extension
github.com/charmbracelet/glamour
github.com/projectdiscovery/utils/update

I'm not very familiar to golang, so IDK why we really need all this.

Small utility program

ls -lh mapcidr

26M before strip -s, 20M after

Issue with sort / reverse sort option

Describe the bug
When -s or -sr flag is used, it looks like -a option is used internally, and sorting/reverse sorting is performed on the aggregated data, whereas default sorting and reverse sorting need to performed on the expanded IPs.

Mapcidr version
master

$ cat input.txt

1.1.1.1
8.8.8.8
255.255.255.255
2.2.2.2
2.4.4.4
2.4.3.2
9.9.9.9

Current:

cat input.txt | mapcidr -s

1.1.1.1/32
2.2.2.2/32
2.4.3.2/32
2.4.4.4/32
8.8.8.8/32
9.9.9.9/32
255.255.255.255/32

Expected:

cat input.txt | mapcidr -s

1.1.1.1
2.2.2.2
2.4.3.2
2.4.4.4
8.8.8.8
9.9.9.9
255.255.255.255

IP range input support (ipv4, ipv6)

Currently, IP or CIDR-based input is supported which can be extended to support range-based IP input which later can be utilized by other features like expansion/aggregation and other options.

Input: 192.168.0.1-192.168.0.255

echo 192.168.0.1-192.168.0.255 | mapcidr -a

192.168.0.1/32
192.168.0.2/31
192.168.0.4/30
192.168.0.8/29
192.168.0.16/28
192.168.0.32/27
192.168.0.64/26
192.168.0.128/25

Example is taken from https://www.ipaddressguide.com/cidr

image

[request] check if a specific ipaddress belongs to a specific cidr.

Hi guys,
first of all, thanks for this tool. I found it VERY useful.

I'm trying to lear Go by translating my python scripts to go and also improving dumb steps on my recon process.
With this in mind, I've identified a small step that can be translated to a go tool.

Here is some context:

I'm matching a domain-based recon against a ASN-based recon in order to filter which hosts are "hosted" inside an org.

In order to do that, I'm using mapcidr to expand each ipv4 prefix and save into a (500mb) file. The next step is to loop every single ip address found by dnsx/massdns and save all matches into a separated file. That "match" is done using grep, which is dumb and terrible.

My snippet

My snippet will loop each ipaddr found by dnsx, loop the prefixes, expand a single prefix at a time (might store on a slice or list to be faster in reuse). Then it should look if a specific ipaddress belongs to that prefix, if we have a match, just save that IP on a slice to handle it later.

Why I'm coming to you

This is a very useful feature that poses a critical step on my recon process and I believe it might help a lot of people.
My snippet won't be elegant, won't be the best way to do this, as I'm just taking my firsts steps into golang.
As this might be a simple update, would it be possible to add this feature in the core of mapcidr at some point in the future?

If you guys feel this won't be useful nor interesting enough, I'll understand.
Ty!

Rename asn test folder

Is your feature request related to a problem? Please describe.
Rename ASN test folder from goldenfiles to tests

CLI flag enhancement

Is your feature request related to a problem? Please describe.

Currently, single CIDR input is accepted using cidr flag and list of cidr can be provided using another flag list which can be combined into a single flag to accept single and file input.

   -cidr string          CIDR to process
   -l, -list string      File containing list of CIDRs to process
   -il, -ip-list string  File containing list of IPs to process

Describe the solution you'd like

  • Combine cidr and list CLI flag to accept single / file input
  • Update ip-list to accept single / file input
   -cl, -cidr string     CIDR / File containing list of CIDRs to process
   -il, -ip string       IP / File containing list of IPs to process

Create a global default asn.DefaultClient

Description

The package asn/asn.go should contain a DefaultClient and global helper functions using this default client. Similarly to net/http and projectdiscovery/gologger:

DefaultClient = New()
...
func GetCIDRsForASNNum(value string) ([]*net.IPNet, error) { ... }
func GetIPAddressesAsStream(value string) (chan string, error) { ... }

Maps CIDR Range of Not Specified Range in List

Describe the bug

cat test.txt

10.35.148.0/22
10.35.20.0/24

the test.txt file contains only subnet of 10.35.148 & 10.35.20

but when i ran this command

mapcidr -cidr test.txt 

It displays results of range that are not specified example output

10.35.150.252
10.35.150.253
10.35.150.254
10.35.150.255
10.35.151.0
10.35.151.1
10.35.151.2

From the above , it can be observed that 10.35.150 & 10.35.151 range is not in scope of the list that is supplied, but mapcidr displays those results as well

Mapcidr version
v1.0.3

Complete command you used to reproduce this

Screenshots
Add screenshots of the error for a better context.

IP Filter / Conversion does not work with CIDR / stdin input

Mapcidr version
dev

Describe the bug

Current:

echo 192.168.1.224/30 | mapcidr -t6 -silent

192.168.1.224
192.168.1.225
192.168.1.226
192.168.1.227

Expected:

echo 192.168.1.224/30 | mapcidr -t6 -silent

00:00:00:00:00:ffff:c0a8:01e0
00:00:00:00:00:ffff:c0a8:01e1
00:00:00:00:00:ffff:c0a8:01e2
00:00:00:00:00:ffff:c0a8:01e3

Note: Most possibly this is happening because IP filter / conversion flag expecting input from ip / il flag which is originally used for IP Match filter and set to deprecate with #48 to avoid the known confusion with flag name and it's feature, as such, cidr/cl needs to be used as only input source to feed IP/CIDR/LIST to process.

Rename test folder

Description

Rename cmd/intergration-test/goldenfiles folder to tests or assets

Count mismatch with count flag

Mapcidr version
dev

Complete command you used to reproduce this

mapcidr -cidr 15.181.232.0/21 -count -silent
2047

mapcidr -cidr 15.181.232.0/21 -silent | wc
2048

Discrepancy in ipv4 and ipv6 ips count

Match and Filter IP/CIDR support

Is your feature request related to a problem? Please describe.

For the cases, where we have known list of IPs/CIDRs that I wanted to match or filter to have desired list of IPs/CIDRs to process/scan as output and would be good addition for the project.

Describe the solution you'd like

This can be done by introducing 2 additional CLI flag to accept IP/CIDR to perform said action, for example:

   -mi, -match-ip string[]  IP/CIDR/FILE containing list of IP/CIDR to match (comma-separated, file input)
   -fi, -filter-ip string[] IP/CIDR/FILE containing list of IP/CIDR to filter (comma-separated, file input)

Match IP/CIDR

Display IPs from input that belongs to given IP/CIDR ranges.

$ mapcidr -cidr 192.168.1.0/24 -mi 192.168.1.253,192.168.1.252

192.168.1.252
192.168.1.253

More input examples:

$ mapcidr -cidr 192.168.1.0/24 -mi 192.168.1.0/25
$ mapcidr -cidr 192.168.1.0/24 -mi cidrs.txt

Filter IP/CIDR

Filter IPs from input that belongs to given IP/CIDR ranges.

$ mapcidr -cidr 192.168.1.224/28 -fi 192.168.1.233,192.168.1.234

192.168.1.224
192.168.1.225
192.168.1.226
192.168.1.227
192.168.1.228
192.168.1.229
192.168.1.230
192.168.1.231
192.168.1.232
192.168.1.235
192.168.1.236
192.168.1.237
192.168.1.238
192.168.1.239

More input examples:

$ mapcidr -cidr 192.168.1.0/24 -fi 192.168.1.0/25
$ mapcidr -cidr 192.168.1.0/24 -fi cidrs.txt

Note:

Currently available ip/il cli flag was introduced as part of the feature request in #4 to match the IPs, which is same as m/match feature, as such as per this change ip/il can be deprecated to avoid any possible confusion.

mapcidr uses too much RAM and crashes laptop

Describe the bug
echo 17.0.0.0/8 | mapcidr -silent uses up almost 1.5 GB RAM memory, and echo 17.0.0.0/2 | mapcidr -silent makes the computer unresponsive after consuming all of my 4 GB RAM. I'm submitting this as a bug report, since such high memory usage should have been documented well as a clear WARNING in the README.md, and even though I'm not very knowledgeable about mapcidr, I think such high memory requirement shouldn't be necessary for basic use as shown above.

Obviously some huge amount of dynamic memory allocation is happening inside the code.
Is that really required for expanding out the given input into IPs?
I would imagine it would be possible to simply print out the information directly without any memory allocation, or at least do it in batches so that memory requirement isn't so high.

Mapcidr version
v0.0.7

Complete command you used to reproduce this
echo 17.0.0.0/8 | mapcidr -silent and echo 17.0.0.0/2 | mapcidr -silent

Invalid "version" in v0.0.8

Describe the bug
Running mapcidr -version with v0.0.8 indicates wrong release:

mapcidr -version


                   ____________  ___    
  __ _  ___ ____  / ___/  _/ _ \/ _ \   
 /  ' \/ _ '/ _ \/ /___/ // // / , _/   
/_/_/_/\_,_/ .__/\___/___/____/_/|_| v0.0.8
          /_/                                                            

                projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
[INF] Current Version: v0.0.7

Make compatible with gologger >= 1.1.0

github.com/projectdiscovery/gologger was rewritten partially with breaking change wrt 1.0.1. It would be nice for mapcidr to be compatible with the latest version, so distro packagers don't have to use a compatibility package.

[Issue] Unable to install via go get missing package.

host@host:~/tools/awscidr$ sudo snap install go --classic
[sudo] password for host:
go 1.16.6 from Michael Hudson-Doyle (mwhudson) installed

Issues with 3rd party repo it seems

host@host:~/tools/awscidr$ GO111MODULE=on go get -v github.com/projectdiscovery/mapcidr/cmd/mapcidr
github.com/pierrec/lz4 (download)
package github.com/pierrec/lz4/v4: cannot find package "github.com/pierrec/lz4/v4" in any of:
	/usr/local/go/src/github.com/pierrec/lz4/v4 (from $GOROOT)
	/home/host/GO/src/github.com/pierrec/lz4/v4 (from $GOPATH)

Request for exclusion of .0 and .255 from IP generations

Is your feature request related to a problem? Please describe.
Scanning on .0 and .255 is not necessary to be scanned. It's theoretically possible to assign .0 IP. But it's rare in practical network. .255 is always a broadcast IP.

Describe the solution you'd like
I wish the tool provides a switch not to include .0 and .255 in output generation.

Thank you Team for simple yet awesome useful tool.

[Issue] Converting an IPv4 to decimal form returns the representation for the same address but in the IPv6 space

Describe the bug
It looks like that converting an IPv4 to decimal form returns the decimal representation for the same address in the IPv6 address space:

$ echo '127.0.0.1' | mapcidr -ip-format 5
..
[INF] Current mapcidr version v1.1.2 (latest)
281472812449793

As it should read 2130706433.

Mapcidr version
v1.1.2 on macOS 13.5.1

Complete command you used to reproduce this
echo '127.0.0.1' | mapcidr -ip-format 5

Screenshots
image

Implement test cases

Description

The repository should be covered by integration tests covering various option usages

when get to install it, give me blank terminal!

Describe the bug
i try to install the tool and nothing happen just give me a blank terminal, and try to do it manually by download it by git command rather than go command and it not work and give me an error
"go: no module declaration in go.mod. To specify the module path:go mod edit -module=example.com/mod"

Mapcidr version
latest

Complete command you used to reproduce this
this is NOT work
go install -v github.com/projectdiscovery/mapcidr/cmd/mapcidr@latest

Screenshots
Screenshot 05-22-2023 10 39 02

[Feature] deaggregate function

Hi

It would be nice to have Deaggregate function to convert first and last IP of range to slice of prefixes.
For example RIPE returns following json output:

"166.8.0.0/16",
"166.9.0.0-166.10.255.255",
"166.11.0.0/16",

So I need to convert IP range "166.9.0.0-166.10.255.255" to list of prefixes [166.9.0.0/16, 166.10.0.0/16]
It could be something like this:

func Deaggregate(first, last *net.IP) []*net.IPNet

thx

[Issue] Multiple bugs with AggregateApproxIPV4s and CoalesceCIDRs when used as library

There are multiple very unexpected results when working with mapcidr as a library. I assume this mostly won't normally happen on CLI runs (except item 1 in the list):

Describe the bug

  1. AggregateApproxIPV4s works incorrect when input contains masks (not always)
  2. AggregateApproxIPV4s modifies the input list (directly modifies ips[].IP)
  3. CoalesceCIDRs modifies the internal representation of the input array such that IPv4s are represented as 16-bytes long
  4. AggregateApproxIPV4s produces incorrect output when working directly on the result of CoalesceCIDRs due to item 3

Mapcidr version

  • v1.1.2
  • v1.1.16

Test file

package mapcidrtester

import (
	"net"
	"testing"

	"github.com/projectdiscovery/mapcidr"
	"github.com/stretchr/testify/assert"
)

// TestMaskedApproximation shows that mapcidr.AggregateApproxIPV4s is incorrectly treating input
// with a /24 mask, effectively reducing matched IPs compared to original
func TestMaskedApproximation(t *testing.T) {
	ipStrs := []string{
		"152.58.98.0/24",
		"152.58.98.5/32",
	}
	expected := []string{
		"152.58.98.0/24",
	}

	inputs := parseCIDRs(ipStrs)

	approximated := mapcidr.AggregateApproxIPV4s(inputs)
	assert.Equal(t, expected, ipNetsToStrs(approximated))
}

// TestApproximationInputModify shows that mapcidr.AggregateApproxIPV4s modifies the input value, thus
// making it unpredictable if value is later used by other code
func TestApproximationInputModify(t *testing.T) {
	ipStrs := []string{
		"10.10.0.0/31",
		"10.10.0.2/32",
		"10.10.0.255/32",
		"10.10.1.0/31",
		"10.10.1.2/32",
		"10.10.1.255/32",
	}
	expected := []string{
		"10.10.0.0/24",
		"10.10.1.0/24",
	}

	inputs := parseCIDRs(ipStrs)

	approximated := mapcidr.AggregateApproxIPV4s(inputs)
	assert.Equal(t, expected, ipNetsToStrs(approximated))

	assert.Equalf(t, parseCIDRs(ipStrs), inputs, "AggregateApproxIPV4s shouldn't modify inputs")
}

// TestApproximateCoalesced show that running mapcidr.AggregateApproxIPV4s on the result of mapcidr.CoalesceCIDRs
// produces an unexpected result, even though CoalesceCIDRs didn't actually modify the eventual "string" values.
// This is further demonstrated in TestUseCoalesced
func TestApproximateCoalesced(t *testing.T) {
	ipStrs := []string{
		"10.10.0.0/31",
		"10.10.0.2/32",
		"10.10.0.255/32",
		"10.10.1.0/31",
		"10.10.1.2/32",
		"10.10.1.255/32",
	}
	expected := []string{
		"10.10.0.0/24",
		"10.10.1.0/24",
	}

	// convert to IPNet masks
	inputs := parseCIDRs(ipStrs)

	coalesced, _ := mapcidr.CoalesceCIDRs(inputs)
	assert.Equalf(t, parseCIDRs(ipStrs), inputs, "CoalesceCIDRs shouldn't modify inputs")

	approximated := mapcidr.AggregateApproxIPV4s(coalesced)
	assert.Equal(t, expected, ipNetsToStrs(approximated))
}

// TestUseCoalesced shows that mapcidr.CoalesceCIDRs modifies the internal representation of IPv4 to be
// 16-bytes long, which affects how mapcidr.AggregateApproxIPV4s works, causing incorrect results, as
// demonstrated in TestApproximateCoalesced
func TestUseCoalesced(t *testing.T) {
	// these are already coalesced values, no further coalescing is needed
	ipStrs := []string{
		"10.10.0.0/31",
		"10.10.0.2/32",
		"10.10.0.255/32",
		"10.10.1.0/31",
		"10.10.1.2/32",
		"10.10.1.255/32",
	}
	expectedApprox := []string{
		"10.10.0.0/24",
		"10.10.1.0/24",
	}

	// convert to IPNet masks
	inputs := parseCIDRs(ipStrs)
	t.Logf("Inputs (%d): %s", len(inputs), inputs)

	// compact
	coalesced, _ := mapcidr.CoalesceCIDRs(inputs)
	assert.Equalf(t, ipStrs, ipNetsToStrs(coalesced), "CoalesceCIDRs should produce the same string output as was the input")
	assert.Equalf(t, inputs, coalesced, "CoalesceCIDRs should keep the same internal values in IPv4s")

	approximated := mapcidr.AggregateApproxIPV4s(coalesced)
	assert.Equal(t, expectedApprox, ipNetsToStrs(approximated))
}

func parseCIDRs(inp []string) []*net.IPNet {
	out := make([]*net.IPNet, len(inp))
	for i, pfx := range inp {
		_, mask, err := net.ParseCIDR(pfx)
		if err != nil {
			panic(err)
		}
		out[i] = mask
	}
	return out
}

func ipNetsToStrs(inp []*net.IPNet) []string {
	out := make([]string, len(inp))
	for i, ip := range inp {
		out[i] = ip.String()
	}
	return out
}

[Feature] -quiet -q -silent flags to match/filter logic

Is your feature request related to a problem? Please describe.
Similar to grep usage

echo 'test' | grep 'te' -q && echo 'te found'

add quiet to just return true/false exit status

       -q, --quiet, --silent
              Quiet; do not write anything to standard output.  Exit immediately with zero status  if  any  match  is
              found, even if an error was detected.  Also see the -s or --no-messages option

#48
Describe the solution you'd like

mapcidr -cidr 192.168.1.0/24 -m 192.168.1.0/25 -q && echo ipfound

[Issue] IPv6 approximate aggregation is broken

Describe the bug

$ echo "2001:db8:eb5a:1c8a:7fd8:fc5f:3c71:b3f6
2001:db8:eb5a:2d79:4ba2:3f33:aaa5:b78
2001:db8:eb5a:2e91:a11b:39df:37a7:2d44
2001:db8:eb5a:2fab:944e:a1f0:3825:1cc1
2001:db8:eb5a:3c2b:db3b:44db:77b:a8d3
2001:db8:eb5a:3f14:a08:5787:43e5:54b2
2001:db8:eb5a:4b17:e65b:547c:589d:f580
2001:db8:eb5a:4b43:7813:4492:3430:88fe" \
| /usr/local/bin/mapcidr -aa -silent

Produces

2001:db8:eb5a:1c8a:7fd8:fc5f:3c71:b3f6/128

Normal aggregation behaves correctly:

$ echo "2001:db8:eb5a:1c8a:7fd8:fc5f:3c71:b3f6
2001:db8:eb5a:2d79:4ba2:3f33:aaa5:b78
2001:db8:eb5a:2e91:a11b:39df:37a7:2d44
2001:db8:eb5a:2fab:944e:a1f0:3825:1cc1
2001:db8:eb5a:3c2b:db3b:44db:77b:a8d3
2001:db8:eb5a:3f14:a08:5787:43e5:54b2
2001:db8:eb5a:4b17:e65b:547c:589d:f580
2001:db8:eb5a:4b43:7813:4492:3430:88fe" \
| /usr/local/bin/mapcidr -a -silent

Produces

2001:db8:eb5a:1c8a:7fd8:fc5f:3c71:b3f6/128
2001:db8:eb5a:2d79:4ba2:3f33:aaa5:b78/128
2001:db8:eb5a:2e91:a11b:39df:37a7:2d44/128
2001:db8:eb5a:2fab:944e:a1f0:3825:1cc1/128
2001:db8:eb5a:3c2b:db3b:44db:77b:a8d3/128
2001:db8:eb5a:3f14:a08:5787:43e5:54b2/128
2001:db8:eb5a:4b17:e65b:547c:589d:f580/128
2001:db8:eb5a:4b43:7813:4492:3430:88fe/128

Mapcidr version
Include the version of mapcidr you are using, mapcidr -version


                   ____________  ___    
  __ _  ___ ____  / ___/  _/ _ \/ _ \   
 /  ' \/ _ '/ _ \/ /___/ // // / , _/   
/_/_/_/\_,_/ .__/\___/___/____/_/|_|
          /_/                                                     	 

		projectdiscovery.io

[INF] Current Version: v1.1.16

Screenshots

Not applicable

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.