ripe-and-apnic-db-ip-parser

Introduction

ripe-db-ip-parser is a tool designed to parse the RIPE database of IP address assignments (ripe.db.inetnum) and import the data into a SQLite database for easy querying and analysis. This tool also provides flexibility for users to create their custom parsers to generate JSON or any other schema/database format.

Releases:

You will find all db(sql,mmdb) in the repository release section , do you need to use the repo only if you need to coastumize something

Download latest DB's

To download latest `SQLITE_DB`:

    wget -O "./ripe_data.db.zip" "https://github.com/midoelhawy/ripe-db-ip-parser/releases/latest/download/db.sqlite.zip"
    unzip -o ".ripe_data.db.zip" -d "./"
    rm "./ripe_data.db.zip"

To download latest `MMDB`:

    wget -O "./ripe_data.db.zip" "https://github.com/midoelhawy/ripe-db-ip-parser/releases/latest/download/ASN_COUNTRY_AND_CITY.mmdb
"
    unzip -o ".ripe_data.db.zip" -d "./"
    rm "./ripe_data.db.zip"

see Releases

How to Use

Prerequisites

Before using the tool, ensure that you have the following prerequisites installed:

Python 3.x
pip (Python package manager)

Setup

Clone the repository to your local machine:

git clone https://github.com/midoelhawy/ripe-and-apnic-db-ip-parser.git

Navigate to the project directory:
```
cd ripe-db-ip-parser
```
Install the required Python packages:
```
pip install -r requirements.txt
```

Parsing the RIPE Database

NOTE: you can run all process's in one step directly by running this command ./scripts/parse_all_ripe_db_and_generate_mmdb.sh

download Ripe and apnic DB by running the following script:
```
./scripts/download-ripe-data.sh
```
Run the SQL generator to import the parsed data into a SQLite database:
```
python3 sqllite_importer.py
```

Custom Parser

You can also write your custom parser to generate JSON or another type of schema/database format. Follow these steps:

Create a file named myCustomParser.py.

Paste the following code into myCustomParser.py:

from pathlib import Path
from lib.db import SQLiteHandler
from lib.ripe_parser import RIPE_PARSER

if __name__ == "__main__":
    default_ripe_data = str(Path.joinpath(Path(__file__).parents[0],'db/ripe.db.inetnum')) # PUT HERE YOUR FILE PATH
    def on_single_block_process(block):
        pass
        # THIS FUNCTION WILL BE EXECUTED WITH EVERY BLOCK OF IPS 
        # YOU CAN PUT IT TO A JSON FILE FOR EXAMPLE 

    RIPE_PARSER.parse_file(default_ripe_data, on_single_block_process)
    print("Done")

Customize the on_single_block_process function to handle each IP block as per your requirements.
Run the myCustomParser.py script to execute your custom parsing logic.

How to generate `MMDB` database

NOTE: YOU NEED SQLITE DB TO GENERATE MMDB

install and configure go-lang
Run cd ./scripts && go run generate_mmdb.go

midoelhawy / ripe-and-apnic-db-ip-parser Goto Github PK