palinkas-jo-reggelt / geolite2sql Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 6.0 1.53 MB

Import MaxMinds GeoLite2 data into MySQL

PowerShell 100.00%

geolite2sql's People

Contributors

Stargazers

Watchers

Forkers

lcamilo rvdhout gotspatel ethicalsecurity-agency fitear alexlaforge

geolite2sql's Issues

IPv6

Hi palinka,

Would it be possible to also include IPv6 addresses in the import? (for now i am only using the GeoLite2-Country database)

Column 'geoname_id' cannot be null and Out of range value for column 'metro_code' at row 59301

I'm using the command:

powershell ".\GeoLite2SQL.ps1 city"

Here is my log:

::: MaxMind GeoCity Database Import Routine Tuesday, April 23, 2024 :::
 
4/23/2024 12:12:36 AM : GeoIP City Update Start
4/23/2024 12:12:36 AM : ----------------------------
4/23/2024 12:12:36 AM : Checking for script update at GitHub
4/23/2024 12:12:36 AM : [ERROR] Version test failed : Could not obtain local install version information
4/23/2024 12:12:36 AM : ----------------------------
4/23/2024 12:12:36 AM : Deleting old files
4/23/2024 12:12:36 AM : No old files to delete
4/23/2024 12:12:36 AM : ----------------------------
4/23/2024 12:12:36 AM : Downloading MaxMind data
4/23/2024 12:13:38 AM : MaxMind data successfully downloaded in 1 minute 2 seconds
4/23/2024 12:13:41 AM : MaxMind data successfully unzipped in 2 seconds
4/23/2024 12:13:41 AM : Locations CSV successfully renamed
4/23/2024 12:13:41 AM : ----------------------------
4/23/2024 12:13:41 AM : Counting database records for comparison
4/23/2024 12:13:41 AM : 0 database records prior to starting update
4/23/2024 12:13:41 AM : ----------------------------
4/23/2024 12:13:41 AM : Counting CSV records for comparison
4/23/2024 12:13:50 AM : Counted 3,807,222 records in new IPv4 & IPv6 CSVs in 8 seconds
4/23/2024 12:13:50 AM : ----------------------------
4/23/2024 12:13:50 AM : Converting CSVs to hex-range
4/23/2024 12:13:58 AM : City IPv4 CSV successfully converted to hex-range in 7 seconds
4/23/2024 12:14:02 AM : City IPv6 CSV successfully converted to hex-range in 4 seconds
4/23/2024 12:14:02 AM : ----------------------------
4/23/2024 12:14:02 AM : Drop and recreate database tables
4/23/2024 12:14:02 AM : Database tables successfully dropped and created
4/23/2024 12:14:02 AM : ----------------------------
4/23/2024 12:14:02 AM : Import converted CSVs to database
4/23/2024 12:14:02 AM : [ERROR] DATABASE ERROR : Unable to run query : 
				LOAD DATA INFILE 'C:\\GeoLite2SQL\\Script-Created-Files\\GeoLite2-City-CSV\\GeoCityIPv4.csv'
				INTO TABLE geocity
				FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 ROWS
				(@network_start_hex, @network_last_hex, @geoname_id, @registered_country_geoname_id, @represented_country_geoname_id, @is_anonymous_proxy, @is_satellite_provider, @postal_code, @latitude, @longitude, @accuracy_radius)
				SET 
					network_start = UNHEX(@network_start_hex),
					network_end = UNHEX(@network_last_hex),
					geoname_id = NULLIF(@geoname_id, ''),
					registered_country_geoname_id = NULLIF(@registered_country_geoname_id, ''),
					represented_country_geoname_id = NULLIF(@represented_country_geoname_id, ''),
					is_anonymous_proxy = NULLIF(@is_anonymous_proxy, ''),
					is_satellite_provider = NULLIF(@is_satellite_provider, ''),
					postal_code = NULLIF(@postal_code, ''),
					latitude = NULLIF(@latitude, ''),
					longitude = NULLIF(@longitude, ''),
					accuracy_radius = NULLIF(@accuracy_radius, '');
			 
Exception calling "Fill" with "2" argument(s): "Column 'geoname_id' cannot be null"



4/23/2024 12:14:02 AM : [OK] City IPv4 data imported in less than 1 second
4/23/2024 12:14:03 AM : [ERROR] DATABASE ERROR : Unable to run query : 
				LOAD DATA INFILE 'C:\\GeoLite2SQL\\Script-Created-Files\\GeoLite2-City-CSV\\GeoCityIPv6.csv'
				INTO TABLE geocity
				FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 ROWS
				(@network_start_hex, @network_last_hex, @geoname_id, @registered_country_geoname_id, @represented_country_geoname_id, @is_anonymous_proxy, @is_satellite_provider, @postal_code, @latitude, @longitude, @accuracy_radius)
				SET 
					network_start = UNHEX(@network_start_hex),
					network_end = UNHEX(@network_last_hex),
					geoname_id = NULLIF(@geoname_id, ''),
					registered_country_geoname_id = NULLIF(@registered_country_geoname_id, ''),
					represented_country_geoname_id = NULLIF(@represented_country_geoname_id, ''),
					is_anonymous_proxy = NULLIF(@is_anonymous_proxy, ''),
					is_satellite_provider = NULLIF(@is_satellite_provider, ''),
					postal_code = NULLIF(@postal_code, ''),
					latitude = NULLIF(@latitude, ''),
					longitude = NULLIF(@longitude, ''),
					accuracy_radius = NULLIF(@accuracy_radius, '');
			 
Exception calling "Fill" with "2" argument(s): "Column 'geoname_id' cannot be null"



4/23/2024 12:14:03 AM : [OK] City IPv6 data imported in less than 1 second
4/23/2024 12:14:04 AM : [ERROR] DATABASE ERROR : Unable to run query : 
			LOAD DATA INFILE 'C:\\GeoLite2SQL\\Script-Created-Files\\GeoLite2-City-CSV\\GeoLocations.csv'
			INTO TABLE citylocations
			FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 ROWS
			(@geoname_id, @locale_code, @continent_code, @continent_name, @country_iso_code, @country_name, @subdivision_1_iso_code, @subdivision_1_name, @subdivision_2_iso_code, @subdivision_2_name, @city_name, @metro_code, @time_zone, @is_in_european_union)
			SET 
				geoname_id = NULLIF(@geoname_id, ''), 
				locale_code = NULLIF(@locale_code, ''), 
				continent_code = NULLIF(@continent_code, ''), 
				continent_name = NULLIF(@continent_name, ''), 
				country_code = NULLIF(@country_iso_code, ''), 
				country_name = NULLIF(@country_name, ''), 
				subdivision_1_iso_code = NULLIF(@subdivision_1_iso_code, ''), 
				subdivision_1_name = NULLIF(@subdivision_1_name, ''), 
				subdivision_2_iso_code = NULLIF(@subdivision_2_iso_code, ''), 
				subdivision_2_name = NULLIF(@subdivision_2_name, ''), 
				city_name = NULLIF(@city_name, ''), 
				metro_code = NULLIF(@metro_code, ''), 
				time_zone = NULLIF(@time_zone, ''), 
				is_in_european_union = NULLIF(@is_in_european_union, '');
		 
Exception calling "Fill" with "2" argument(s): "Out of range value for column 'metro_code' at row 59301"



4/23/2024 12:14:04 AM : [OK] City name data imported in 1 second
4/23/2024 12:14:05 AM : ----------------------------
4/23/2024 12:14:05 AM : [ERROR] record count mismatch:
4/23/2024 12:14:05 AM :  records imported to database
4/23/2024 12:14:05 AM : 3,807,222 records in MaxMind CSV
4/23/2024 12:14:05 AM : Completed update in 1 minute 28 seconds
4/23/2024 12:14:05 AM : GeoIP update finished

::: MaxMind GeoCity Database Import Routine Monday, April 22, 2024 :::

email message body had:

GeoIP City update Start: 4/22/2024 11:59:36 PM
 
[ERROR] Version check failed
[OK] MaxMind data downloaded
[OK] MaxMind data unzipped
[OK] Converted IPv4 City block CSV
[OK] Converted IPv6 City block CSV
[OK] Database tables dropped & recreated
[ERROR] Count mismatched by 3807222 records
 records imported to database
3,807,222 records in MaxMind CSV
Completed update in 1 minute 24 seconds
 
GeoIP update finish: 4/23/2024 12:01:01 AM

HTTP Error 304

`
Mode LastWriteTime Length Name

-a---- 02/03/2023 04:36 PM 0 DebugLog.log
-a---- 02/03/2023 04:36 PM 0 EmailBody.txt
02/03/2023 04:36:56 PM : GeoIP Update Start
02/03/2023 04:36:56 PM : ----------------------------
02/03/2023 04:36:56 PM : Checking for script update at GitHub
02/03/2023 04:37:00 PM : ----------------------------
02/03/2023 04:37:00 PM : Deleting old files
02/03/2023 04:37:00 PM : ----------------------------
02/03/2023 04:37:00 PM : Downloading MaxMind data
02/03/2023 04:37:08 PM : [ERROR] : Unable to download MaxMind data : HTTP status 304: The server's response was not valid. Th
e server was not following the defined protocol. Resume the job, and then Background Intelligent Transfer Service (BITS) will
try again.

02/03/2023 04:37:08 PM : [ERROR] : Quitting Script
02/03/2023 04:37:08 PM : GeoIP update finished
`

Getting This Error. Any Pointer what may be wrong, tried from three different locations but same error.

Unable to download MaxMind data.

Hi,
I've registered for a MaxMind key and have entered it correctly in the config file along with all other parameters.
Running the script for the first time, I'm getting this error continously.

2/29/2024 5:11:47 PM : Downloading MaxMind data
2/29/2024 5:11:47 PM : [ERROR] : Unable to download MaxMind data on try # 1
2/29/2024 5:11:47 PM : [ERROR] : Error Message : The request was aborted: Could not create SSL/TLS secure channel.

Any help would be much appreciated!
Thanks

Feature Update

Hi @palinkas-jo-reggelt

Thank you for wonderful script this and others.

I have updated your script to cover below,

Download and Populate ASN data also (just in case someone wants it)
Email From will now shows "GeoIP CITY Update [email protected]", "GeoIP COUNTRY Update [email protected]", "GeoIP ASN Update [email protected]", (just to differentiate and sort in inbox)
Import Query loads data without any error when used "LOAD DATA LOCAL INFILE '$strFileLocName'" - FROM HERE

How to send to you the updated files, Don't know much about github except downloading ;-)

Now just figuring how to run this query to get the data in all columns, (currently it shows ASN and ISP data only)

SET @IP := '1.1.1.1';
SELECT
	continent_code AS `Continent Code`, 
	continent_name AS Continent, 
	country_code AS `Country Code`, 
	country_name AS Country, 
	subdivision_1_name AS Province, 
	subdivision_2_name AS `Sub Region`, 
	city_name AS City, 
	autonomous_system_number AS ASN, 
	autonomous_system_organization AS ISP, 
	time_zone AS `Time Zone`
FROM
	geocity
	LEFT JOIN
	citylocations
	ON 
		geocity.geoname_id = citylocations.geoname_id
	LEFT JOIN
	geocountry
	ON 
		geocountry.geoname_id = citylocations.geoname_id,
	geoasn
WHERE
	INET6_ATON( @IP ) <= geocity.network_end AND
	geocity.network_start <= INET6_ATON( @IP ) AND
	INET6_ATON( @IP ) <= geoasn.network_end AND
	geoasn.network_start <= INET6_ATON( @IP )
LIMIT 1

Incorrectly named variables

GeoLite2SQL-Config.ps1

$LocationLang = "en"

GeoLite2SQL-Country.ps1

$CountryLocations = "$PSScriptRoot\GeoLite2-Country-CSV\GeoLite2-Country-Locations-$LocationLanguage.csv"

same in GeoLite2SQL-City.ps1

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble