luka1199 / geo-heatmap

:world_map: Generate an interactive geo heatmap from your Google location data

Geo Heatmap

This is a script that generates an interactive geo heatmap from your Google location history data using Python, Folium and OpenStreetMap.

Getting Started

1. Install Python 3+

If you don't already have Python 3+ installed, grab it from https://www.python.org/downloads/. You'll want to download install the latest version of Python 3.x. As of 2019-11-22, that is Version 3.8.

2. Get Your Location Data

Here you can find out how to download your Google data: https://support.google.com/accounts/answer/3024190?hl=en
Here you can download all of the data that Google has stored on you: https://takeout.google.com/

To use this script, you only need to select and download your "Location History", which Google will provide to you as a JSON file by default. KML is also an output option and is accepted for this program.

You can also import GPS Exchange Format (GPX) files, e.g. from a GPS tracker.

3. Clone This Repository

On https://github.com/luka1199/geo-heatmap, click the green "Clone or Download" button at the top right of the page. If you want to get started with this script more quickly, click the "Download ZIP" button, and extract the ZIP somewhere on your computer.

4. Install Dependencies

In a command prompt or Terminal window, navigate to the directory containing this repository's files. Then, type the following, and press enter:

pip install -r requirements.txt

5. Run the Script

In the same command prompt or Terminal window, type the following, and press enter:

python geo_heatmap.py <file> [<file> ...]

Replace the string <file> from above with the path to any of the following files:

• The Location History.json JSON file from Google Takeout
• The Location History.kml KML file from Google Takeout
• The takeout-*.zip raw download from Google Takeout that contains either of the above files
• A GPS Exchange Format (GPX) file

Examples:

Single file:

python geo_heatmap.py C:\Users\Testuser\Desktop\Records.json

python geo_heatmap.py "C:\Users\Testuser\Desktop\Location History.json"

python geo_heatmap.py Records.json

Multiple files:

python geo_heatmap.py Records.json locations.kml takeout.zip

Using the stream option (for users with Memory Errors):

python geo_heatmap.py -s Records.json

Set a date range:

python geo_heatmap.py --min-date 2017-01-02 --max-date 2018-12-30 Records.json

Advanced heatmap settings:

python geo_heatmap.py -z 15 -r 12 -b 7 -mo 0.3 -mz 20 Records.json

Usage:

usage: geo_heatmap.py [-h] [-o OUTPUT] [--min-date YYYY-MM-DD]
                      [--max-date YYYY-MM-DD] [-s] [--map MAP] [-z ZOOM_START]
                      [-r RADIUS] [-b BLUR] [-mo MIN_OPACITY] [-mz MAX_ZOOM]
                      file [file ...]

positional arguments:
  file                  Any of the following files:
                        - Your location history JSON file from Google Takeout
                        - Your location history KML file from Google Takeout
                        - The takeout-*.zip raw download from Google Takeout
                        that contains either of the above files
                        - A GPX file containing GPS tracks

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        Path of heatmap HTML output file.
  --min-date YYYY-MM-DD
                        The earliest date from which you want to see data in the heatmap.
  --max-date YYYY-MM-DD
                        The latest date from which you want to see data in the heatmap.
  -s, --stream          Option to iteratively load data.
  --map MAP, -m MAP     The name of the map tiles you want to use.
                        (e.g. 'OpenStreetMap', 'StamenTerrain', 'StamenToner', 'StamenWatercolor')
  -z ZOOM_START, --zoom-start ZOOM_START
                        The initial zoom level for the map. (default: 6)
  -r RADIUS, --radius RADIUS
                        The radius of each location point. (default: 7)
  -b BLUR, --blur BLUR  The amount of blur. (default: 4)
  -mo MIN_OPACITY, --min-opacity MIN_OPACITY
                        The minimum opacity of the heatmap. (default: 0.2)
  -mz MAX_ZOOM, --max-zoom MAX_ZOOM
                        The maximum zoom of the heatmap. (default: 4)

6. Review the Results

The script will generate a HTML file named heatmap.html. This file will automatically open in your browser once the script completes. Enjoy!

FAQ

I'm getting an "Out of Memory" error or MemoryError when I try to run the script. What's going on?

Your LocationHistory.json file is probably huge, and Python is running out of memory when the script tries to parse that file.

To fix this, download and install the 64-bit version of Python. To do this:

1. Go to python.org.
2. Click the link corresponding to your OS next to "Looking for Python with a different OS?"
3. Click the "Latest Python 3 Release" link.
4. Scroll down to "Files".
5. Click to download the x64 release. For example, on Windows, that's the "Windows x86-64 executable installer" link.
6. Install!

If this does not fix the issue you can use the stream option:

python geo_heatmap.py -s <file>

This will be slower but will use much less memory to load your location data.

I'm getting a SyntaxError when running pip install -r requirements.txt or python geo_heatmap.py <file>. What am I doing wrong?

You are probably using the python interpreter to run these commands. Try to run them in cmd.exe or Windows PowerShell (Windows) or the Terminal (Linux, MacOS).

I'm getting the error message TypeError: __init__() got an unexpected keyword argument 'max_value'. What can I do to fix this?

Try to run:

pip uninstall progressbar
pip install progressbar2

You probably have progressbar installed, which uses maxval as an argument for __init__. Progressbar2 uses max_value.

geo-heatmap's Issues

Load data from multiple location data files

As suggested here: https://www.reddit.com/r/dataisbeautiful/comments/e2zv4w/i_wrote_a_python_script_that_generates_a_geo/f91pjr9?utm_source=share&utm_medium=web2x

Google Photos Integration - Suggestion

Looking at the generated map has brought back a lot of good memories from trips. It may not make sense in this project, but have you considered adding photos from Google Photos onto the map using their locations? Exploring cities you've traveled to and seeing the photos you took could be a fun experience.

Add option to stream data for users with memory problems

Progressbar won't be useable with this and it would be way slower.

Error: list indexes must be integers or slides, not strings

I get the following error message when running this script:

|##############################################################################################################################|100% ETA:  00:00:00
Traceback (most recent call last):
  File "geo_heatmap.py", line 222, in <module>
    generator.run(data_file, output_file, date_range)
  File "geo_heatmap.py", line 177, in run
    self.loadKMLData(data_file, date_range)
  File "geo_heatmap.py", line 93, in loadKMLData
    if self.timestampInRange(loc['timestampMs'], date_range):
TypeError: list indices must be integers or slices, not str

Input KML file:

<kml xmlns='http://www.opengis.net/kml/2.2' xmlns:gx='http://www.google.com/kml/ext/2.2'>
 <Document>
  <Placemark>
   <open>1</open>
   <gx:Track>
    <altitudeMode>clampToGround</altitudeMode>
    <when>2013-12-22T11:38:47Z</when>
    <gx:coord>-1.1 52.1 0</gx:coord>
   </gx:Track>
  </Placemark>
 </Document>
</kml>

Any idea what might cause this?

Carriage return in script throws error on Linux

I ran the python script on a Pixelbook and received this error:

/usr/bin/env: ‘python3\r’: No such file or directory

I fixed this by removing the carriage returns:
cat geo_heatmap.py | tr -d "\r" > geo_heatmap_noCR.py

Add option to specify a time span for the location data

--min-date=yyyy-mm-dd
--max-date=yyyy-mm-dd

Can't finish generating.

Getting this in the console.

Loading data from Location History.json...
|##################################################################################################|100% Time:  0:00:01
Traceback (most recent call last):
  File "geo_heatmap.py", line 85, in <module>
    generator.run(data_file, output_file)
  File "geo_heatmap.py", line 73, in run
    self.loadData(data_file)
  File "geo_heatmap.py", line 30, in loadData
    lat = round(loc["latitudeE7"] / 1e7, 6)
KeyError: 'latitudeE7'

kml file does not open properly

Describe the bug
I have both GPX and KML files for the same activity from Garmin connect (attached)

To Reproduce
Steps to reproduce the behavior:
Executing python geo_heatmap.py activity_5972548926.klm opens the browser in coordinates 0:0

Executing python geo_heatmap.py activity_5972548926.gpx opens the browser with heatmap in the center

Software version - commit 7c12365
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec 7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)] on win32
Chrome version 87.0.4280.66 (Official Build) (64-bit)
Win 10

10X

Import CSV or Json simple

Hello, excellent project, I tested it with the data that I was able to download from Google takeout and it works perfectly.

I would like to be able to use your project to try to graph the route of a GPS tracker, from which I can only obtain LAT, LONG and Date/Time.

I tried to replicate the format of the "Records.json" file but still can't get it to work.

Here below I am going to leave an example of the json file that I want to make work with this project, could you look at it and tell me what is wrong?
Greetings

{ "locations": [ { "latitudeE7": -3472408, "longitudeE7": -5851444, "source": 110144, "deviceTag": 110144, "accuracy": 33, "timestamp": "2022-04-01T10:50:01.269Z" }, { "latitudeE7": -3472407, "longitudeE7": -5851443, "source": 110144, "deviceTag": 110144, "accuracy": 33, "timestamp": "2022-04-01T10:50:05.269Z" }, { "latitudeE7": -3472407, "longitudeE7": -5851443, "source": 110144, "deviceTag": 110144, "accuracy": 33, "timestamp": "2022-04-01T10:52:05.269Z" } ] }

It would be interesting to be able to use it with csv or json files with a simple format that contains LAT, LONG and date/time

I leave it as a suggestion, but I would appreciate help with this problem

Sorry for my English.

Frederic from Argentina

JSON file has no `timestampMs` field

Describe the bug
Downloaded location data file contains timestamp field instead of timestampMs field.

$ python3 geo_heatmap.py ~/Downloads/takeout-20220203T164644Z-001/Takeout/Historique\ des\ positions/Records.json -s

(1/3) Loading data from /home/theo/Downloads/takeout-20220203T164644Z-001/Takeout/Historique des positions/Records.json

Traceback (most recent call last):
  File "/home/theo/git/geo-heatmap/geo_heatmap.py", line 303, in <module>
    generator.run(data_file, output_file, date_range, stream_data, settings)
  File "/home/theo/git/geo-heatmap/geo_heatmap.py", line 232, in run
    self.streamJSONData(json_file, date_range)
  File "/home/theo/git/geo-heatmap/geo_heatmap.py", line 75, in streamJSONData
    if timestampInRange(loc["timestampMs"], date_range):
KeyError: 'timestampMs'

To Reproduce
Steps to reproduce the behavior:

1. Get your location data at https://takeout.google.com/
2. Run the script

Expected behavior
Normal generation of heatmap.html file.

Screenshots
Content of Records.json file

$ head -20  ~/Downloads/takeout-20220203T164644Z-001/Takeout/Historique\ des\ positions/Records.json 
{
  "locations": [{
    "latitudeE7": 413522827,
    "longitudeE7": 6672174,
    "accuracy": 20,
    "source": "WIFI",
    "deviceTag": 1611787950,
    "timestamp": "2013-12-30T20:30:26.396Z"
  }, {
    "latitudeE7": 417153171,
    "longitudeE7": 75560909,
    "accuracy": 736,
    "source": "CELL",
    "deviceTag": 1611787950,
    "timestamp": "2014-01-01T07:44:41.819Z"
  }, {
    "latitudeE7": 417153171,
    "longitudeE7": 75560909,
    "accuracy": 736,`
    "source": "CELL",

The file does not contain any timestampMs field :

$ grep -ri timestampMs ~/Downloads/takeout-20220203T164644Z-001/Takeout/Historique\ des\ positions/
$

Desktop:

• OS: Ubuntu 21.04

ValueError: Custom tiles must have an attribution.

Describe the bug
Folium throws an exception due to a missing field:

(2/3) Generating heatmap
Traceback (most recent call last):
  File "/home/savvasdalkitsis/geo-heatmap/geo_heatmap.py", line 302, in <module>
    generator.run(data_file, output_file, date_range, stream_data, settings)
  File "/home/savvasdalkitsis/geo-heatmap/geo_heatmap.py", line 245, in run
    m = self.generateMap(settings)
  File "/home/savvasdalkitsis/geo-heatmap/geo_heatmap.py", line 192, in generateMap
    m = folium.Map(location=self.max_coordinates,
  File "/home/savvasdalkitsis/.local/lib/python3.9/site-packages/folium/folium.py", line 288, in __init__
    tile_layer = TileLayer(tiles=tiles, attr=attr,
  File "/home/savvasdalkitsis/.local/lib/python3.9/site-packages/folium/raster_layers.py", line 111, in __init__
    raise ValueError('Custom tiles must have an attribution.')
ValueError: Custom tiles must have an attribution.

Desktop (please complete the following information):

• OS: Windows

Could not locate runnable browser

Hello, I currently have Firefox dev, Firefox, and Chrome on my computer and I was given the error:

Traceback (most recent call last):
  File "geo_heatmap.py", line 251, in <module>
    if not isTextBasedBrowser(webbrowser.get()):
  File "/usr/lib/python3.6/webbrowser.py", line 51, in get
    raise Error("could not locate runnable browser")
webbrowser.Error: could not locate runnable browser

I this might be a problem because I am running Windows Subsystem for Linux, making the installing paths of my applications a little weird. There is an option in webbrowser that allows for registration of a browser and it accepts a hard-coded path for the browser, but I don't see this as working generally enough for everbody.

Add option to print out stats about your data after loading

• Amount of location points
• Amount of countries with location points

There is no file named locations.json in the google download

$ uname -a
Linux kihikihi 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ python3 --version
Python 3.8.10

$ python3 geo_heatmap.py ~/Projects/Trips/Takeout/Location\ History/Semantic\ Location\ History/2022/2022_AUGUST.json

(1/3) Loading data from /home/jsibert/Projects/Trips/Takeout/Location History/Semantic Location History/2022/2022_AUGUST.json
Traceback (most recent call last):
File "geo_heatmap.py", line 327, in
generator.run(data_file, output_file, date_range, stream_data, settings)
File "geo_heatmap.py", line 258, in run
self.loadJSONData(json_file, date_range)
File "geo_heatmap.py", line 57, in loadJSONData
first_element = data["locations"][0]
KeyError: 'locations'

Make it possible to import output from Google MyMaps (KML / KMZ / GPS)

Desired solution
I wish to import items / locations I exported from a custom Google MyMaps. I created the pins by uploading a csv with addresses in the format of street city country to Google MyMaps that were correctly parsed and are shown on the Google MyMaps Map. Then, I downloaded the points through this dialogue, trying KML and KMZ format:

I wish I could pass the export files along to be executed by the Python script to have a heat map generated. I also used this converter to convert the data to gpx: https://mygeodata.cloud/conversion.

Errors encountered
When trying to pass the respective export file, I got the respective errors:

1. Converted .gpx file and kml file: Execution was successful, but the map was empty
2. kmz file: NotImplementedError: Unsupported file extension for 'International_Addresses.kmz'

Is there any workaround to get the script working for single points in Google MyMaps?
Many thanks for your help!

Non English formatted google services data

When I ran the script and pointed to the .zip archive with the location data, I received the following error:

Loading data from ../data/takeout-20191130T173344Z-001.zip...
Reading location data file from zip archive: 'Takeout/Locatiegeschiedenis/Locatiegeschiedenis.json'
Traceback (most recent call last):
File "./geo_heatmap2.py", line 179, in
generator.run(data_file, output_file)
File "./geo_heatmap2.py", line 135, in run
self.load_zip_data(data_file)
File "./geo_heatmap2.py", line 87, in load_zip_data
self.loadJSONData(read_file)
File "./geo_heatmap2.py", line 33, in loadJSONData
data = json.load(json_file)
File "/usr/lib/python3.5/json/init.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.5/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'bytes'

I think this is because geo_heatmap.py assumes a folder structure in the archive that uses English terms. If your Google services language is set to another language, these paths don't work out.

I was able to run geo_heatmap.py by first unzipping the archive and directly pointing to the .json file.

2 Google Accounts

Is there anyway to have it process 2 .json files. or could someone help me with the structure so I can combine the .json files from 2 separate google accounts?

Optimizations for loading KML data

Are there optimization possibilities for loading data from a KML file, because for me loading is very slow compared to loading with a JSON file?

Combine multiple location data files

It would be nice to be able to call the script with multiple files and merge the timestamps all together.

Still getting MemoryError after installing a 64bit version

Does it mean I need a more powerful computer?

Animation Option

The location data contains a timestamp, it would be cool if we could generate a day-by-day (or week-by-week or more options) animation of the heatmap, although I'm not sure how this would fit into the OpenStreetMap, but at the very least it would be interesting to see it if it could stitch them together as a gif.

Memory Error when Loading JSON Data

Traceback (most recent call last):
File "geo_heatmap.py", line 260, in
generator.run(data_file, output_file, date_range, tiles)
File "geo_heatmap.py", line 207, in run
self.loadJSONData(json_file, date_range)
File "geo_heatmap.py", line 88, in loadJSONData
data = json.load(json_file)
File "C:\Users\pgroy\Python\Python38-32\lib\json_init_.py", line 293, in load
return loads(fp.read(),
File "C:\Users\pgroy\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
MemoryError

C:\Users\pgroy\Python\Python38-32\geo-heatmap>python geo_heatmap.py LocationHistory.json
Loading data from LocationHistory.json...
Traceback (most recent call last):
File "geo_heatmap.py", line 260, in
generator.run(data_file, output_file, date_range, tiles)
File "geo_heatmap.py", line 207, in run
self.loadJSONData(json_file, date_range)
File "geo_heatmap.py", line 88, in loadJSONData
data = json.load(json_file)
File "C:\Users\pgroy\Python\Python38-32\lib\json_init_.py", line 293, in load
return loads(fp.read(),
File "C:\Users\pgroy\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
MemoryError

When running under WSL, heatmap.html opens in w3m browser

The line that opens heatmap.html in the default browser doesn't work when running the program in the Windows Subsystem For Linux (WSL). This is because the default browser in WSL is w3m, a text-only browser.

This is the line that causes the issue: https://github.com/luka1199/geo-heatmap/blob/master/geo_heatmap.py#L83

It would be nice if the output didn't automatically open in a browser, but instead notifies over standard output what file was created.

Here's a screenshot of what the result looks like:

Create a Flask App for easier interaction

WIP. All data will be processed instantly without including a SQL or any such database to store any form of data.

Add OpenStreetMap copyright

Suggested by https://www.reddit.com/r/dataisbeautiful/comments/e2zv4w/i_wrote_a_python_script_that_generates_a_geo/f924sza?utm_source=share&utm_medium=web2x

https://www.openstreetmap.org/copyright/

Script not compatible with the latest Takeout zip structure

I downloaded my location data today and tried to run the script but I'm getting the following exception:

python geo_heatmap.py takeout-20230101T215713Z-001.zip

(1/3) Loading data from takeout-20230101T215713Z-001.zip
Traceback (most recent call last):
  File "/Users/davidmigloz/repos/geo-heatmap/geo_heatmap.py", line 327, in <module>
    generator.run(data_file, output_file, date_range, stream_data, settings)
  File "/Users/davidmigloz/repos/geo-heatmap/geo_heatmap.py", line 252, in run
    self.loadZIPData(data_file, date_range)
  File "/Users/davidmigloz/repos/geo-heatmap/geo_heatmap.py", line 173, in loadZIPData
    (data_path,) = fnmatch.filter(
ValueError: not enough values to unpack (expected 1, got 0)

The Takeout folder structure doesn't seem to match the one the script is expecting:

geo-heatmap/geo_heatmap.py

Line 175 in fc25368

"Takeout/{name}/{name}.*".format(name=name))

JSONDecodeError

File "geo-heatmap/geo_heatmap.py", line 104, in <module> generator.run(data_file, output_file) File "geo-heatmap/geo_heatmap.py", line 77, in run self.loadData(data_file) File "geo-heatmap/geo_heatmap.py", line 28, in loadData data = json.load(json_file) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 296, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

TypeError: __init__() got an unexpected keyword argument 'max_value'

Followed the steps, had to install folium and progreessbar, using python 3.7 on an anaconda env

python geo_heatmap.py Standortverlauf.json
Loading data from Standortverlauf.json...
Traceback (most recent call last):
  File "geo_heatmap.py", line 111, in <module>
    generator.run(data_file, output_file)
  File "geo_heatmap.py", line 83, in run
    self.loadData(data_file)
  File "geo_heatmap.py", line 34, in loadData
    widgets=w) as pb:
TypeError: __init__() got an unexpected keyword argument 'max_value'

Remove personal information

Probably worth mentioning that you have stored your own location history in a few of the commits. I'd recommend following these steps: https://help.github.com/en/github/authenticating-to-github/removing-sensitive-data-from-a-repository

This can be personal information and would be best not to end up in the wrong hands!

Add argparse for arguments and features in the future

progressbar error whilst streaming json file

once the json file has finished streaming, i immediately get the following error:
Value 2528098 is out of range, should be between 0 and 2528097.328244275
at pb.update(i) line 86