dogsheep / google-takeout-to-sqlite

Save data from Google Takeout to a SQLite database

Dogsheep

Dogsheep is a collection of tools for personal analytics using SQLite and Datasette.

Big internet companies know a lot about us. By exporting that data back out of them we can see what they know and maybe learn something interesting about ourselves.

Read more about Dogsheep on my blog: simonwillison.net/tags/dogsheep

Watch Personal Data Warehouses: Reclaiming Your Data for a demo of Dogsheep in action.

Dogsheep tools

These tools, maintained by the Dogsheep project, let you export your data into a SQLite database for further analysis.

• twitter-to-sqlite uses the Twitter API or Twitter's "export your data" zip archive to create a database of your tweets, followers, people you follow, favourited tweets and much more besides.
• healthkit-to-sqlite converts your Apple HealthKit export into a database, with details of your heart rate, workouts, step count and more.
• swarm-to-sqlite imports your checkin history from Foursquare Swarm.
• apple-notes-to-sqlite saves Apple Notes data to a database.
• inaturalist-to-sqlite imports the wildlife and plants you have spotted on iNaturalist.
• google-takeout-to-sqlite imports data from Google Takeout, which can include your Google Maps location history, your search history on Google and more to come.
• genome-to-sqlite imports your genome from 23andMe into a SQLite database.
• github-to-sqlite imports repositories you have created or starred on GitHub.
• pocket-to-sqlite imports articles you have saved using Pocket.
• hacker-news-to-sqlite imports stories and comments posted to Hacker News.
• dogsheep-photos imports metadata about photos from Apple Photos, including machine learning labels. See Using SQL to find my best photo of a pelican according to Apple Photos.
• dogsheep-beta is a search index which combines data from all of the other sources into a single faceted search interface.
• evernote-to-sqlite converts exported Evernote files in the ENEX format into a SQLite database.

Tools by other developers

These tools help bring the Dogsheep philosophy to life.

• beeminder-to-sqlite by Ben Congdon saves your goals and data points from Beeminder.
• goodreads-to-sqlite by Tobias Kunze imports your reading history from Goodreads.
• pinboard-to-sqlite by Jacob Kaplan-Moss saves your bookmarks from Pinboard.
• todoist-to-sqlite by Ben Congdon saves your tasks and projects from Todoist.
• parkrun-to-sqlite by Mark Woodbridge imports your parkruns.

google-takeout-to-sqlite's Issues

sqlite3.OperationalError: no such table: main.my_activity

Hello,
When i run the command google-takeout-to-sqlite my-activity db.db takeout-20220203T174446Z-001.zip, i get this error :

Traceback (most recent call last):
  File "c:\users\julie\appdata\local\programs\python\python39-32\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\julie\appdata\local\programs\python\python39-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\julie\AppData\Local\Programs\Python\Python39-32\Scripts\google-takeout-to-sqlite.exe\__main__.py", line 7, in <module>
  File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\google_takeout_to_sqlite\cli.py", line 31, in my_activity
    utils.save_my_activity(db, zf)
  File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\google_takeout_to_sqlite\utils.py", line 19, in save_my_activity
    db["my_activity"].create_index(["time"])
  File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\sqlite_utils\db.py", line 629, in create_index
    self.db.conn.execute(sql)
sqlite3.OperationalError: no such table: main.my_activity

Thank you for your help
Sorry for my bad English
EDIT: i used the json format

Add more details on how to request data from google takeout correctly.

The default is to download everything. This can result in an enormous amount of data when you only really need 2 types of data for now:

• My Activity
• Location History

In addition unless you specify that "My Activity" is downloaded in JSON format the default is HTML. This then causes the

google-takeout-to-sqlite my-activity takeout.db takeout.zip

command to fail as it only contains html files not json files.

Thanks

killed by oomkiller on large location-history

memory seems to grow unbounded and is oom-killed after about 20GB memory usage.

this is happening while loading a ~1GB uncompressed location history.

KeyError: 'accuracy' when processing Location History

I'm new to both the dogsheep tools and datasette but have been experimenting a bit the last few days and these are really cool tools!

I encountered a problem running my Google location history through this tool running the latest release in a docker container:

Traceback (most recent call last):
  File "/usr/local/bin/google-takeout-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/cli.py", line 49, in my_activity
    utils.save_location_history(db, zf)
  File "/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py", line 27, in save_location_history
    db["location_history"].upsert_all(
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1105, in upsert_all
    return self.insert_all(
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 990, in insert_all
    chunk = list(chunk)
  File "/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py", line 33, in <genexpr>
    "accuracy": row["accuracy"],
KeyError: 'accuracy'

It looks like the tool assumes the accuracy key will be in every location history entry.

My first attempt at a local patch to get myself going was to convert accessing the accuracy key to a .get instead to hopefully make the row nullable but I wasn't quite sure what sqlite_utils would do there. That did work in that the import happened and so I was going to propose a patch that made that change but in updating the existing test to include an entry with a missing accuracy entry, I noticed the expected type of the field appeared to be changing to a string in the test (and from a quick scan through the sqlite_utils code, probably TEXT in the database). Given this change in column type, it seemed that opening an issue first before proposing a fix seemed warranted. It seems the schema would need to be explicitly specified if you wanted a nullable integer column.

Now that I've done a successful import run using my initial fix of calling .get on the row dict, I can see with datasette that I only have 7 data points (out of ~250k) that have a null accuracy column. They are all from 2011-2012 in an import that includes points spanning ~2010-2016 so perhaps another approach might be to filter those entries out during import if it really is that infrequent?

I'm happy to provide a PR for a fix but figured I'd ask about which direction is preferred first.

location history changes

not sure if each download is unique, but I had to change some things to work with the takeout zip I made 2023-01-25

filename changed from "Location History.json" to "Records.json"

"timestampMs" is not present, "timestamp" is roughly iso timestamp

def get_timestamp_ms(raw_timestamp):
    try:
        return datetime.datetime.strptime(raw_timestamp, "%Y-%m-%dT%H:%M:%SZ").timestamp()
    except ValueError:
        return datetime.datetime.strptime(raw_timestamp, "%Y-%m-%dT%H:%M:%S.%fZ").timestamp()

def save_location_history(db, zf):
    location_history = json.load(
        zf.open("Takeout/Location History/Records.json")
    )
    db["location_history"].upsert_all(
        (
            {
                "id": id_for_location_history(row),
                "latitude": row["latitudeE7"] / 1e7,
                "longitude": row["longitudeE7"] / 1e7,
                "accuracy": row["accuracy"],
                "timestampMs": get_timestamp_ms(row["timestamp"]),
                "when": row["timestamp"],
            }
            for row in location_history["locations"]
        ),
        pk="id",
    )


def id_for_location_history(row):
    # We want an ID that is unique but can be sorted by in
    # date order - so we use the isoformat date + the first
    # 6 characters of a hash of the JSON
    first_six = hashlib.sha1(
        json.dumps(row, separators=(",", ":"), sort_keys=True).encode("utf8")
    ).hexdigest()[:6]
    return "{}-{}".format(
        row['timestamp'],
        first_six,
    )

example locations from mine

{
    "latitudeE7": 427220206,
    "longitudeE7": -923423972,
    "accuracy": 10,
    "deviceTag": -1312429967,
    "deviceDesignation": "PRIMARY",
    "timestamp": "2019-01-08T23:31:50.867Z"
  }

{
    "latitudeE7": 427011317,
    "longitudeE7": -923448300,
    "accuracy": 5,
    "deviceTag": -1312429967,
    "deviceDesignation": "PRIMARY",
    "timestamp": "2019-01-08T23:33:53Z"
  }, 

sqlite-utils error on takeout import

$ google-takeout-to-sqlite my-activity takeout.db /path/to/zip
...
sqlite3.OperationalError: no such table: main.my_activity

there is no table create in utils.py, unlike other importers such as github-to-sqlite

additionally, this package and hackernews-to-sqlite have conflicting sqlite-utils dep with datasette and dogsheep-beta

Upgrade to latest sqlite-utils

This is pinned to v1 at the moment.

Feature Request: Gmail

From takeout, I only exported my Gmail account. Ideally I could parse this into sqlite via this tool.