simonw / mbox-to-sqlite

Load email from .mbox files into SQLite

mbox-to-sqlite

Load email from .mbox files into SQLite

Installation

Install this tool using pip:

pip install mbox-to-sqlite

Usage

Use the mbox command to import a .mbox file into a SQLite database:

mbox-to-sqlite mbox emails.db path/to/messages.mbox

You can try this out against an example containing a sample of 3,266 emails from the Enron corpus like this:

curl -O https://raw.githubusercontent.com/ivanhb/EMA/master/server/data/mbox/enron/mbox-enron-white-s-all.mbox
mbox-to-sqlite mbox enron.db mbox-enron-white-s-all.mbox

You can then explore the resulting database using Datasette:

datasette enron.db

Development

To contribute to this tool, first checkout the code. Then create a new virtual environment:

cd mbox-to-sqlite
python -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest

mbox-to-sqlite's Issues

Consider normalizing To/From/Cc etc contents

The To: and Cc: headers are currently stored as text:

mbox-to-sqlite/tests/test_mbox_to_sqlite.py

Lines 35 to 37 in 9c1e65a

	"To": "[email protected], [email protected], [email protected]",
	"Subject": "12/28 Power VaR",
	"Cc": "[email protected], [email protected], [email protected], \n\t[email protected]",

Maybe normalize these and extract them out nto many-to-many tables?

This would also need to handle Name <[email protected]> formatted data.

Initial tool design

Tool for importing mbox files into SQLite using the Python mailbox standard library module.

Prototype here: https://gist.github.com/simonw/f442dc725aaad9a92af3850ad4c41839

Parse dates into a more useful format

mbox-to-sqlite/tests/test_mbox_to_sqlite.py

Line 33 in 9c1e65a

"Date": "Fri, 29 Dec 2000 03:49:00 -0800 (PST)",

If this was converted to ISO 8601 this column could be sorted by date.

Need to decide how to handle timezones too.

Handle HTML emails and other attachments

This has only been tested against plain text emails so far.

I think HTML emails should have the HTML stashed in a separate column.

Not sure how best to handle other attachments though, especially attachments that might be images included in the HTML.

Should also think about XSS attacks and suchlike: should this tool run HTML through https://bleach.readthedocs.io/ or should tha be left as an exercise for the user?

Failing with sqlite3.OperationalError: duplicate column name

Hey,
interesting project! I tried to give it a spin, but the mbox file I'm trying to analyze throws me this error:

Traceback (most recent call last):
  File "/usr/local/bin/mbox-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/mbox_to_sqlite/cli.py", line 33, in mbox
    db[table].upsert_all(to_insert(), alter=True, pk="Message-ID")
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 3183, in upsert_all
    return self.insert_all(
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 3082, in insert_all
    self.create(
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1574, in create
    self.db.create_table(
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 974, in create_table
    self.execute(sql)
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 485, in execute
    return self.conn.execute(sql)
sqlite3.OperationalError: duplicate column name: Message-Id