thephm / email_md

IMAP email to Markdown

email_md

Tool to pull IMAP emails and convert them to atomic Markdown files.

Dependencies

The code in this repo relies heavily on my message_md classes which contain generic Message, Person, Group and other classes and the methods to convert messages to Markdown files. Be sure to read the README and the configuration guide for that repo first.

Also relies on markdownify so do this:

pip install markdownify

Refer to your email provider's help pages to find out how to get an application password. If they require two-factor authentication (2FA), then this tool won't work for you.

Gmail

To setup an application password for Gmail, visit Sign in with app passwords

For the IMAP server use: imap.gmail.com

Fastmail

Here are the steps to setup an application password for Fastmail.

1. Click "Settings" then "Privacy and Security"
2. Click "New App Password"
3. Give it a name like "email_md"
4. Choose "Mail, Contacts, & Calendars" (or just "IMAP")
5. Click "Generate Password"
6. Under "Your new password for email_md is:" copy that password
7. Put the password somewhere secure like a password keeper, I use BitWarden

Figure out the IMAP server address and port.

For Fastmail it's imap.fastmail.com and Port 993.

Setting up the config files

The next step is to configure this tool.

email server and account

In the config.json file, set the following fields. For demonstration purposes, I've used Fastmail's settings:

In this example, emails from only two folders would be fetched: INBOX and Sent Items:

    "imap-server": "imap.fastmail.com",
    "email-folders": "INBOX;Sent Items",
    "not-email-folders": "",
    "email-account": "993",

To use all email folders and exclude specific ones (and their sub-folders), use the not-email-folders setting:

    "imap-server": "imap.fastmail.com",
    "email-folders": "",
    "not-email-folders": "Spam;Shopping;Trash;Health",
    "email-account": "993",

Which can also be overriden on the command line:

Command line	Alternate	Description
-i	--imap	IMAP server address
-r	--folders	IMAP folders to retrieve from
-e	--email	email address to retrieve from

People and groups

You'll need to define each person that you communicate with in people.json and the groups in groups.json. This way the tool can associate each message with the person that sent it and who it was sent to.

Samples of these configuration files are in the message_md repo upon which this tool depends.

This part is tedious the first time and needs to be updated when you add new contacts, i.e. a pain.

Using email_md

Once you've configured the tool and the people.json file is setup, you're ready to run the tool.

The command line options are described in the message_md repo.

Example:

python3 email_md.py -c ../../dev-output/config -s ../../dev-output -o ../../dev-output -d -e [email protected] -p lifeisahighway2! -i imap.fastmail.com -m spongebob -x 20 -b 2024-01-01

where:

• config settings are in ../../dev-output/config
• source folder is ../../dev-output
• output the Markdown files to ../../dev-output
• debug messages are enabled
• email address is [email protected]
• password for the email is lifeisahighway2!
• email server is imap.fastmail.com
• my slug is spongebob
• maximum of 20 messages should be converted
• begin the export from 2024-01-01

After you've used it

The script is REALLY slow because it goes through every email just to determine if it's sent/received since the -b begin date. It took 30 mins or more to run on my laptop. After I run the script, I now archive all the messages to a big folder that is not included in the scanning.

email_md's Issues

Need to support multiple email addresses for the same person

Only reads from the INBOX folder right now

Even though there are options for folder, it currently ignores those and uses INBOX folder.

If attachments folder doesn't exist, attachments are not saved

Seeing errors like this:

ERROR:root:[Errno 2] No such file or directory: '../../dev-output/attachments/Flyer Invite - Fall 2023.pdf'
ERROR:root:[Errno 2] No such file or directory: '../../dev-output/attachments/Toronto-20110904-00018.jpg'

because I had renamed the attachments folder. The script should create it if it doesn't exist

Broke the script because the message_md went snake case

I had changed everything to snake_case in message_md and other scripts so need to modify this one.