simsong / mailtools

Tools for working with email: sending bulk mail, extracting email from MacOS address books, and analyzing mail

Mail Tools

Right now, this git repository is where we are developing code to process and analyze mail.

What's here?

• misc -- Miscellaneous tools, including:
  • The autoresponder.py.
  • [imaptool.py(misc/imaptool.py) --- for

Using with GMAIL

1. Request an App Password https://support.google.com/mail/answer/185833?hl=en
2. Access GMAIL through imap.gmail.com https://developers.google.com/gmail/imap/imap-smtp

Functionality we want

• Basic retrieval tools:

  • List email addresses
  • Dates for sender
  • Dates for sender, and # of emails sent/received per day

• Analytics:

  • Identify cliques
  • Draw a map of what times during the day a person sends and reads mail.
  • Identify where a person is located.

• parse all iCal entries in mail

• LDA on email?

• Top emails every week.

• Report breaks in mail file. (Days on which mail was not received)

• Heatmap of when mail is sent

  • weekday / weekend
  • day of week

• Topic modeling of received messages

  • how many topics?
  • mapping of users to specific topics
  • identify users who email on same topics but do not email each other...?

• Plug-in to Autopsey

Implementation notes

• Metadata is stored in an sqlite3 databsae.
• The schema we use is the same as used by Apple mail client
  • Makes mail tool work with Apple Mail out of the box.
  • Schema is well developed, stored in schema.sql
• Apple AddressBook groups senders together
  • On non-apple systems, need a tool to do this.

See Also

• Email Maining Toolkit, Java: https://github.com/hjast/NLPWorkspace

• See also https://github.com/mihaip/mail-trends.git

• Identify people who always CC you a dn askt them to stop.

• Identify email sending times. He map through the day, week.

• tool for assigning a title to a topic model?

mailtools's Issues

Replicate functionality in email toolkit.

autoresponder: use amazon's SMTP server?

Literature Review

Find all academic articles involving mail analysis and link to them in the repo.

Implement digest maker tests

• Create a test mailbox with username & password credentials
• Create a config.ini.dist template
• Create a set of email messages and send them to the test mailbox
• Implement a program to download the messages from the test mailbox to a local mailbox.
• Add the local messages to the repo as a set of test files
• Implement a program to upload the messages from the local mailbox to the test mailbox.
• Create a py.test fixture that clears the mailbox and uploads the test messages.
• Create a test for the py.test fixture.

With this infrastructure, each person working on the mailtools can create their own test mailbox and they won't impact on each other.

implement DB for autresponder

• have autoresponder store form in DB, not in CSV.
• Send out a link that people click on to see results.
• Make link work just once, but allow people to request another link.

Create plug-in for Autopsy

Develop a list of analytics and visualizations, then implement them.

Risks Digest 31.85

Arrange the messages in your preferred order and click [submit]

Message subject lines takes from Risks Digest 31.85:

• A Case for Cooperation Between Machines and Humans (John Markoff)
• Scammers steal > $100m in Wash. State unemployment fraud (Seattle Times)
• Satellites and spacecraft malfunction as Earth's magnetic field mysteriously weakens (Sky)
• Microsoft: Beware this massive phishing campaign using malicious Excel macros to hack PCs (ZDNet)
• Ransomware deploys virtual machines to hide itself from antivirus software (ZDNet)
• Students are failing AP tests because the College Board can't handle iPhone photos (The Verge)
• How Do Astronauts Escape When a Space Launch Goes Wrong? (WiReD)
• How a Chinese AI Giant Made Chatting -- and Surveillance -- Easy (WiReD)
• 90-Day Security Plan Progress Report: May 20 (Zoom Blog)
• How the CDC is misreporting COVID-19 testing data (The Atlantic)
• Re: COVID codebase [D Maziuk)
• Re: The ultimate Turing test (Arthur Flatau)
• Re: Teen Hacker and Crew of Evil Geniuses Accused of $24 Million Crypto
• Theft (Gabe Goldberg)
• Re: The FBI Just Unlocked an iPhone Without Apple's Help (Keith Medcalf)
• Re: AI gets the attention, but biotechnology is poised to change the world (Dan Jacobson)
• Abridged info on RISKS (comp.risks)

distinguish between original messages and followups

• who knows who?
• who sends email to multiple people?
• Who do my friends know best that I can see?