Last updated 2024-08-14 14:51:05 -0700
- 1. Questions
- 2. Purpose
- 3. The Problem
- 4. Possible Solutions
- 5. Tool Choices
- 6. Usage-Scenario Challenges in Priority-First Order
- 6.1. Which MUA(s)?
- 6.2. Support old-style IMAP-folder paradigm
- 6.3. Support IMAP-account separation
- 6.4. Initial tagging
- 6.5. Moving msgs after applying tags
- 6.6. Viewing HTML-formatted messages
- 6.7. MUA folder-based searching with name auto-completion
- 6.8. Piping email-message content through shell commands
- 6.9. Mapping keyboard shortcuts to any command
- 6.10. IMAP folders as tags
- 6.11. Address-book/contacts integration
- 6.12. Highlight folders with unread or new emails
- 6.13. Sync Notmuch tags with maildir flags
- 6.14. Managing message attachments
- 6.15. Automatically tagging new messages (after initial tagging)
- 6.16. Offline editing - postponing drafts
- 6.17. Writing HTML-formatted messages
- 6.18. Forwarding messages with attachments
- 7. Appendix 1: Purpose: More Details
- 8. Appendix 2: More Problem Details
- 9. Appendix 3: General Setup Guides
Here’s the most-important questions, given the purpose and the problem:
-
Tools. Am I accurately assessing all available tools?
-
User scenarios. What are the best tools, or trade-offs of each tool, for each user scenario?
-
Workflows. Do you know a better workflow than what’s presented for any tool or usage scenario?
I want to transition my email environment from a classic setup to a tag-and-index-based system (most likely Notmuch- or mu-based), and I’m seeking help.
-
Near-term. I’m seeking significant feedback on this document from the Notmuch, mu, and related user communities. Most importantly, I seek answers to these questions.
-
Any written feedback works, including: email-list replies, issue posts, git forks plus pull requests. Please signal if you’d like write access to the repo for this document.
-
-
Longer-term. This document might be reusable by others, including future new users, if someone matures it moving forward. (In such case, the tone would change from being centered on my problems to a more-general-purpose document.) I couldn’t find similar content on the web. Please advise if I missed something.
With the right system, I’m hopeful I can make email management a much-less-time-consuming burden on my work… and my life.
DISCLAIMER: this document may be in flux as I learn and attempt to master this domain.
Why write this doc instead of just trudging through the tool testing myself? Read on.
See Appendix 1: Purpose: More Details for more.
I find email classification, recall, and translation to work/task tracking to be much more laborious than it should be. I expect the computer to be faster than me, and tailor to my preferred workflow. Instead, I’m constantly waiting for the computer, or forced to type or mouse-click inefficiently (I’m a vim-loving keyboard jockey), and/or experiencing a poor workflow.
More specifically:
-
I’m seeking a better paradigm over my current, IMAP-folder-only, high-capacity email management.
-
Among other things, moving and copying emails to a variety of IMAP folders--in order to "classify" said emails—gets cumbersome.
-
Searching my all my IMAP folders (many of them quite large) with an MUA like Thunderbird takes impossibly too long, and in many cases, isn’t powerful or granular enough to get the job done.
-
Lack of support for several of these user scenarios makes general email management much more cumbersome.
-
Thunderbird (TB) in general is slow, sluggish, and generally annoying in many ways in part because it’s a GUI-only interface, but TB annoys beyond just the GUI-only problem—including it’s struggles to handle a large number of IMAP folders. Other MUA’s like Outlook, Postbox (a TB variant), Mail.app, and Evolution are either worse (than TB, for me) or unavailable in my environment.
-
-
My existing environment
-
At least 4300 IMAP-based email folders with 1MM (million) email messages served via multiple IMAP servers.
-
Primary MUA = Mozilla Thunderbird 17.0.11
-
Mac OS 10.9.5 (Mavericks)
-
See Appendix 2: More Problem Details and user scenarios for more context and requirements.
(An aside: why title this document "search-based email"? Suggestion from the community. It seems better than "tag-based email," my original title. I’ve also considered "index-based email," but "search" seems a bit more descriptive than "index," and less esoteric. I’m open to any title that best groks with the community.)
After reviewing currently-available tools, I’m seeking:
-
fast search on my entire email-message/IMAP collection (my "email database"),
-
to leverage search/index/tag-based email categorization, and
-
provide easy extensibility with Linux and MacOSX command-line "primitives" or other, custom scripts/commands.
It’s fair to ask: did I come up with the above "requirements" by a) adjusting to features from currently-available tools, or b) by independently dreaming up my own specifications? More the former/(a). I’d much prefer Jarvis be my email interface, but he’s not currently (or at least not economically) available.
My investigation thus far suggests the implementation path hinges on choosing 1 of the following 2 applications, as they seem to mutually-exclusively represent the best (or at least most-popular) of the core of email-message indexing and tagging tool suites:
Is this assessment accurate? What other tools/options might I be overlooking?
My comparison analysis:
-
Initial tests show Notmuch performing approximately 15 times faster than mu.
-
Question: were these tests configured and executed correctly? The performance difference is remarkable, generating concerns about correct application setup, environment.
-
Performance optimizations:
-
As of 2015-07-14, I’ve not yet retested with above optimizations.
-
-
mu can embed its metadata (tags, etc) "natively" into the IMAP content/messages. Notmuch can not. However, muchsync (maybe other tools?) can replicate this metadata, but it takes additional process+infrastructure.
-
#1 greatly outweighs #2. Because of this, Notmuch "wins" (with me), pending the retest results of #1 (above) and additional feedback from others.
Notmuch seems to work best (or maybe requires?) the Maildir format. The following tools (presumably) all sync an IMAP server to a Maildir filesystem.
I’ve currently chosen mbsync, aka isync.
-
I’ve used mbsync more than any other tool listed here, and it’s thus far working nicely.
-
Search "mbsync vs offlineimap" to see more.
-
I understand getmail the least. It’s less referenced (on the web) for this usage/context than either offlineimap or mbsync. Why is this? Is it not a viable alternative to the above? getmail’s website seems to primarily (?) pitch it as a fetchmail replacement.
-
A seemingly-popuplar reference: "OfflineIMAP sucks," 2012-08-30
(I’ve not yet started this implementation.)
-
afew best?
-
See Initial tagging and Automatically tagging new messages (after initial tagging) for more.
(I’ve not yet started this implementation.)
-
mbsync-watcher
-
my take: it’s good for client→server updates, and not vice versa
-
Problem: I do not want it to sync all my 4k+ folders every 5 minutes, as that’s too much overhead. Hopefully there’s a way to disable this.
-
https://github.com/athoune/imapidle + some of my own Python scripting, which I’m hopeful will not be difficult.
-
mswatch
-
requires IMAP-server-side shell access - difficult if not impossible to get for all my IMAP accounts.
-
this might also be a client→server option
-
wrapping
imapidle
with my own Python script that triggersmbsync
seems like a better, more-flexible alternative
Given the problem, the work to find and master the best/better MUA(s) for me and concurrently learn a new search-and-tag-based email classification paradigm seems like my biggest challenge.
A dark-horse candidate: notmuch.el, an Emacs front-end.
-
-
seems to be the most-popular MUA in this space
-
https://raw.githubusercontent.com/karelzak/mutt-kz/master/README.notmuch
-
-
-
alot looks tremendously promising, possibly my best long-term solution, especially given my user profile (namely I’m a vim user and a Python programmer—seems to mirror well). However, the available documentation/resources are far more sparse than say mutt-kz. The user-manual content is almost impeccable, and pazz seems to do a great job to stay on top of all issues and offer a professional solution. For example, I significantly appareciate the up-front, informationally-dense, bulleted feature list at the top of the alot README.
-
However, it’s thus far been hard to find practical resources like example config files, procedural setup, etc. Maybe this is due in part because it’s not yet as popular, or caters to a user base more willing to spend time learning/configuring/tinkering with one tool, or something else?
-
Speculating: a hopefully-small effort to provide setup + config-file examples might go a long way to solve this problem, and boost alot’s "new user uptake" populartiy.
-
-
-
Emacs front-end(s)
-
Since I’m a vim user, I’ve initially shied away an Emacs-based front-end. However, I’ve now seen it referenced multiple times as a potentially-uniquely-powerful interface.
-
Questions: Does notmuch.el provide enough uniquely-powerful capability to merit special consideration? And if so, as a vim user will I find special difficulty trying to learn an Emacs-based paradigm?
-
-
vim front-end for Notmuch
-
http://git.notmuchmail.org/git/notmuch/blob/HEAD:/vim/README
-
I’m a heavy vim user, and while this approached seemed initially appealing, it’s feature depth seems small enough that I haven’t yet attempted to run this application—but to be fair, I haven’t "dug deep." Am I overlooking a powerful (in comparison to the others) tool by not vetting this vim front-end further?
-
-
There’s other frontends…
-
…but none seem as appealing to me as the above. Am I overlooking any solutions that might fit well with my user profile?
-
(I’ve not yet started this implementation.)
-
muchsync currently looks best.
-
In lieu of testing, this seems like the clear winner.
-
muchsync apparently syncs metadata and data (it seems less efficient to be forced to copy the data, but this may be unavoidable), but claims to do it as efficiently as possible.
-
Problem solved: the muchsync author ported to Mac OS 10.9.5 (at least) as of 2015-08-16 although this port may not yet be fully, empirically tested. Details here.
-
-
Others
My usage-scenario challenges include but may not be limited to:
Decide which MUA(s) to use, particularly deciding on a primary MUA. This is technically not a usage-scenario, but currently represents my biggest challenge. See the MUA options.
While I may be be moving to a search/index/tag-based paradigm, I still need to access my 4k+ IMAP folders as I did before, at least while I’m transitioning my current folder-based paradigm that I currently employ with Mozilla Thunderbird (TB), which leverages the TB’s Nostalgy add-on to do it.
TB-Nostalgy also offers excellent keyboard-shortcut-mapping capability and is one of the few great features of Thunderbird that I’d like to replicate in my new MUA.
-
I have multiple email accounts, which is not uncommon. I want to "view" each one differently, such that emails and folders from account X does not clutter my view of emails/folders when viewing account Y.
-
It would be extremely helpful to additionally support a "combined" view/mode of all my accounts. But this is not an absolute requirement, simply because #1 is currently more important than #2.
-
"tagging" my large set of IMAP folders
-
in particular:
Inbox
andSpam
folders → tags -
Is afew best for this?
-
See post-initial, automated tagging for more.
-
Context, details: mutt-kz thread: "Moving msgs after applying tags?".
-
Will messages retain Notmuch-associated metadata (tags, etc) for lifetime of any message, including post-folder moves - without any special configuration?
-
I’m used to moving messages between folders in order to classify. Further, I will like to keep a clean Inbox and other folders, for my non-Notmuch-based email clients, thus (presumably) requiring message moving.
-
Once I associate Notmuch-metadata (by adding tags, or whatever metadata/etc scenarios might be involved with Notmuch) with a message, I need said metadata/tags/etc to associate with a message forever, regardless of wherever I put said message. Is this the way it works "out of the box" with Notmuch-based systems?
-
-
I need to be able to find, goto/view, and copy/move messages to any IMAP folder (in the IMAP "folder tree") via a quick, keyboard-driven fashion with folder-name auto-completion. Thunderbird’s Nostalgy add-on does a great job of this.
-
I’m not yet certain how IMAP folders as tags affects this usage scenario.
-
It’s possible the more I employ "tags" instead of "folder" classification the less I needed this capability.
-
http://notmuchmail.org/pipermail/notmuch/2011/thread.html#3707
I want to "pipe" the content of:
-
one email message,
-
many email messages (by selecting multiple emails at the same time), or
-
an entire IMAP folder of emails
to any command/script of my choosing.
Example, potential solutions, not yet tested:
Example potential solutions, not yet tested:
How to do this? Including support for virtual folders. Details here.
Does anyone use notmuchsync, and does it work well?
-
opening attachments from MUA
-
afew?
-
See Initial tagging for more.
-
The Homely Mutt: Postponing Drafts
-
Does anyone employ this, and does it work well?
-
-
Haven’t yet seen this solved.
-
This discussion might be useful.
-
alot appears to have a problem with this.
-
Do mutt-kz or other MUA’s also experience this problem?
-
In summary, I’m a vim and Python lover, a keyboard jockey, and a reasonably-experienced, fairly-technical, demanding user. And like many others, I receive a remarkable amount of email in diverse contexts.
-
I’m historically-trained as a software and computer-systems engineer.
-
I’ve significant experience with programming in a variety of programming languages and system-administering a variety of OSes including but not limited to: C, C\++, Java, Ada, perl, Python; Windows, many commercial Unix-es, Linux, VMS, MacOSX. My favorite "Swiss army knife" language is Python. If I’ve time, I’m open to extending/fixing Python programs. I’d like to learn Ruby and Go.
-
-
I’m now more of a "business person." In spite of this:
-
vim remains my primary editor (I hate moving my hand from the keyboard to the mouse or trackpad),
-
Mac OS X is my primary computing machine,
-
and I still significantly code in Python to solve "glueware" problems.
-
I also still dabble in Linux (mostly Debian/Ubuntu) and MacOSX sysadmin.
-
-
Learning new systems/languages/applications/software is old hat…
-
…but it’s now harder only because of time constraints from expanded business responsibilities.
-
-
Some might describe me as an impatient, unforgiving computing user. I hate being faster than the computer. Further, when the computer/software/application says it’s job is done, I want it to be done. However, some environments and applications perform significant, asynchronous activity even after reporting they are done servicing a request. (Thunderbird is notorious for this.) And this drives me nuts. "Computer, if you need more time to complete a job, don’t lie to me. I can go do other things while I wait for you. But please do not delay me further after you already said you were done."
Despite my history assimilating to new applications/environments, the search-and-tag-based-classification paradigm still seems significantly different and a bit daunting to this "old school IMAP-folder user", and may (or may not?) take some time to master. See Usage-Scenario Challenges in Priority-First Order. For example, opening alot for the first time and looking at a staggering 50k+ emails in my "inbox" can give someone pause; hopefully Initial tagging will take care of that.
Additionally, the search-and-tag-based documentation resources—to describe new-user-paradigm shifts and present the most-popular toolsets—seem disjointed and/or non-existent. Hence some of the motivation to present this document.
Why did I spend the time to write this document, instead of just trying all the tools?
-
Email is too important not to "get it right." Or at least, email is too "frequent," probably my most-frequent life activity (very unfortunately).
-
Brute-force "experience" may be too inefficient. I’d rather learn from others' experiences rather than inefficiently reply them all myself.
-
This document may help future newbies. And possibly accelerate new-user population growth.
-
Defining requirements up front: this usually works. Rarely have I regretted taking the time to well-define requirements (separate from design and/or solution) for any significant software or tool-adoption project.
-
I might learn something I wouldn’t have previously found. It’s possible this document might attract enough attention for people to offer solutions (applications, workflows, or whatever) I might not have otherwise discovered.
-
Breaking my production email "IMAP database" testing new apps would be very… bad. My businesses and projects rely on my email system to be top-notch solid. If my email gets corrupted, lost, etc - things go very bad, very fast. Especially if I’m unknowingly messing up my email. Hence, I’m rather cautious about correct implementation.
In any case, I’m hopeful that experienced and diverse feedback from the search/index/tag-based-email-using communities can help avoid these problems. At least, it seemed like the most-effective way, as the space doesn’t (yet) seem friendly to newbies.
(DISCLAIMER: This section’s under construction, and not complete.)
OS X is great, but TB is difficult. Thunderbird is old, buggy, troublesome, slow, basically inextensible (for me, anyway), and as I understand it, feature frozen. I’m tired of debating with the mozillaZine support team about TB’s bugs and limitations. Among other things, it’s IMAP sync is slow and unreliable. It literally (and unfortunately, inconsistently) deletes IMAP folders on it’s own whim, asynchronously, sometimes when I least expect it. Sometimes it loses track of the folders it didn’t delete, and simply creates new ones, bloating my mbox (TB only reliably supports mbox) files terribly over time.
Additionally, the TB text/formatting editor is legendarily bad/buggy. I’d desperately prefer to simply edit in vim, and edit rich/html text in markdown or asciidoc and convert to html with a rendering engine, and I suspect I could script-integrate such capability… if I had an MUA that could play nicely with external scripts.
Further, I’m a keyboard jockey—eg: vim lover—and Python programmer. I’ve maxed out TB’s keyboard-shortcut-ness (eg: TB’s Nostalgy add-on) best I can tell, and it’s still limiting. I have external tools (some developed by me and/or my team) to parse and perform "magic" (like task-tracking and bug-report integration) on email folders and individual messages, and TB—with it’s lack of proper maildir support and difficult extensibility—makes it extremely difficult if not impossible to integrate with the external tools.
In short, it’s time to move on from Thunderbird.
(Previously-referenced guides or sections of guides listed elsewhere in this doc are not duplicated here. The following is provided here for my general reference; maybe others will find these references useful.)
-
Mutt + Notmuch (non- mutt-kz style)
-
http://stevelosh.com/blog/2012/10/the-homely-mutt/
-
may get replaced by mutt-kz, but other things possibly still useful:
-
-
-
mutt in general
-
http://bit.ly/notmuch—how-i-learned-to-stop-worrying-and-love-the-mail
-
http://www.benfrancom.com/2014/11/24/mutt-offline-with-mbsync/
-
http://www.techrepublic.com/article/10-helpful-tips-for-mutt-e-mail-client-power-users/
-
http://lifehacker.com/5574557/how-to-use-the-fast-and-powerful-mutt-email-client-with-gmail
From the mbsync(1) man page as of 2015-07:
Flatten delim
Flatten the hierarchy within this Store by substituting the canonical hierarchy delimiter / with delim. This can be useful when the MUA used to access the Store provides suboptimal handling of hierarchical mailboxes, as is the case with Mutt. A common choice for the delimiter is .. Note that flattened sub-folders of the INBOX always end up under Path, including the "INBOXdelim" prefix.
Does this mean mutt does not work with Maildir hierarchical subfolders?