razziel89 / go-imapgrab

A re-implementation of the amazing imapgrab in plain Golang.

Hi 👋, I'm Torsten

I'm a physicist and software developer with a passion for CLIs and backends.

• 🔭 At the moment, I am developing a customised home automation solution.
• 🌱 I’m currently learning Rust and Haskell. I'm also learning about Arduinos.
• 💬 Ask me about physics, computational quantum mechanics, Linux, shell scripts, software testing, Golang, CLIs, GitHub Actions, ...
• 📫 You can reach me by opening an issue and @-mentioning me here.
• ⚡Fun fact: I walk around in sandals or without shoes all year round, everywhere. 👣 🦶🦶

Languages, tools, and platforms that I enjoy using:

go-imapgrab's Issues

Fix downloads for folders whose name contain filesystem path separators

IMAP mailboxes can contain path separators (at least slashes /) that are used by the file system. For such a folder, go-imapgrab looks for the oldmail file in the wrong location.

Replace the slash in the folder name by a dot when determining the path to the oldmail file, but don't replace it in the path to the maildir (that's how the original imapgrap behaves).

Implement root command

It does nothing other than providing the help message.

Implement download command

Was: Using the getmail executable, implement downloading the content of a mailbox.

Updated: The getmail executable will not be used. Instead, an implementation in pure Go is desired. The implementation will likely be as follows:

• use go-imap to retrieve the content of emails
• use some as yet undefined Go library to add new emails to a maildir
  • possible options: go-message/mail

Parallelise folder downloads

Currently, multiple folders are downloaded sequentially. Since we spend most of the time waiting for remote systems or the file system, the entire process could be accelerated by parallelisation.

Thus, add a flag --threads that takes an integer and specifies how many folders can be downloaded at the same time.

Check where mutexes are needed, such as for the process-global delivery counter.

Don't implement before there are tests.

Add a graphical user interface

Some users prefer interacting with go-imapgrab via a graphical user interface. Add one. If possible, keep the feature platform-independent, which will likely require the use of a web-based UI. The library gowut looks like it could be an option.

Improve help and error messages

Print a typical command flow in the help text (login -> list -> download). If no secret can be found in the keyring, display a how to for using the login command.

Recover from errors when downloading

Currently, the first error stops execution. It would be great if emails and mailboxes could still be downloaded if one of them fails.

Implement list command

Imapgrab has a command used to list folders in a mailbox. Implement that in a very basic way.

Support system keyring

Passwords should not be stored in plain text anywhere. Support system keyrings.

Fix union tests of ordered set

The tests check the returned values, which is the exclusion not the union. The receiver set is modified.

Actually return the union set instead and leave the original set unmodified.

Implement system tests

The application has quite a few tests already, but no system tests yet that cover the entire path from CLI args to listing folders to emails showing up in a directory. Add some using a dummy in-memory IMAP server provided by go-imap.

Safeguard against parallel execution

Use file locking to ensure we do not modify the same maildir with parallel invocations.

Add tests

The code has only been tested against live systems so far. That will not do in the long run. Thus, add tests, both on the level of unit tests and integration tests. Do not add system tests as that would mean emulating an IMAP server.

Have login test the given credentials

It makes sense to have the login command test the provided credentials instead of simply storing them in the keyring. If the credentials don't work, the user should be asked for the password again until the password works or an empty password is provided.

Login command does not work

The login command does not take any flags and thus does not work.

Support multi-folder downloads

Currently, exactly one folder can be downloaded. It would be great to have a shorthand to download multiple ones.

Allow using a config file

It can be preferable to provide configs for different email accounts in a config file to allow easy reuse via an id. Furthermore, users might want to disable the keyring globally or change the verbosity level globally. Consider what makes sense and add a config file.

Simplify viewing stored emails

It would be nice if go-imapgrab could itself work as a read-only IMAP server that any mail client can connect to. That way, users could use their favourite program to view locally stored emails. Implement that. Also document how to use the feature with a common mail client such as thunderbird.

Assess severity of possilble race condition

There is a slight possibility for a race condition triggered by the following happening:

1. go-imapgrab obtains the list of emails known remotely
2. Two things happen at the same time:

• go-imapgrab determines which emails have already been downloaded
• some emails are deleted remotely

3. go-imapgrab requests the download of emails, but emails are not identified by their UIDs but rather by their index on the IMAP server

For example, assume there is a mailbox with 103 emails, the first 100 of which are already known locally. Then, go-imapgrab retrieves the list of 103 emails from the remote server. Then, someone deletes the 3 first emails, bringing the total count down to 100 again. That happens while go-imapgrab determines that the emails with indices 101 to 103 shall be downloaded. Then, go-imapgrab requests the download of those emails, receiving an error because no such emails are there. There can be other cases where incorrect emails would be downloaded instead.

This looks like a race condition that cannot be avoided (any suggestions to that end are welcome). Instead of avoiding it, we can live with detecting it. Thus, after downloading an email but before storing it on disk, we can perform a check for whether the UID of the email we did download is identical to the one we expected to be downloading. Such a sanity check would require some refactoring of the code base but seems like a viable option.

Add command to store credentials in keyring

Having to set an environment variable is cumbersome. It would be great to have a command that asked the user for the password interactively and stored it in the keyring.

Fix shell completion output

Currently, the completion command cannot be run because of how authentication is being handled at the root command level. Fix that. A likely fix is to run the authentication pre-run function only for those commands that require authentication.

Assess viability of using OAuth

Some email providers prefer OAuth for authorisation. Assess whether it can be supported with reasonable effort. Use GMail as an example provider that prefers OAuth.

Relevant articles:

• https://support.google.com/cloud/answer/9110914#restricted-scopes
• https://developers.google.com/gmail/imap/xoauth2-protocol
• https://developers.google.com/identity/protocols/oauth2
• https://stackoverflow.com/questions/75327403/oauth-authorization-in-golang-cli-applications

Document usage of the tool

The repo currently lacks any documentation outside the code. Document the tool and add examples for usage.

Provide binary distributions

Use goreleaser to release the binary using a GitHub Actions workflow.

It is not enough to require users to compile the binary themselves. There should be releases provided on GitHub, too.

Determine relevant flags from original imapgrab

The original implementation provides a lot of flags. Determine which ones are relevant in which case. There are the following ones

• General flags useful everywhere
• Flags only for listing
• Flags only for downloading

Upgrade `go-imap` to v2 once released

One of the central dependencies of this tool is go-imap. Currently, v2 is in the works and has been for a while. It seems as if v2 comes with some significant changes that may help simplify the code base of go-imapgrab a lot (e.g. no more channels but more straightforward iteration). Once v2 of go-imap has been released, an upgrade path should be determined and followed. Assume that this might require significant reword with the potential for much simplification.

Ensure graceful shutdown on interrupt

Listen to signals from the operating system and ensure we don't leave anything in a half-downloaded state.