gauteh / lieer

Fast email-fetching and sending and two-way tag synchronization between notmuch and GMail

Lieer

This program can pull, and send, email and labels (and changes to labels) from your GMail account and store them locally in a maildir with the labels synchronized with a notmuch database. The changes to tags in the notmuch database may be pushed back remotely to your GMail account.

Disclaimer

Lieer will not and can not:

• Add or delete messages on your remote account (except syncing the trash or spam label to messages, and those messages will eventually be deleted)
• Modify messages other than their labels

While Lieer has been used to successfully synchronize millions of messages and tags by now, it comes with NO WARRANTIES.

Requirements

• Python 3
• Notmuch 0.30+ for notmuch2 Python bindings
• google_api_python_client (sometimes google-api-python-client)
• google_auth_oauthlib
• tqdm (optional - for progress bar)

Installation

After cloning the repository Lieer can be installed through pip by using the command pip install .

Usage

This assumes your root mail folder is in ~/.mail and that this folder is already set up with notmuch.

1. Make a directory for the lieer storage and state files (local repository).

$ cd    ~/.mail
$ mkdir account.gmail
$ cd    account.gmail/

All commands should be run from the local mail repository unless otherwise specified.

2. Ignore the .json files in notmuch. Any tags listed in new.tags will be added to newly pulled messages. Process tags on new messages directly after running gmi, or run notmuch new to trigger the post-new hook for initial tagging. The new.tags are not ignored by default if you do not remove them, but you can prevent custom tags from being pushed to the remote by using e.g. gmi set --ignore-tags-local new. In your notmuch config file (usually ~/.notmuch-config):

[new]
tags=new
ignore=/.*[.](json|lock|bak)$/

3. Initialize the mail storage:

$ gmi init [email protected]

gmi init will now open your browser and request limited access to your e-mail.

The access token is stored in .credentials.gmailieer.json in the local mail repository. If you wish, you can specify your own api key that should be used.

4. You're now set up, and you can do the initial pull.

Use gmi -h or gmi command -h to get more usage information.

Pull

will pull down all remote changes since last time, overwriting any local tag changes of the affected messages.

$ gmi pull

the first time you do this, or if a full synchronization is needed it will take longer. You can try to use the --resume option if you get stuck on getting the metadata and have to abort (this will cause local changes made in the interim to be ignored in the next push).

Push

will push up all changes since last push, conflicting changes will be ignored unless -f is specified. these will be overwritten with the remote changes at the next pull.

$ gmi push

Normal synchronization routine

$ cd ~/.mail/account.gmail
$ gmi sync

This effectively does a push followed by a pull. Any conflicts detected with the remote in push will not be pushed. After the next pull has been run the conflicts should be resolved, overwriting the local changes with the remote changes. You can force the local changes to overwrite the remote changes by using push -f.

Note: If changes are being made on the remote, on a message that is currently being synced with lieer, the changes may be overwritten or merged in weird ways.

See below for more caveats.

Sending

Lieer may be used as a simple stand-in for the sendmail MTA. A typical configuration for a MUA send command might be:

gmi send -t -C ~/.mail/account.gmail

Like the real sendmail program, the raw message is read from stdin.

Most sendmail implementations allow passing additional recipients in additional arguments. However, the GMail API only supports the -t (--read-recipients) mode of sendmail, without additional recipients.

We try to support valid combinations from MUAs that make use of recipients passed as arguments. Additional recipients are ignored, but validated. The following combinations are OK:

• When -t is passed, we need to check for the CLI-passed recipients to be equal or a subset of the ones passed in the headers.

• When -t is not passed, all header-passed recipients need to be provided in the CLI as well.

This avoids silently not sending mail to some recipients (pretending we did), or sending mail to recipients we didn't want to send to again.

One of the implication of -t is you have to keep Bcc: header in your message when passing it to sendmail. It is not enough to just put the additional recipient on the command line. For mutt, this means setting write_bcc option.

Lieer will try to associate the sent message with the existing thread if it has an In-Reply-To header. According to the Gmail API the Subject: header must also match, but this does not seem to be necessary (at least not where just Re: has been prepended).

If the email address in the From: header does not match exactly the one of your account, it seems like GMail resets the from to your account address only.

Note that the following flags are ignored for sendmail compatibility:

• -f (ignored, set envelope From: yourself)
• -o (ignored)
• -i (always implied, not bothered by single .'s)

There are instructions for using this in your email client (for example Emacs) in the wiki.

Settings

Lieer can be configured using gmi set. Use without any options to get a list of the current settings as well as the current history ID and notmuch revision.

Account is the GMail account the repository is synced with. Configured during setup with gmi init.

historyId is the latest synced GMail revision. Anything since this ID will be fetched on the next gmi pull (partial).

lastmod is the latest synced Notmuch database revision. Anything changed after this revision will be pushed on gmi push.

Timeout is the timeout in seconds used for the HTTP connection to GMail. 0 means the forever or system error/timeout, whichever occurs first.

File extension is an optional argument to include the specified extension in local file names (e.g., mbox) which can be useful for indexing them with third-party programs.

Important: If you change this setting after synchronizing, the best case scenario is that all files will appear to not have being pulled down and will be re-downloaded (and duplicated with a different extension in the maildir). There might also be changes to tags. You should in theory be able to change it by renaming all files, but since this will update the lastmod you will get a check on all files.

Drop non existing labels can be used to silently ignore errors where GMail gives us a label identifier which is not associated with a label. See Caveats.

Replace slash with dot is used to replace the sub-label separator (/) with a dot (.). I think this is easier to work with. Important: See note below on changing this setting after initial sync.

Ignore tags (local) can be used to specify a list of tags which should not be synced from local to remote (e.g. new). In addition to the user-configured tags these tags are ignored: 'attachment', 'encrypted', 'signed', 'passed', 'replied', 'muted', 'mute', 'todo', 'Trash', 'voicemail'. Some are special tags in notmuch and some are unsupported by GMail. See Caveats below for more explanations. Note: This setting expects translated tags.

Important: See note below on changing this setting after initial sync.

Ignore tags (remote) can be used to specify a list of tags (labels) which should not be synced from remote (GMail) to local. By default the CATEGORY_* type labels which are mapped to the Personal/Promotions/etc tabs in the GMail interface are ignored. You can specify that no label should ignored by doing: gmi set --ignore-tags-remote "". Note: This setting expects untranslated tags.

Important: See note below on changing this setting after initial sync.

Local Trash Tag (local) can be used to set the local tag to which the remote GMail 'TRASH' label is translated.

Important: See note below on changing this setting after initial sync.

Translation List Overlay can be used to add or change entries in the translation mapping between local and remote tags. Argument is a comment-separated list with an even number of items. This is interpreted as a list of pairs of (remote, local), where each pair is added to the tag translation overwriting any existing translation for that tag if any. For example, --translation-list-overlay CATEGORY_FORUMS,my_forum_tag will translate Google's CATEGORY_FORUMS tag to my_forum_tag.')

Important: See note below on changing this setting after initial sync.

Changing ignored tags and translation after initial sync

If you change the ignored tags after the initial sync this will not update already synced messages. This means that if a change is made locally on an already synced message the previously ignored remote labels may be deleted. Conversely, if a change occurs remotely on a message which previously which has local tags that were ignored before, these ignored tags may be deleted.

The best way to deal with this is to do a full push or pull after having changed one of the settings. Do not change both --ignore-tags-locally and --ignore-tags-remote at the same time.

Before changing either setting make sure you are fully synchronized. After changing e.g. --ignore-tags-remote do first a dry-run and then a real run of full gmi pull -f --dry-run. This will fetch the full tag list for all messages and overwrite the local tags of all your messages with the remote labels.

When changing the opposite setting: --ignore-tags-local, do a full push (dry-run first): gmi push -f --dry-run.

The same goes for the options --replace-slash-with-dot and --local-trash-tag. I prefer to do gmi pull -f --dry-run after changing this option. This will overwrite the local tags with the remote labels.

Translation between labels and tags

We translate some of the GMail labels to other tags. The default map of labels to tags are:

  'INBOX'     : 'inbox',
  'SPAM'      : 'spam',
  'TRASH'     : 'trash',
  'UNREAD'    : 'unread',
  'STARRED'   : 'flagged',
  'IMPORTANT' : 'important',
  'SENT'      : 'sent',
  'DRAFT'     : 'draft',
  'CHAT'      : 'chat',

  'CATEGORY_PERSONAL'     : 'personal',
  'CATEGORY_SOCIAL'       : 'social',
  'CATEGORY_PROMOTIONS'   : 'promotions',
  'CATEGORY_UPDATES'      : 'updates',
  'CATEGORY_FORUMS'       : 'forums',

The 'trash' local tag can be replaced using the --local-trash-tag option.

Using your own API key

Lieer ships with an API key that is shared openly, this key shares API quota, but cannot be used to access data unless access is gained to your private access_token or refresh_token.

You can get an api key for a CLI application to use for yourself. Store the client_secret.json file somewhere safe and specify it to gmi auth -c. You can do this on a repository that is already initialized, possibly using -f to force reauthorizing with the new client secrets.

Privacy policy

Lieer downloads e-mail and labels to your local computer. No data is sent elsewhere.

Lieers use and transfer to any other app of information received from Google APIs will adhere to Google API Services User Data Policy, including the Limited Use requirements

Caveats

• The GMail API does not let you sync muted messages. Until this Google bug is fixed, the mute and muted tags are not synchronized with the remote.

• The todo and voicemail labels seem to be reserved and will be ignored.

• The draft and sent labels are read only: They are synced from GMail to local notmuch tags, but not back (if you change them via notmuch).

• Only one of the tags inbox, spam, and trash may be added to an email. For the time being, trash will be preferred over spam, and spam over inbox.

• Trash (capital T) is reserved and not allowed, use trash (lowercase, see above) to bin messages remotely.

• archive or arxiv are reserved and not allowed; see issue/109 and issue/171. To archive e-mails remove the inbox tag.

• Sometimes GMail provides a label identifier on a message for a label that does not exist. If you encounter this issue you can get around it by using gmi set --drop-non-existing-labels and re-try to pull. The labels will now be ignored, and if this message is ever synced back up the unmapped label ID will be removed. You can list labels with gmi pull -t.

• Sometimes GMail indicates that there are more changes when doing a partial pull, but an empty set is returned. The default is to fail, but you can ignore empty history by setting: gmi set --ignore-empty-history.

• You cannot add any new files (files starting with . will be ignored) to the lieer repository. Lieer uses the directory content an index of local files. Lieer does not push new messages to your account (note that if you send messages with GMail, GMail automatically adds the message to your mailbox).

• Make sure that you use the same domain for you GMail account as you initially created your account with: usually @gmail.com, but sometimes @googlemail.com. Otherwise you might get a Delegation denied error.

Development

Github actions are configured to check for python code formatted by black.

lieer's Issues

Unhandled exception: cannot reduce request any further

During the initial clone I got this error:

log.txt

Sync drafts

According to the readme, gmailieer cannot add email. This is somewhat problematic for drafts. I could set up mbsync just for drafts, but I think that would create duplication since once a draft is pushed, gmi sync would add it to its maildir as well.

I'm not familiar with gmail API, but can drafts be somehow supported? If not, is it possible to black list some tags?

AttributeError while pulling mail

$ gmi sync                                                              ~/Mail/personal.gmail
receiving metadata: 0it [00:00, ?it/s]
resolving changes: 0it [00:00, ?it/s]
push: nothing to push
remote historyId: 12267967
pull: full synchronization (no previous synchronization state)
fetching messages: 139732it [11:53, 134.01it/s]                                               
receiving content:   0%|                               | 1/139732 [00:03<155:04:25,  4.00s/it]Traceback (most recent call last):
  File "/usr/bin/gmi", line 4, in <module>
    __import__('pkg_resources').run_script('gmailieer==0.2', 'gmi')
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 742, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1510, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/EGG-INFO/scripts/gmi", line 8, in <module>
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 136, in main
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 190, in sync
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 307, in pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 527, in full_pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 595, in get_content
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 100, in func_wrap
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 271, in get_messages
  File "/usr/lib/python3.6/site-packages/oauth2client-4.1.2-py3.6.egg/oauth2client/_helpers.py", line 133, in positional_wrapper
  File "/usr/lib/python3.6/site-packages/google_api_python_client-1.6.3-py3.6.egg/googleapiclient/http.py", line 1464, in execute
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 251, in _cb
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 593, in _got_msg
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/local.py", line 294, in store
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/local.py", line 335, in update_tags
AttributeError: 'Database' object has no attribute 'index_file'

"sent" label is not synced back

Apparently, the "sent" label (pseudo label) is only synced from GMail to the notmuch database, not vice versa:

If I send an e-mail using GMail's web interface (or IMAP) it gets saved by GMail and appears under "sent messages"; when synced by gmailieer, it gets the notmuch tag "sent". So far so good.

If I tag a message as sent using "notmuch tag +sent" it gets the tag, and gmailieer seems to update something on sync but the message does not appear under "sent messages" in GMail (only "all mail").

I know that "sent" is not quite a label for GMail but something else - but maybe it could be synced both ways nonetheless?

[Background: I sometimes send using none-GMail-SMTP servers and want to mbsync (via IMAP) a copy of the sent message to my GMail account; this works fine, but I can't tag the copy as sent (i.e. sync the sent label) using gmailieer.]

Sync SPAM and TRASH

At the moment the value of remote.not_sync is fixed to
not_sync = set (['CHAT', 'SPAM', 'TRASH'])
so that SPAM and TRASH are not downloaded.

In order to be able to undelete or remove from spam, I would need to sync these as well. Is it possible to make this a configurable option?

Error while syncing cleaned up inbox

I removed some 4000 emails on Gmail side (i.e. emptying trash) and when I do a sync I occasionally get the following:

Traceback (most recent call last):
  File "/nix/store/q1khwyfbip46yyddm0li2fsn0c3y9mx4-python3.6-gmaileer-v0.3-20171115/bin/.gmi-wrapped", line 9, in <module>
    g.main ()
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 190, in sync
    self.pull (args, True)
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 315, in pull
    self.partial_pull ()
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 457, in partial_pull
    self.local.remove (m['id'], db)
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 298, in remove
    nmsg  = db.find_message_by_filename (fname)
  File "/nix/store/raxsnrd51a6j0f8qpmi3f4ri1lx4r0p4-python3-3.6.2-env/lib/python3.6/site-packages/notmuch/database.py", line 560, in find_message_by_filename
    raise NotmuchError(status)
notmuch.errors.XapianError

"cannot reduce request any further"

I set up alot to use the maildir syncronized by gmailieer as send_box. When I send a mail (and store and tag it "sent") and then syncronize I get the following error.

[~/mail/account.patricktotzke.gmail.com] ~/projects/gmailieer/gmi sync                                                    
receiving metadata:   0%|                                                                                               | 0/9 [00:00<?, ?it/s]reducing batch request size to: 25
reducing batch request size to: 12
reducing batch request size to: 6
Traceback (most recent call last):
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 282, in get_messages
    batch.execute (http = self.http)
  File "/usr/lib/python3/dist-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/googleapiclient/http.py", line 1451, in execute
    self._callback(request_id, response, exception)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 258, in _cb
    raise Remote.BatchException(excep)
lieer.remote.BatchException: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/patricktotzke%40gmail.com/messages/1511291035.M158427P31771Q1.piu?format=minimal&alt=json returned "Invalid id value">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pazz/projects/gmailieer/gmi", line 8, in <module>
    g.main ()
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 143, in main
    args.func (args)
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 193, in sync
    self.push (args, True)
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 238, in push
    self.remote.get_messages (gids, _got_msg, 'minimal')
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 305, in get_messages
    raise Remote.BatchException ("cannot reduce request any further")
lieer.remote.BatchException: cannot reduce request any further
[1]    431 abort      ~/projects/gmailieer/gmi sync

Can't initialize

I get this error when trying to run init; creating the file results in gmi saying that '.gmailieer.json' exists: this repository seems to already be set up.

username@computername:~/.mail> gmi init [email protected]
initializing repository in: /home/username/.mail..
Traceback (most recent call last):
File "/usr/bin/gmi", line 4, in
import('pkg_resources').run_script('gmailieer==0.3', 'gmi')
File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 748, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 1524, in run_script
exec(script_code, namespace, namespace)
File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/EGG-INFO/scripts/gmi", line 8, in
File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 136, in main
File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 140, in initialize
File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/local.py", line 199, in initialize_repository
File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/local.py", line 81, in write
File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/home/username/.mail/.gmailieer.json'

KeyError: '15bac9a5ba0b826b' in update_tags: self.mids[mid]

Traceback (most recent call last):
  File "/home/jak/bin/gmi", line 8, in <module>
    g.main ()
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 190, in sync
    self.pull (args, True)
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 315, in pull
    self.partial_pull ()
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 463, in partial_pull
    r = self.local.update_tags (m, None, db)
  File "/home/jak/Downloads/gmailieer/lieer/local.py", line 303, in update_tags
    fname = self.mids[mid]
KeyError: '15bac9a5ba0b826b'```

MAX_CONNECTION_ERRORS is not defined

fetching messages: 436942it [20:23, 228.73it/s]
receiving content:  28%|███████████████████████████████████████████████████████▏                                                                                                                                           | 123800/436942 [6:43:02<18:42:11,  4.65it/s]connection failed, re-trying: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/remote.py", line 282, in get_messages
    batch.execute (http = self.http)
  File "/nix/store/d5bc937ndi1rd1kd693c3l11ziz2qcjl-python3.6-oauth2client-1.4.12/lib/python3.6/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/nix/store/hvhy4l4zl4m8rh2q5pdd9b8pa2r5bz2v-python3.6-google-api-python-client-1.5.1/lib/python3.6/site-packages/googleapiclient/http.py", line 1409, in execute
    self._refresh_and_apply_credentials(request, http)
  File "/nix/store/hvhy4l4zl4m8rh2q5pdd9b8pa2r5bz2v-python3.6-google-api-python-client-1.5.1/lib/python3.6/site-packages/googleapiclient/http.py", line 1123, in _refresh_and_apply_credentials
    creds.refresh(http)
  File "/nix/store/d5bc937ndi1rd1kd693c3l11ziz2qcjl-python3.6-oauth2client-1.4.12/lib/python3.6/site-packages/oauth2client/client.py", line 597, in refresh
    self._refresh(http.request)
  File "/nix/store/d5bc937ndi1rd1kd693c3l11ziz2qcjl-python3.6-oauth2client-1.4.12/lib/python3.6/site-packages/oauth2client/client.py", line 780, in _refresh
    self._do_refresh_request(http_request)
  File "/nix/store/d5bc937ndi1rd1kd693c3l11ziz2qcjl-python3.6-oauth2client-1.4.12/lib/python3.6/site-packages/oauth2client/client.py", line 799, in _do_refresh_request
    self.token_uri, method='POST', body=body, headers=headers)
  File "/nix/store/d5bc937ndi1rd1kd693c3l11ziz2qcjl-python3.6-oauth2client-1.4.12/lib/python3.6/site-packages/oauth2client/client.py", line 562, in new_request
    redirections, connection_type)
  File "/nix/store/xcyj7i0gghrh4r2v4v91aa2ia370z0wc-python3.6-httplib2-0.9.2/lib/python3.6/site-packages/httplib2/__init__.py", line 1314, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/nix/store/xcyj7i0gghrh4r2v4v91aa2ia370z0wc-python3.6-httplib2-0.9.2/lib/python3.6/site-packages/httplib2/__init__.py", line 1064, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/nix/store/xcyj7i0gghrh4r2v4v91aa2ia370z0wc-python3.6-httplib2-0.9.2/lib/python3.6/site-packages/httplib2/__init__.py", line 988, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/http/client.py", line 1065, in _send_output
    self.send(chunk)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/http/client.py", line 986, in send
    self.sock.sendall(data)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/ssl.py", line 972, in sendall
    v = self.send(byte_view[count:])
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/ssl.py", line 941, in send
    return self._sslobj.write(data)
  File "/nix/store/p1bp0kpmxi7nzrla7n7c4waqic0a2myk-python3-3.6.4/lib/python3.6/ssl.py", line 642, in write
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/bin/.gmi-wrapped", line 9, in <module>
    g.main ()
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/gmailieer.py", line 143, in main
    args.func (args)
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/gmailieer.py", line 314, in pull
    self.full_pull ()
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/gmailieer.py", line 537, in full_pull
    updated = self.get_content (message_gids)
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/gmailieer.py", line 602, in get_content
    self.remote.get_messages (need_content, _got_msg, 'raw')
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/nix/store/3qdxkgd7808m8c6wikrpxjnbr2sw39hl-gmailieer/lib/python3.6/site-packages/lieer/remote.py", line 314, in get_messages
    if conn_errors > MAX_CONNECTION_ERRORS:
NameError: name 'MAX_CONNECTION_ERRORS' is not defined

On NixOS with version 0.5.

Initial `push` (after `pull`) hits API limits

Hello,

I successfully moved two of three accounts to gmailieer. The problem is with the third and biggest account (roughly 200.000 messages):

I can successfully init and pull the messages, but a following push or sync will get throttled more and more and finally stop as gmailieer can't reduce the batch size any further.

Is there something I can do about this?

Unexpected behavior: inbox emptied

Unexpected behavior: inbox emptied

I've just switched to gmailieer, and followed the instructions on the webpage,
and did an initial download, and there are several things that seem odd to me,
(I've tried finding a IRC-channel or FAQ to ask these questions).

1. After running gmi sync from the folder it has emptied my inbox. Prior
   to the sync, I had +2000 conversation threads in the web-interface of gmail,
   after the sync I have 7. The rest seems to reside in .mail/gmail/mail/cur
   (about 20k of files).

2. Most importantly for me: Is it possible to push the messages back up to
   web-gmail some how? And why/how can this happen?

3. When sending a mail, and using mbsync and notmuch, I'd have a fcc of the
   mail to my "sent" folder. Should I do something similar for gmailieer, or it
   should not be fcc-ed anywhere?

4. Looking at the email in notmuch (emacs), not all messages that I've
   trashed, using web-interface of gmail, are marked with the trash tag,
   only some are.

Note, I ran gmi sync several times, to properly propagate tag changes I had
done to the downloaded mail in notmuch.el, and on one occasion, gmailieer
produced thousands of rows like the following:

update: gid does not match any file name of message, probably a draft, skipping: 12e247f736244708
update: gid does not match any file name of message, probably a draft, skipping: 12e2479a1d3a6ec0
update: gid does not match any file name of message, probably a draft, skipping: 12e2475f9eba8394

EDIT: Using my browser to check gmail, I see that the mail that was
trashed before is back, so I assume, the trash tag issue is because they've
ended back in my inbox for some reason.

Invalid label name ?

I see this during gmi sync

Traceback (most recent call last):
  File "/home/pazz/projects/gmailieer/gmi", line 8, in <module>
    g.main ()
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 143, in main
    args.func (args)
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 193, in sync
    self.push (args, True)
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 245, in push
    actions.append (self.remote.update (rm, nm, self.local.state.last_historyId, self.force))
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 503, in update
    return self.__push_tags__ (gid, add, rem)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 519, in __push_tags__
    (lid, ll) = self.__create_label__ (a)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 630, in __create_label__
    lr = self.service.users ().labels ().create (userId = self.account, body = label).execute ()
  File "/usr/lib/python3/dist-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/googleapiclient/http.py", line 838, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/octo%40gmail.com/labels?alt=json returned "Invalid label name">

Replied and Passed maildir flags are not pulled

Initial pull fails with KeyError

Output:

pull: full synchronization (no previous synchronization state)
fetching messages: 353225it [18:57, 310.48it/s]                                 
receiving content:  73%|████████▋   | 257019/353225 [6:17:36<2:56:26,  9.09it/s]Traceback (most recent call last):
  File "/usr/bin/gmi", line 8, in <module>
    g.main ()
  File "/usr/lib/python3.6/site-packages/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/usr/lib/python3.6/site-packages/lieer/gmailieer.py", line 307, in pull
    self.full_pull ()
  File "/usr/lib/python3.6/site-packages/lieer/gmailieer.py", line 527, in full_pull
    updated = self.get_content (message_gids)
  File "/usr/lib/python3.6/site-packages/lieer/gmailieer.py", line 592, in get_content
    self.remote.get_messages (need_content, _got_msg, 'raw')
  File "/usr/lib/python3.6/site-packages/lieer/remote.py", line 106, in func_wrap
    return func (self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/lieer/remote.py", line 277, in get_messages
    batch.execute (http = self.http)
  File "/usr/lib/python3.6/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/googleapiclient/http.py", line 1464, in execute
    self._callback(request_id, response, exception)
  File "/usr/lib/python3.6/site-packages/lieer/remote.py", line 257, in _cb
    cb (resp)
  File "/usr/lib/python3.6/site-packages/lieer/gmailieer.py", line 590, in _got_msg
    self.local.store (m, db)
  File "/usr/lib/python3.6/site-packages/lieer/local.py", line 343, in store
    self.update_tags (m, p, db)
  File "/usr/lib/python3.6/site-packages/lieer/local.py", line 351, in update_tags
    labels = [self.gmailieer.remote.labels[l] for l in labels]
  File "/usr/lib/python3.6/site-packages/lieer/local.py", line 351, in <listcomp>
    labels = [self.gmailieer.remote.labels[l] for l in labels]
KeyError: 'Label_9'

gmi init breaks with error: "ImportError: No module named 'oauth2client'"

I have next to zero knowledge of these kind of "python problems". I tried to do eveything by the book, however: selected py36 trough macports' sudo port select --set python python36, same with pip, installed oauth2 with pip --upgrade ...

Everything seemed to have gone fine. Then, when I tried gmi init my.account@gmail :

Traceback (most recent call last):
  File "/Users/m2habert/gitRepo/gmailieer/gmi", line 4, in <module>
    from lieer import Gmailieer
  File "/Users/m2habert/gitRepo/gmailieer/lieer/__init__.py", line 1, in <module>
    from .gmailieer import *
  File "/Users/m2habert/gitRepo/gmailieer/lieer/gmailieer.py", line 8, in <module>
    from    oauth2client import tools
ImportError: No module named 'oauth2client'

I've tried googling the error message, but couldn't make heads or tails of anything (it seems all related complaints are posted by developers programming their own APIs, nothing like a software user with installation problems)

¿Can anybody point me to the right direction? Thanks
Using MacOS 10.11.6

<HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/julian.klode%40gmail.com/messages/15c69db4486d0718/modify?alt=json returned "Can only add on e of: INBOX, SPAM, TRASH">

If I locally tag a mail as both +inbox and +trash (which makes no sense, really), the sync fails. gmailieer should instead just drop the inbox tag and add the trash tag I guess.

README.md -- Installation instructions need help

I found the README.md insufficient to help me do a normal install of gmailieer.

Here are the problems I encountered, along with some suggestions for solutions. I don't have all the solutions, though, because I simply got blocked, and don't know the answer.

1. In the requirements section of the README.me, it is unclear what these names are. Are they pip packages? .deb's or .rpm's? Also, setup.py will not run without setuptools, which is not mentioned. So:
   a. add setuptools to the requirements list as the 6th item:

• setuptools

b. Give explicit examples as to how these requirements may actually be met. For instance, add the following text:

In Debian 9 ("Stretch"), for instance, these prerequisites can be obtained this way:

$ sudo apt-get -y install python3-tqdm python3-googleapi python3-oauth2client python3-notmuch python3-setuptools

2. "cart before horse"? -- notmuch seems to be required before running.
   In the installation section, item 2 seems to assume that one already has a ~/.mail/[account].gmail/.notmuch directory set up. (I only this having worked with notmuch already. But I would have been greatly confused as a newbie, if I had not. I would have had NO idea what this even means.) But the instructions to create a .notmuch subdir do not exist here, and are only implied. Also, the instructions do not say that what they are really referring to is ~/.mail/[account].gmail/.notmuch/hooks/post-new . They also apparently refer to ~/.notmuch-config , and assume the ~/.notmuch-config is already set up and pointing to ~/.mail/[account].gmail -- But again, this is not stated, and if I had not already worked with notmuch, I would not have known this. I would have been completely lost.

3. Even setting up a ~/.notmuch-config and pointing it to ~/.mail/[account].gmail , and setting up a ~/.mail/[account].gmail/.notmuch/hooks/post-new script: When I cd to ~/.mail/[account].gmail, and run,

gmi init [account]@gmail

as per the instructions, I get:

A Xapian exception occurred opening database: Couldn't stat '/home/[me]/.mail/[account].gmail/.notmuch/xapian'

So, what? gmailieer expects notmuch to already have been run? How is that possible, when the maildir is not yet downloaded?

(I have no solution for this. I am stuck.)

4. If I try to run

gmi init [account]@gmail

a second time, now I get a different error message:

lieer.local.RepositoryException: '.gmailieer.json' exists: this repository seems to already be set up!

(and if I remove this, I get a similar message for the 'mail' subdir previously created.)

5. Also, the README.md needs to make it clear whether it is able to work with a previously-existing maildir or not. And whether offlineimap or mbsync may be assumed. In another issue, it appears to be no for both of these. If that is the case, this should be made explicit in the README.md. (Although that is too bad, because it means having do re-download all my emails.)

But if a previously-existing maildir may not be used, then how is it that it is assumed that notmuch is already installed and its database initialized?

---

In the end, the README.md installation instructions simply do not work for me. I am dead in the water. I have spent all morning on it and tried to add some possible solutions, but I think really the best solution is that you run an actual test installation yourself from a virgin system and document it, step by step, in the README.md, so that the instructions actually work. I would do this for you (with a fork) but I myself cannot figure out what you wanted.

Hope this helps.

KeyError: 'history'

During a sync I got this exception:

log.txt

After first pull, all emails have the inbox and unread labels

Other labels are fine though. Did I do something wrong? Only 2 emails do not have the inbox and unread labels.

Assertion Error status=count(messages)- can't really pinpoint the source

I can pull message, but not push.
Using the current HEAD, notmuch in git package version in Arch

During push an Assertion Error kills the script

Traceback (most recent call last):
  File "/usr/bin/gmi", line 4, in <module>
    __import__('pkg_resources').run_script('gmailieer==0.1', 'gmi')
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 739, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1507, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python3.6/site-packages/gmailieer-0.1-py3.6.egg/EGG-INFO/scripts/gmi", line 8, in <module>
  File "/usr/lib/python3.6/site-packages/gmailieer-0.1-py3.6.egg/lieer/gmailieer.py", line 136, in main
  File "/usr/lib/python3.6/site-packages/gmailieer-0.1-py3.6.egg/lieer/gmailieer.py", line 186, in sync
  File "/usr/lib/python3.6/site-packages/gmailieer-0.1-py3.6.egg/lieer/gmailieer.py", line 213, in push
  File "/usr/lib/python3.6/site-packages/notmuch/query.py", line 204, in count_messages
    raise NotmuchError(status)
  File "/usr/lib/python3.6/site-packages/notmuch/errors.py", line 129, in __new__
    subclass = cls.get_exc_subclass(status)  # which class to use?
  File "/usr/lib/python3.6/site-packages/notmuch/errors.py", line 115, in get_exc_subclass
    assert 0 < status <= len(subclasses)
AssertionError

status seems like the count of my messages: ~22k, and len(subclasses): 14,

The Function in count_messages in "notmuch/query.py" tries to count messages and should give a status back, But it seems, the status is the message count.
It seems the python script doesn't match with the notmuch library.

def count_messages(self):
        '''
        This function performs a search and returns Xapian's best
        guess as to the number of matching messages.

        :returns: the estimated number of messages matching this query
        :rtype:   int
        '''
        self._assert_query_is_initialized()
        count = c_uint(0)
        status = Query._count_messages(self._query, byref(count))
        if status != 0:
            raise NotmuchError(status)
        return count.value

If see the problem in the packaging of notmuch as well, you can close this issue.

README.md instructions for setup.py need to be fleshed out

In the README.md, it is suggested to symlink gmi, "or use setup.py".

When I tried running python setup.py and just got this:

$ python3 setup.py
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: no commands supplied
$ _

so I ran:

$ python3 setup.py --help-commands
Standard commands:
  build             build everything needed to install
  build_py          "build" pure Python modules (copy to build directory)
  build_ext         build C/C++ extensions (compile/link to build directory)
  build_clib        build C/C++ libraries used by Python extensions
  build_scripts     "build" scripts (copy and fixup #! line)
  clean             clean up temporary files from 'build' command
  install           install everything from build directory
  install_lib       install all Python modules (extensions and pure Python)
  install_headers   install C/C++ header files
  install_scripts   install scripts (Python or otherwise)
  install_data      install data files
  sdist             create a source distribution (tarball, zip file, etc.)
  register          register the distribution with the Python package index
  bdist             create a built (binary) distribution
  bdist_dumb        create a "dumb" built distribution
  bdist_rpm         create an RPM distribution
  bdist_wininst     create an executable installer for MS Windows
  check             perform some checks on the package
  upload            upload binary package to PyPI

Extra commands:
  easy_install      Find/get/install Python packages
  bdist_egg         create an "egg" distribution
  alias             define a shortcut to invoke one or more commands
  install_egg_info  Install an .egg-info directory for the package
  rotate            delete older distributions, keeping N newest files
  develop           install package in 'development mode'
  upload_docs       Upload documentation to PyPI
  saveopts          save supplied options to setup.cfg or other config file
  test              run unit tests after in-place build
  setopt            set an option in setup.cfg or another config file
  egg_info          create a distribution's .egg-info directory

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help
$ _

..which I didn't know what to do with.

Ultimately, I just gave up and symlinked gmi, as suggested. Just simpler.

Database object has no attribute get_revision

pull: full synchronization (no previous synchronization state)
fetching messages: 7479it [00:16, 322.74it/s]                                                                                                                                                  
receiving content: everything up-to-date.
receiving metadata:  59%|████████████████████████████████████████████████████████████████████████████▍                                                     | 4401/7479 [00:47<00:34, 88.41it/s]
reducing batch request size to: 100
receiving metadata: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7479/7479 [01:22<00:00, 90.88it/s]
Traceback (most recent call last):
  File "/home/aureooms/.bin/gmi", line 8, in <module>
    g.main ()
  File "/home/aureooms/.opt/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/home/aureooms/.opt/gmailieer/lieer/gmailieer.py", line 272, in pull
    self.full_pull ()
  File "/home/aureooms/.opt/gmailieer/lieer/gmailieer.py", line 499, in full_pull
    (rev, uuid) = db.get_revision ()
AttributeError: 'Database' object has no attribute 'get_revision'

Multiple gmail accounts?

Is there a good way of setting up multiple accounts in a single repository?

Label name exists or conflicts

Traceback (most recent call last):
  File "/home/maxking/Documents/gmailieer/lieer/remote.py", line 569, in __create_label__
    lr = self.service.users ().labels ().create (userId = self.account, body = label).execute ()
  File "/usr/local/lib/python3.5/dist-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/googleapiclient/http.py", line 840, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 409 when requesting https://www.googleapis.com/gmail/v1/users/raj.abhilash1%40gmail.com/labels?alt=json returned "Label name exists or conflicts">

If a local label exists, it tries to create a create same label remotely and fails

After initial pull, too many messages have inbox tag

This is my post-new hook (the only hook I have):

#!/bin/sh

# immediately archive all messages from "me"
notmuch tag -new -- tag:new and from:email@domain

# finally, remove new
notmuch tag -new -- tag:new

and my .notmuch_config:

[database]
...
[user]
...
[new]
tags=new
ignore=*.json;
[search]
exclude_tags=deleted;spam;
[maildir]
synchronize_flags=true
[crypto]
gpg_path=gpg

After the first full pull (gmi pull), I see about >80k messages tagged with inbox while gmail shows me only 1700? I guess I miss something, but I don't understand the discrepacy between the numbers?

http.client.RemoteDisconnected not handled

I got this while running gmi pull

log.txt

AttributeError exception

I hit this while testing #4

log.txt

ship oauth2 client id in some safe way

So that users don't need an API key. Should be configurable, since there is an API limit on requests which may easily be transcended when doing initial pulls.

lieer.remote.GenericException: no 'history' record returned, even though several pages were indicated.

Note sure what happened. I was removing the inbox tag on about 30 messages or so, and then syncing.

pull: partial synchronization.. (hid: 12008538)
fetching changes: 0it [00:00, ?it/s]Traceback (most recent call last):
  File "/home/jak/Downloads/gmailieer/gmi", line 8, in <module>
    g.main ()
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 190, in sync
    self.pull (args, True)
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 315, in pull
    self.partial_pull ()
  File "/home/jak/Downloads/gmailieer/lieer/gmailieer.py", line 324, in partial_pull
    for hist in self.remote.get_history_since (self.local.state.last_historyId):
  File "/home/jak/Downloads/gmailieer/lieer/remote.py", line 166, in get_history_since
    raise Remote.GenericException ("no 'history' record returned, even though several pages were indicated.")
lieer.remote.GenericException: no 'history' record returned, even though several pages were indicated.
fetching changes: 100it [00:00, 607.63it/s]

Implement gmi send

Implement gmi send which sends messages using Users.messages.send, adds the messages instantly to the local repository and applies specified tags. Should be a drop-in for sendmail and support common functionality (e.g. the subset implemented by msmtp).

If drafts are supported this should use Drafts.send, but as far as I can see this will add a significant layer of complexity and we could skip that for now.

"Backend Error" not handled

I noticed I had a "new" tag in every message. A leftover from my old notmuch config. I removed it from all messages, and ran "gmi sync". It started checking every message flags against the remote, and eventually failed with the attached log. Maybe it should retry?

log.txt

customize ignoring tags

I'm using "snoozing" mails quite often. Atm i'm removing the inbox tag and adding two new tags: snooze, snoozeuntil-. If matches the current date the email will get readded to inbox :)

gmi syncs both tags towards gmail and clutters the taglist.

Ignoring tags is a thing already. It would be nice to customize it :)

segmentation fault

I got the following backtrace during a sync

log.txt

use a batch method when pushing

just like pulling. This is useful if you decide to run notmuch tag --batch and it touches 80k files.

muted tag is invalid to push

pushing, 0 changed:   0%|                                                                                                                                                               | 0/162 [00:00<?, ?it/s]
push: creating label: muted..
Traceback (most recent call last):
  File "/Users/charlie/src/gmailieer/gmi", line 8, in <module>
    g.main ()
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 186, in sync
    self.push (args, True)
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 224, in push
    r = self.remote.update (m, self.local.state.last_historyId, self.force)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 83, in func_wrap
    return func (self, *args, **kwargs)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 376, in update
    self.__push_tags__ (mid, add, rem)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 83, in func_wrap
    return func (self, *args, **kwargs)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 396, in __push_tags__
    (lid, ll) = self.__create_label__ (a)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 83, in func_wrap
    return func (self, *args, **kwargs)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 433, in __create_label__
    lr = self.service.users ().labels ().create (userId = self.account, body = label).execute ()
  File "/Users/charlie/src/gmailieer/venv/lib/python3.6/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.6/site-packages/googleapiclient/http.py", line 840, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/charlie%40eatyourpets.com/labels?alt=json returned "Invalid label name">
pushing, 0 changed:   1%|▉                                                                                                                                                      | 1/162 [00:00<00:30,  5.24it/s]

Apparently, muted isn't available in the API!

I can add it to ignore_labels in local.py but we should note in the README that syncing muting isn't supported.

Use wd instead of wd/mail

Currently, gmi puts the maildir store into a subdirectory mail of the current working directory. This is suboptimal when you want to have your gmi-managed maildir and other maildirs (synced via mbsync or offlineimap, say) under the same maildir root:

mailroot/imapaccount/cur
mailrrot/imapaccount/...
mailroot/gmailaccount/mail/cur
mailroot/gmailaccount/mail/...

This ends up as folders imapaccount and gmailaccount at the root level, but all that gmailaccount has is an extra unnecessary subfolder mail that contains all mails (when imapaccount contains its imap folders).

gmi's json files need to be ignored by notmuch anyways. Therefore, I suggest to put the maildir directly under the workdir (omit /mail), maybe based on some config setting. This could help maildir based MUAs (though notmuch based is preferred, I know).

AttributeError on notmuch database object

Traceback (most recent call last):
  File "/usr/local/bin/gmi", line 8, in <module>
    g.main ()
  File "/home/maxking/Documents/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/home/maxking/Documents/gmailieer/lieer/gmailieer.py", line 307, in pull
    self.full_pull ()
  File "/home/maxking/Documents/gmailieer/lieer/gmailieer.py", line 538, in full_pull
    (rev, uuid) = db.get_revision ()
AttributeError: 'Database' object has no attribute 'get_revision'

Not sure what this means but I was trying to create a fresh maildir using gmailieer. I have python3.5.2 on Linux Mint 18.1.

.gmailieer.json destroyed when running out of space

When the disk is full, gmailier overwrites its file with an empty one and then throws an exception that the disk is full. It should probably write to a temporary file and rename it instead.

Exclude voicemail label in push

Hi. I hit an error trying to push the voicemail tag / label to gmail. I added it to the currently excluded labels in lieer/local.py and it resolved the problem. Here's a back trace:
push: creating label: voicemail.. Traceback (most recent call last): File "/usr/local/bin/gmi", line 8, in <module> g.main () File "/b/git/gmailieer/lieer/gmailieer.py", line 143, in main args.func (args) File "/b/git/gmailieer/lieer/gmailieer.py", line 193, in sync self.push (args, True) File "/b/git/gmailieer/lieer/gmailieer.py", line 246, in push actions.append (self.remote.update (rm, nm, self.local.state.last_historyId, self.force)) File "/b/git/gmailieer/lieer/remote.py", line 110, in func_wrap return func (self, *args, **kwargs) File "/b/git/gmailieer/lieer/remote.py", line 518, in update return self.__push_tags__ (gid, add, rem) File "/b/git/gmailieer/lieer/remote.py", line 110, in func_wrap return func (self, *args, **kwargs) File "/b/git/gmailieer/lieer/remote.py", line 534, in __push_tags__ (lid, ll) = self.__create_label__ (a) File "/b/git/gmailieer/lieer/remote.py", line 110, in func_wrap return func (self, *args, **kwargs) File "/b/git/gmailieer/lieer/remote.py", line 651, in __create_label__ lr = self.service.users ().labels ().create (userId = self.account, body = label).execute () File "/home/xxx/.local/lib/python3.5/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper return wrapped(*args, **kwargs) File "/home/xxx/.local/lib/python3.5/site-packages/googleapiclient/http.py", line 841, in execute raise HttpError(resp, content, uri=self.uri) googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/xxx/labels?alt=json returned "Invalid label name">
Can you commit this to master please?
Thanks :-)
-- Matt

Missing GMail labels

Similarly to #46 I've:

pull: full synchronization (no previous synchronization state)                                                                                    [3/8741]
fetching messages: 123866it [04:10, 414.05it/s]
receiving content:   0%|                                                                             | 3/57916 [00:00<9:22:58,  1.71it/s]Traceback (most r
ecent call last):
  File "/nix/store/5zb6zlyxg2xjz7p3kqr9hahl3vdbzm63-python3.6-gmaileer-v0.3-20171115/bin/.gmi-wrapped", line 9, in <module>
    g.main ()
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 307, in pull
    self.full_pull ()
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 530, in full_pull
    updated = self.get_content (message_gids)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 595, in get_content
    self.remote.get_messages (need_content, _got_msg, 'raw')
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 282, in get_messages
    batch.execute (http = self.http)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/googleapiclient/http.py", line 1437, in execute
    self._callback(request_id, response, exception)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 262, in _cb
    cb (resp)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 593, in _got_msg
    self.local.store (m, db)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 346, in store
    self.update_tags (m, p, db)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 354, in update_tags
    labels = [self.gmailieer.remote.labels[l] for l in labels]
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 354, in <listcomp>
    labels = [self.gmailieer.remote.labels[l] for l in labels]
KeyError: 'Label_3'

I did not create a label while it was running and I do not have Label_3 as label.

Is this a replacement of abunchoftags?

?

Fetching all the metadata every time?

Each time I run "gmi sync" it takes several minutes to fetch metadata for some 70,000 items. Is this intended behavior or is it supposed to be faster once the initial sync has been completed?

Ability to control logging output

Hi. I've used offlineimap for my personal email for a few years now and have recently switched to gmailieer for my gmail account. It's great! Thank you for all the hard work. I also run offlineimap on a system I created at work to harvest email and I would like to switch this to gmailieer too. The periodic mail sync on that machine is driven by cron. Offlineimap is configured on that machine to log to syslog. Since cron treats anything written to stdout as something it needs to notify me about, using gmailieer would incur me receiving email from this machine every mail sync cycle unless I do some redirect tweaking/hacking :-)
Offlineimap offers the ability to use a few different schemes for logging, eg syslog, tty and offers a quiet option too. I really like the gmailieer feedback using tqdm but would it be possible to introduce some sort of logging redirection, maybe just sending stdout to syslog? In that case, maybe the progress bar feedback wouldn't be as important.
Cheers,
-- Matt

gmi pull keeps crashing

$ gmi pull                                                                                                                                                              /data/Mail/personal
pull: full synchronization (no previous synchronization state)
fetching messages: 139863it [12:33, 153.90it/s]                                                                                                                receiving content: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 57/57 [02:58<00:00,  3.28s/it]receiving metadata:   3%|██▌                                                                                             | 3683/139806 [01:14<38:10, 59.42it/s]Traceback (most recent call last):
  File "/usr/bin/gmi", line 4, in <module>
    __import__('pkg_resources').run_script('gmailieer==0.2', 'gmi')
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 742, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1510, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/EGG-INFO/scripts/gmi", line 8, in <module>
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 136, in main
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 307, in pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 531, in full_pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 562, in get_meta
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 100, in func_wrap
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 271, in get_messages
  File "/usr/lib/python3.6/site-packages/oauth2client-4.1.2-py3.6.egg/oauth2client/_helpers.py", line 133, in positional_wrapper
  File "/usr/lib/python3.6/site-packages/google_api_python_client-1.6.3-py3.6.egg/googleapiclient/http.py", line 1464, in execute
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 251, in _cb
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 560, in _got_msg
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/local.py", line 326, in update_tags
lieer.local.RepositoryException: tried to update tags on non-existant file: /data/Mail/personal/mail/cur/12e10631fc4211b6:2,

This is right after the initial pull (which also crashed with similar stack trace).

synchronization state not saved while first pulling

I have a lot of email, and syncing for the first time is a problem, due to Gmail throttling.

I try to cancel the first run, to get reset at the Gmail end and avoid the throttling, but I'm back in the first situation again, and I will end up with duplicate files on disk:

(venv) 20:13 CHARLIE~/Maildir/[email protected][[email protected]] % PYTHONPATH=~/src/notmuch/bindings/python/  ~/src/gmailieer/gmi pull    (chp02-rsvp|✚23…)
pull: full synchronization (no previous synchronization state)
fetching messages: 557143it [19:02, 487.47it/s]
receiving content :   0%|                                                                                             | 129/557143 [00:17<13:22:16, 11.57it/s]
reducing batch request size to: 100
reducing batch request size to: 50
receiving content :  30%|██████████████████████████▎                                                              | 164746/557143 [4:08:49<9:26:39, 11.54it/s]
reducing batch request size to: 25

^Creceiving content :  34%|██████████████████████████████▎                                                          | 190047/557143 [5:23:08<9:19:35, 10.93it/
s]Traceback (most recent call last):
  File "/Users/charlie/src/gmailieer/gmi", line 8, in <module>
    g.main ()
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 272, in pull
    self.full_pull ()
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 488, in full_pull
    updated = self.get_content (message_ids)
  File "/Users/charlie/src/gmailieer/lieer/gmailieer.py", line 556, in get_content
    self.remote.get_messages (need_content, _got_msg, 'raw')
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 82, in func_wrap
    return func (self, *args, **kwargs)
  File "/Users/charlie/src/gmailieer/lieer/remote.py", line 204, in get_messages
    batch.execute (http = self.http)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/googleapiclient/http.py", line 1417, in execute
    self._execute(http, self._order, self._requests)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/googleapiclient/http.py", line 1349, in _execute
    headers=headers)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/httplib2/__init__.py", line 1322, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/httplib2/__init__.py", line 1072, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/Users/charlie/src/gmailieer/venv/lib/python3.5/site-packages/httplib2/__init__.py", line 1054, in _conn_request
    content = response.read()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 455, in read
    return self._readall_chunked()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 561, in _readall_chunked
    value.append(self._safe_read(chunk_left))
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 607, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
KeyboardInterrupt

Then I try again, without the throttle:

(venv) 9:26 CHARLIE~/Maildir/[email protected][[email protected]] % PYTHONPATH=~/src/notmuch/bindings/python/  ~/src/gmailieer/gmi pull     (chp02-rsvp|✚23…)
pull: full synchronization (no previous synchronization state)
fetching messages:  24%|███████████████████████▊                                                                          | 800/3296 [00:02<00:05, 430.98it/s]

Can we sync state during the initial pull, and then resume the pull?

Speed up initial download

My personal email goes back over 10 years, and I'd rather not have all of that downloaded on my laptop (even setting the max messages per label in Gmail doesn't help with that, because I have gained way too many labels over the years). offlineimap allows you to sync mails from certain 'folders'. Would be good if gmailieer allows that.

Updating tags on non-existent file

After performing a sync of my pretty large mailbox, when updating tags, I consistently see an exception about failing to update tags on a non-existent file. I think gmailieer may have somehow got the file name wrong, but I'm not sure... Here's the trace:
...
resolving changes: 100%
receiving content: 100%
updating tags (0Δ): 0%| | 0/13602 [00:00<?, ?it/s]Traceback (most recent call last):
File "/usr/local/bin/gmi", line 8, in
g.main ()
File "/b/git/gmailieer/lieer/gmailieer.py", line 145, in main
args.func (args)
File "/b/git/gmailieer/lieer/gmailieer.py", line 215, in sync
self.pull (args, True)
File "/b/git/gmailieer/lieer/gmailieer.py", line 341, in pull
self.partial_pull ()
File "/b/git/gmailieer/lieer/gmailieer.py", line 492, in partial_pull
r = self.local.update_tags (m, None, db)
File "/b/git/gmailieer/lieer/local.py", line 401, in update_tags
raise Local.RepositoryException ("tried to update tags on non-existant file: %s" % fname)
lieer.local.RepositoryException: tried to update tags on non-existant file: /path/to/toplevelmaildir/mail/cur/162f6b5441a3ab90:2,

The file "/path/to/toplevelmaildir/mail/cur/162f6b5441a3ab90:2," does not exist in the mail store, but file "/path/to/toplevelmaildir/mail/cur/162f6b5441a3ab90:2,S" does.

Could it be possible that gmailieer has had a problem parsing this file name?

notmuch search for "/path/to/toplevelmaildir/mail/cur/162f6b5441a3ab90:2,S" correctly identifies the email and shows the tags applied to it. The message only has one mime part which is text/html (no text/plain part).

Also, is there a particular reason why this exception hangs gmailieer? I think it would be better if it exited.
-- Matt

Weird corner cases with neomutt moving unread emails from cur/ to new/

So, (neo)mutt will apparently move unread emails from cur/ to new/ when marking them as trash (for example, any other label should do the trick as well), gmailieer then just says: Not in the repo.

Problems using notmuch database on a path containing symbolic links

For some reasons particular to my setup, my notmuch database is configured using a path containing a symbolic link. Notmuch does not complain about that and works like a charm.

However, gmailieer does not like symbolic links and always tries to use the real path to the database.

https://github.com/gauteh/gmailieer/blob/f610557c2964e1e3c628e04276b45d20c8ba7066/lieer/local.py#L145

Even if the paths are logically the same, the real path string is evidently different from the path (that contains the symbolic link) that I configured notmuch with. Thus I always, incorrectly ,get the following message.

local mail repository not in notmuch db

For the time being I created a script that, using sed, alters notmuch configuration and after gmailieer is done goes back to the original configuration. It is not ideal, but works for me.

I was wondering, could the code above be altered to try and match the symbolic link path with the absolute one?