rgladwell / imap-upload Goto Github PK

Hi,
I have an error if my folder contain a name with accent: Foncière.mbox
Error message:
An unknown error has occurred [493]: 'ascii' codec can't decode byte 0xc3 in position 51: ordinal not in range(128)

The only tagged release is 1.0.

Hello!
I backed up an mbox (via Google Takeout) to migrate 89,000 emails from one account to another.

Due to an issue, I interrupted the script's process, and now I want to know if running it again will duplicate the emails.
My command is as follows:

python imap_upload.py --ssl [email protected] --password="password" --host=imap.gmail.com --port=993 --dry-run --error="errores.mbox" --google-takeout "original.mbox"

Thank you in advance for your help, and for building this great tool!

There doesn't seem to be a way to import emails to google workspace to a specific sub-label. I've tried using the --box command which creates the label, but the emails end up in the main inbox/drafts/important/sent mail folders. Any way to avoid this and just upload all to a label and have it create the sub-labels under that based on structure? Example structure of Mac Mail mbox eport below:

Just list mail boxes!
"INBOX""
"[Gmail]"" #Noselect
+- "All Mail"" #All
+- "Drafts"" #Drafts
+- "Important"" #Important
+- "Sent Mail"" #Sent
+- "Spam"" #Junk
+- "Starred"" #Flagged
+- "Trash"" #Trash

I have 20GB .MBOX file exported from Google g-suite, but importing using roundcube or horde not possible for files bigger than 2GB.

Can I using this tool to import MBOX to cPanel?

On the CLI, I saw the following error after python imap_upload.py --gmail --box imported mbox_file.mbox:

Username: [email protected]
Password: ........
Connecting to imap.gmail.com:993.
IMAP4 error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)

The error is related to what Google calls "Less secure apps". I fixed it by going to My Account -> Security and then choosing "Turn on access (not recommended)":

I'm using recursive, but I see in line 475 for recursive uploads, it sets the "BOX" to "" - is this by design? For my needs, I've changed this to do the same as non-recursive; options["box"] instead of ""

Thanks your project is working great for me so far!

The new -r option introduced in v1.3 does recurse into local directories but not recreate the same structure on the imap server, if I'm not mistaking. Is this intentional?

Thanks for the great work!

Hello and thanks for this tool.

I keep getting this error:

NG (Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.)

I'm trying this with an empty mailbox that I set up for testing purposes. I'm positive that INBOX is the correct IMAP prefix to use. --list_boxes gives me this result:

"INBOX"                                 
  +- "Sent Messages""                   #UnMarked
  +- "Drafts"                           #UnMarked	#Drafts

Any idea on what the problem might be?

Great work!

I am trying to use this tool to upload my emails from Thunderbird (68.10) on Ubuntu 18.04 to Gmail (Google Worksapce). While setting up Thunderbird for Gmail (IMAP), I realized that Gmail nowadays only allows authentication via OAuth2. Because of this reason, when I try this script with --username and --password it fails with the following error:
Connecting to imap.gmail.com:993. IMAP4 error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)
Is there a workaround?

On a mailcow mail setup, folders are located parallel to INBOX:

Just list mail boxes!
"Test"                                  #UnMarked
"Archive"                               #UnMarked	#Archive
"Trash"                                 #Marked	#Trash
"Junk"                                  #UnMarked	#Junk
"Sent"                                  #UnMarked	#Sent
"Drafts"                                #UnMarked	#Drafts
"INBOX"                                 

on a Hetzner Mailserver (Level 9 Webhosting) all folders are below INBOX

"INBOX"                                 
  +- "test"                             #Marked
  +- "Trash"                            #UnMarked	#Trash
  +- "spambucket"                       #UnMarked	#Junk
  +- "Sent"                             #Marked	#Sent
  +- "Drafts"                           #UnMarked	#Drafts
  +- "Templates"                        #UnMarked

On the latter, creation of folders and append fails (using --google-takeout) with

Traceback (most recent call last):
  File "/home/leonard/src/imap-upload-poldy79/imap_upload.py", line 426, in upload
    raise Exception(r2[0]) # FIXME: Should use custom class
Exception: b'Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.) (0.001 + 0.000 secs).'
)

I am not sure if this is already covered by imap-upload.py and I have overseen something or if this is not yet supported.
If I prefix the path for imap.create_folder and imap.append with "INBOX" everything works fine.

I could provide a solution, might be a switch --everything-below-inbox ? What do you think?

I tried the script and, even though I disabled the Less secure app access, I get this error,

python imap_upload.py --gmail --list_boxes
IMAP4 error: [ALERT] Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)

What is going on?

As --google-takeout-language has a default 'en', the code suspects, that the option was set by the user and complains, that --google-takeout was not supplied. So currently (684cd23) --google-takeout is a mandatory option.

Do you know how to check if the user had supplied --google-takeout-language on cmdline or if it is just the default?
We might could just ignore google-takeout-language, if google-takeout was not supplied... suggestions?

Hello,

at first I want to thank you for this great tool - it works great!

...Well, until it meets an exception.

Configuration: I have Thunderbird 52.2.1 installed on Windows. I come from the German speaking world, so Umlauts (Ä / Ö / Ü / ß) are quite common in my e-mails. The mails go back to 2011 - I used Office Outlook back then, but imported all of them (they work well in Thunderbird).

My folder-structure is something along:

• Inbox
  • Subfolder 1
  • Subfolder 2
    • Sub-Subfolder 1
    • Sub-Subfolder 2
  • ...

and so on.

I tried both branches of your tool, with thunderbird-support working at least for greater time-spans.
As I thought it would be easier, I used my Ubuntu (16.04) VM, copied the thunderbird-folder to it and used the following code:

python imap_upload.py --gmail --box=test [email protected] --password=mypassword --retry=10 -r /custom_thunderbird_folder/Profiles/default/Mail/POP-client/Inbox.sbd

That will (successfully!) upload most of my e-mails - however, some certain subfolders seem to make troubles:
After finishing one subfolder, it almost immediately states:
Found mailbox at /custom_thunderbird_folder/Profiles/default/Mail/POP-client/Inbox.sbd/[subfolder-name].mozmsgs/md5[...].wdseml...
Uploading to [subfolder-name]...
Counting the mailbox (it could take a while for the large one).
Done. (OK: 0, NG: 0)
An unknown error has occurred: 'ascii' codec can't decode byte 0xc3 in position 116: ordinal not in range(128)
(Things in [ ] brackets were changed by me.)

I can reproduce the error, and it always stays the same (and on the same spot). If I delete the subfolder, it may find another subfolder that throws the same error.
I tried to see if I can fix it myself by looking for mentions of "ascii", "utf", "code" and "ordinal" in your script and replacing them with "utf-8" where it seemed to make sense. However, that did not make any difference, so I would assume that it has to do with one of the imported modules.

So I'm out of ideas. Do you have any idea about what the problem might be?

I will gladly upload any sample here IF there is a way to first delete sensitive data - However, I fear that manipulating the files will lead to other errors... Of course, I will do what I can to give you all the information you need - just tell me what it is.

Any help is appreciated - thanks in advance!

https://github.com/rgladwell/imap-upload/blob/master/imap_upload.py#L414
reference msg.boxes
but msg.boxes is assigned at
https://github.com/rgladwell/imap-upload/blob/master/imap_upload.py#L394
only for google_takeout usage

When an mbox or directory name in a scanned path contains a non-ASCII character the script fails with the following error:

An unknown error has occurred [493]:  'ascii' codec can't decode byte 0xcc in position 62: ordinal not in range(128)

Tested with this folder structure:

Archiv
  └── testing
    ├── Drehbücher
    │   ├── Drehbücher\ 2008.mbox
    │   ├── Drehbücher\ 2009.mbox
    │   ├── Drehbücher\ 2010.mbox
    │   ├── Drehbücher\ 2011.mbox
    │   ├── Drehbücher\ 2012.mbox
    │   ├── Drehbücher\ 2013.mbox
    │   ├── Drehbücher\ 2014.mbox
    │   ├── Drehbücher\ 2015.mbox
    │   ├── Drehbücher\ 2016.mbox
    │   ├── Drehbücher\ 2017.mbox
    │   ├── Drehbücher\ 2018.mbox
    │   ├── Drehbücher\ 2019.mbox
    │   ├── Drehbücher\ 2020.mbox
    └── Drehbücher.mbox

Happy to test further and provide more feedback if needed.

Thanks for this tool!

I just ran the script with the following command, on an .mbox file from Google Takeout containing approximately 7,000 emails:

$ python3 imap_upload.py --gmail --box imported takeout.mbox

It seems to have got stuck on an email with a subject line containing an ð ("eth") character. The full subject line is "FW: Youtube Job Wants You 👉 $20K/Month Potential! 80272150" (yes, it appears to be a spam message).

Is there anything I can do to recover from this? If I run the script a second time, will it upload duplicate emails?

I have tried a variety of different options to upload my Google Takeout mbox to my IMAP account. No matter what I do I get:

to [Opened,Category Updates] NG (b'Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX

Here is the command line I am using. Any suggestions would be greatly appreciated...
python3 imap_upload.py --ssl --user=[email protected] --password='test' --host=imap.dreamhost.com --port=993 --error='All mail Including Spam and Trash_errors.mbox' --box='INBOX' --google-takeout --google-takeout-box-as-base-folder 'All mail Including Spam and Trash-002.mbox'

Hi,
Here is my local structure :

• MASTER

• FOLDER1.mbox
• FOLDER1.sbd
  -- SUBFOLDERA.mbox
  -- SUBFOLDERB.mbox
  -- SUBFOLDERB.sbd
  --- LASTFOLDER.mbox

When I try : python imap_upload.py --host=imap.myhost.com --port=143 --user=[email protected] --password=password -r /FOLDER/MASTER

Inbox go to Inbox
MASTER folder is create

But FOLDER1 is not created, either SUBFOLDERB and all mails are going to MASTER.

Any ideas ?

Thanks for the work !

When trying to send emails to an account that uses cpanel, the following error occurs.

(Mailbox name should probably be prefixed with: INBOX.)

This version prematurely exists without any error message if I give it the right credentials to gmail. If I purposely give it wrong login data it displays an error message. (tried with --gmail and with imaps://imap.gmail.com/ ). Not tested with other providers.

I think that ImapUploadMessage.as_string function should be used also for non Google Takeout uploads to fix encoding problems of some messages.

If I import single file without extension it works however recursive option not work. Thunderbird stores mailboxes on Windows without giving them extension so with large number of mailboxes it can be pain to change them all manually. Is it any way for a program to detect mboxes anyway?

However even with that option script seems to stop after processing first folder

322/322   5.3 kB  Nowe wiadomości w skrzynce       OK (1 sec)
Done. (OK: 322, WARNING: 0, ERROR: 0)
Found mailbox at /Volumes/T5/LocalFolders/Archives/2011.mbox...

Tried pointing it at Local Folders and various .sbd folders, it connects to gmail, then there's no output, and it stops in about 2 seconds. Problem is that Thunderbird doesn't add .mbox to the end of the file names. When this is fixed, it works recursively, but doesn't seem to understand the folder hierarchy beyond one level, dumping a bunch of messages into parent folders.

If you're reading this and trying to use this script on linux, here's a helpful bash one-liner to rename all the files without extensions to .mbox. If it renames a file that isn't an mbox file, no big deal, the script will just skip it. Obviously change path to your directory

find /path -type f -not -name "." -exec mv "{}" "{}".mbox ;

Thanks for the script, it's really useful for large mbox files, but not so much for complex hierarchies.

Using current master (684cd23) with python 3.9.2 on debian bullseye with IMAPClient==2.2.0 option --list_boxes requires --google-takeout. Furthermore it then fails with

IMAP Upload (v2.0.0)
Connecting to mail.your-server.de:993.
Just list mail boxes!
An unknown error has occurred [679]:  cannot use a string pattern on a bytes-like object

It seems that uploader.list_boxes() returns the list of boxes as a byte stream, which has to be decoded to a string.

I suspect, that this is because of a change in IMAPClient. You might add a requirements.txt which states the latest version known to work.

Hi,

how can Import from MBOX Google Take Out Gmail to IMAP4 server?

I am running python 3.8 in a Windows 10 machine.

I have changed the headers to

#!/usr/bin/env python3.8

but I keep on getting this error:

Python .\imap_upload.py
File ".\imap_upload.py", line 213
except Exception, e:
^
SyntaxError: invalid syntax

Any way to solve it?

See https://github.com/chriskiehl/Gooey. I tried to get a build working locally, but failed when I got to the pyinstaller -F --windowed build.spec part under https://github.com/chriskiehl/Gooey#packaging. Not too familiar with Python, but thought that you could benefit from adding Gooey to this project so that it would have widespread ease of use (and users wouldn't need to download python/etc).

I was stumped for awhile, thinking the script was not grabbing the dates for my emails correctly, but it was actually an issue with how google handles multiple copies of the same email (apparently it keeps the metadata from the first one for all copies).

What happened was:

1. Upload an mbox (which just happens to have the wrong datetimes in the default from time field.
2. Notice that the date is wrong in gmail.
3. Trash email on gmail.
4. Try the script again with the correct (for my use case) time-fields=date,received parameter.
5. Notice on google that message has been re-uploaded, but still has the wrong date!

The issue was that the first/wrong message was still in my Trash on google, and not actually deleted. Soo, when I was trying the re-upload with the correct date, I think the google servers see the duplicate and simply un-trash it (complete with the original incorrect date).

Anyway, maybe you could add a mention of this behavior to your docs? Maybe it'll help other people in who encounter the same situation.

Thanks for the AWESOME work!

I want to use this script to import mbox file created via Google Takeout service. They export all mail as a single mbox file. However, each message has an additional header specifying to what folder the message belongs to. E.g:

X-Gmail-Labels: Archived,Sent

It would be great if this script can handle these.

it looks like based on this snippet of code and confirmed by my server logs, that on each email upload it tries to create the dest folder (either INBOX or --box) i presume. Can the code be modified to check for the existence of the folder first and if it doesn't exit then create it

383         try:
384             self.open()
385             self.imap.create(box)
386             return self.imap.append(box, [], delivery_time, message)