Copying "sent messages", "Sender" is empty about imapsync

[retorquere]

correction: this happens only sometimes on the sent messages folder, not always, for messages that were copied in the same run.

[gilleslamiral]

Hi Retorquere,

Do you mean
a) The sender exists in the message on host1 but it disappeared in the transfer.
or
b) No sender in the message on host1 but you want one on host2.

By sender, do you mean "From:" header line?

Le 12/01/2016 20:45, retorquere a écrit :

I'm copying two folders using imapsync; one regular folder, which copies over perfectly, and one where I copy |Sent Messages| (from Exchange IMAP) to |[Gmail]Sent Mail| (GMail). In the second case, where I am the sender, the messages are copied over but there's no sender in the copied message.
Would it be possible to specify a default sender if it is empty?

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

[retorquere]

Yes, I mean the "From:" header line. I honestly don't know which of the two it is; host1 is Exchange IMAP, and I don't know how it serves its messages, but since they come from the "Sent items" folder, the sender is always going to be myself. But I'd be more inclined to think a), since some of the copied messages do have a sender.

[gilleslamiral]

Le 13/01/2016 14:08, retorquere a écrit :

Yes, I mean the "From:" header line. I honestly don't know which of the two it is;
host1 is Exchange IMAP, and I don't know how it serves its messages,
But I'd be more inclined to think a), since some of the copied messages /do/ have a sender.

When you look at a message missing a sender on host2, does the same message on host1 miss also a sender?

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

[retorquere]

Can I safely run the copy more than once? The name would imply yes, but I've been using imapsync to move messages so far.

[gilleslamiral]

Yes.
Add --dry on the first resync if you want to be sure there won't be duplicates.

On 15/01/2016 23:18, retorquere wrote:

Can I safely run the copy more than once? The name would imply yes, but I've been using imapsync to move messages so far.

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

[retorquere]

But then how will I be able to tell whether the copied messages have senders or not? Dry-run won't actually copy the messages right? I see there's a filter option where I can pass messages to a script, can I use that in some way to copy the messages in transit for debugging?

[gilleslamiral]

Hi,

Le 16/01/2016 11:46, retorquere a écrit :

But then how will I be able to tell whether the copied messages have senders or not?

No. This --dry advice is to be sure imapsync doesn't generate duplicates, since you asked "Can I safely run the copy more than once?"

Dry-run won't actually copy the messages right?

Right.

I see there's a filter option where I can pass messages to a script, can I use that in some way to copy the messages in transit for debugging?

You can see the message content with --debugcontent

imapsync ... --debugcontent

We haven't answered yet if the issue is
a) The sender exists in the message on host1 but it disappeared in the transfer.
or
b) No sender in the message on host1 but you want one on host2.
or both, or something else.

A solution, if any, will depend on this answer.

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

[retorquere]

Cool -- I've added --debugcontent to the flags; when I see a message without sender, I'll get you the relevant log.

[retorquere]

There really isn't a sender in the messages (the "From:" line is missing), which seems to be Exchange's way of saying "the mailbox owner sent this". I've added this pipemess command and that seems to fix it:

#!/home/emile/.rvm/wrappers/ruby-2.2.1@misc/ruby
require 'mail'
message = Mail.read_from_string(STDIN.read)
message.from = '[email protected]' if message.sender.nil?
puts message.to_s

[gilleslamiral]

Hi retorquere,

Great and clear small code!

On 16/02/2016 16:20, retorquere wrote:

There really isn't a sender in the messages (the "From:" line is missing), which seems to be Exchange's way of saying "the mailbox owner sent this". I've added this pipemess command and that seems to fix it.

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

[retorquere]

It seems to be happening also for other messages. Is there a way for me to leave messages in place when I detect certain characteristics, like can I have pipemess return a non-zero exit value and have imapsync behave like it's in dry-run mode for those? That would give me a repeatable way to test my way through this.

[gilleslamiral]

Hi retorquere,

It seems to be happening also for other messages.

What's the problem?

Is there a way for me to leave messages in place when I detect certain characteristics,

Yes.

like can I have pipemess return a non-zero exit value and have imapsync behave like it's in dry-run mode for those?

This is currently the case.
Just exit any pipemess command with non-zero value and imapsync will consider it a failure and then will not transfer the message.

That would give me a repeatable way to test my way through this.

Have a try with:

imapsync ...--pipemess /bin/false

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

[retorquere]

I don't quite know yet what the problem is, other than the more general problem "Exchange is a really odd IMAP server". I'm seeing messages that I know are being sent by a colleague but which have no sender. I'm looking to isolate these messages so I can device a strategy to fix them.

I'll let you know what I find.

[retorquere]

My bad; the test in the script was in error. It needs to be:

#!/home/emile/.rvm/wrappers/ruby-2.2.1@misc/ruby

require 'mail'

data = STDIN.read

exit(0) if data.strip == ''

message = Mail.read_from_string(data)
if message.from && message.from.length > 0
  puts data
else
  message.from = '[email protected]'
  puts message.to_s
end

so it is still the case that Exchange sends out "sent by me" without the "from" (which is pretty dense), but it is only doing this for "sent by me".

[gilleslamiral]

Ciao Retorquere,

I tried it but it fails on
require 'mail'

What Ruby library provides 'mail' package?

Le 25/02/2016 17:28, retorquere a écrit :

My bad; the test in the script was in error. It needs to be:

|#!/home/emile/.rvm/wrappers/ruby-2.2.1@misc/ruby require 'mail'

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

[retorquere]

the mail package, installable using gem install mail if you have bundler (recommended). I'm sure there are perl packages that can do the same, but I'm not familiar with perl. In all honesty, this also ought to work (untested though), and doesn't rely on any packages:

from = false
STDIN.lines.each{|line|
  if line.strip == ''
    write "From: [email protected]\r\n" unless from
    from = true
  elsif line =~ /^From: /
    from = true
  end

  write(line)
}

[gilleslamiral]

Ciao,

It looks like what's needed is a Ruby package named 'mail'.
Perl or classical mail packages are something else.

Le 27/02/2016 15:14, retorquere a écrit :

the |mail| package, installable using |gem install mail| if you have bundler (recommended). I'm sure there are perl packages that can do the same, but I'm not familiar with perl. In all honesty, this also ought to work (untested though), and doesn't rely on any packages:

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

[retorquere]

That is correct; it is a ruby gem, but what I meant to say is that since imapsync is written in perl, others may want to use that in stead of ruby; I don't know perl, but I'm sure there must be perl packages that do what the mail gem does.

[gilleslamiral]

I've put the tested (not by me) ruby code in the file W/tools/addFromIfMissing
I guess the Ruby "mail" gem is
https://github.com/mikel/mail

I don't know how to install a ruby gem...

[retorquere]

If ruby is installed, gems can be installed using "gem install [gem name]"

[gilleslamiral]

Thanks Emiliano!
The command gem install mail worked like a charm, while a strangely bit long during
Parsing documentation for mail-2.7.0. (I was near to kill it...)
Now I've tested your code and it works fine!
I guess next improvement will be to make the email address a parameter.

{ echo X-Fake: blabla ; echo ;  echo "Hello World!" ; } | ./addFromIfMissing
Date: Fri, 31 Aug 2018 13:18:41 +0200
From: [email protected]
Message-ID: <[email protected]>
Mime-Version: 1.0
Content-Type: text/plain;
 charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Fake: blabla

Hello World!

Current code

#!/usr/bin/ruby

# $Id: addFromIfMissing,v 1.6 2018/08/31 11:21:54 gilles Exp gilles $

# Credits: Emiliano Heyns

# https://github.com/imapsync/imapsync/issues/54
# https://github.com/mikel/mail

# Test example in command line:
# { echo  "Date: Fri, 31 Aug 2018 12:58:59 +0000"  ; echo ;  echo "Hello World!" ; } | ./addFromIfMissing

# Add mail gem with the command:
#   gem install mail

require 'mail'

data = STDIN.read

exit(0) if data.strip == ''

message = Mail.read_from_string(data)
if message.from && message.from.length > 0
  puts data
else
  message.from = '[email protected]'
  puts message.to_s
end

[retorquere]

this one works in python2 and python3 (more likely to already be installed on systems) and needs no extra packages:

#!/usr/bin/env python
import email
import sys

msg = email.message_from_string(sys.stdin.read())
if msg['from'] is None:
  msg['From'] = sys.argv[1]
print(msg)

[retorquere]

(the python script demands the email address as the parameter)

[gilleslamiral]

Ciao Emiliano,

Thanks for your input!

It doesn't work well for me, it adds a From line (mbox format) that will hurt most imap servers.
Do you encounter the same behavior?

{ echo  "X-Fake: blabla"  ; echo ;  echo "Hello World!" ; } | ./addFromIfMissing2 [email protected]
From nobody Sun Sep  2 00:53:54 2018
X-Fake: blabla
From: [email protected]

Hello World!