Mac mail.app, and iPhone mail.app: Sometimes there's weird characters in a diff about git-commit-notifier

[akzhan]

@krunkosaurus, large commits will be handled correctly this month.

About your trouble, - this is bug of native Apple mail clients. Gmail, Thunderbird, The Bat, Outlook, Perl software etc., - parses emails correctly.

[akzhan]

Our managers who look at commit emails of GCN through native Mac app have this trouble too.

[bitboxer]

I have seen this, too. But I don't understand why this is showing. It's only showing on a few mails, most of them are displayed correctly. This started with the switch to premailer.

[akzhan]

Today my email (written by hands in GMail) has been incorrectly parsed by mac.app.

All other known mail clients read it ok.

So I mention that's not a bug of GCN.

[bitboxer]

No, I don't think it's not a bug in GCN...I think its both. A problem in mail.app and in GCN. Something triggers this misinterpretation of the mail encoding. We should fix this somehow :(

[jdberry]

This is definitely a bug in GCN, or in one of the libraries it uses. I've looked at the raw mail messages that cause this problem to occur: in each of the cases I've seen, they contain invalid UTF8 characters. Mac OS Mail looks at the mime body, tries to read it as UTF8, and gives up, because it is not valid UTF8. The funny characters seen are thus the result of displaying the (mostly valid) UTF8 as something else as a fallback.

In the cases I've seen, these invalid characters occur when a UTF8 non-breaking space sequence 0xC2 0xA0 is broken into two lines, with 0x0A 0x20 inserted in between. During processing GCN (or one of the libraries it calls), changes 0xC2A0 --> 0xC20A20A0 in order to break the long line.

Here's a hex dump of an example from a commit of jquery 1.7.1 minimized. Note that the problem occurs at offset 28793:

00028780 28 61 2c 62 29 7b 62 3d 62 7c 7c 22 66 78 22 3b |(a,b){b=b||"fx";|
00028790 76 61 72 c2 0a 20 a0 63 3d 66 2e 71 75 65 75 65 |var.. .c=f.queue|
000287a0 28 61 2c 62 29 2c 64 3d 63 2e 73 68 69 66 74 28 |(a,b),d=c.shift(|

Note that minimized jquery has very long lines, and these lines do need to be split at some point in order to be valid for email. I'm not sure who is splitting them, however (it could even be the mailer?), but whoever it is that is doing it is splitting in the middle of a valid UTF8 sequence, by adding 0x0a20, and in doing so, is causing the valid sequence 0xC2A0 to become the invalid sequence 0XC20A20A0.

[akzhan]

Wow. Thanks for details. AFAIK, it should be premailer issue.
Create it in alexdunae/premailer please. I will try to create assocuated pull request when have free time.

[jdberry]

I've looked a little bit at Premailer: I don't think that it does any line splitting. It probably returns the very long lines back to you? And if you aren't splitting them, then it must be sendmail/postfix that is splitting them? If so, then I think this could be fixed by using either base64 or quoted-printable encoding for the outgoing message, with appropriate line-breaking. quoted-printable is more readable as a raw message.

[bitboxer]

Look into my fork of premailer. That will fix that issue i think

On Saturday, December 3, 2011, James Berry <
[email protected]>
wrote:

I've looked a little bit at Premailer: I don't think that it does any
line splitting. It probably returns the very long lines back to you? And if
you aren't splitting them, then it must be sendmail/postfix that is
splitting them? If so, then I think this could be fixed by using either
base64 or quoted-printable encoding for the outgoing message, with
appropriate line-breaking. quoted-printable is more readable as a raw
message.

---

Reply to this email directly or view it on GitHub:

https://github.com/bitboxer/git-commit-notifier/issues/83#issuecomment-3002654

[jdberry]

bitboxer: I don't think that's going to help. Your fork changes something about input encoding, but nothing about the output, mail transfer encoding, correct? I think the basic problem here is the GCN is putting out illegally long lines, which the mail transfer agent (sendmail, etc) is being forced to break in order for the email to be valid. To fix that problem, GCN should be properly breaking lines. Using a mail transfer encoding of base64 or quoted-printable will make that easy to do.

[akzhan]

@jdberry, can you pull request on this?

[jdberry]

@jdberry, can you pull request on this?

@akzhan: not sure what you're asking. I don't have a fix to issue a pull-request for.

[jdberry]

Note that another way to fix this would be for GCN to simply truncate long lines at some reasonable length, like 500 characters. Such line lines are pretty meaningless for the purposes of a commit email. This could be done fairly early in the process, and might resolve some of the other performance issues as well...

Note that the suggested approach of using quoted-printable encoding would, I believe, result in that jquery file being displayed as a single very, very long line, which wouldn't do anybody much good. The better solution is probably to think of an extremely long line as something akin to a binary file: not very useful, and something that could be truncated.

[jdberry]

See my pull request for a fix to this problem. I ended up doing two things: limiting line lengths from the diff to 512 characters, which alone would help solve this problem, and also improves performance, and using quoted-printable encoding of the message, which solves this problem.

[akzhan]

There is another problem found - massive broken specs on some rubies. Will be reviewed.

[akzhan]

Thanks for the patch. It works.