monde / mms2r Goto Github PK

Hi, just ran into very strange bug when media filename? returned filename with newline at the end. Email was sent from Zimbra system. Quick fix was to strip output of newline with adding following code as Rails initialializer:

module MMS2R
  class MMS2R::Media
    def filename_with_strip?(part)
      filename_without_strip?(part).strip
    end
    alias_method_chain :filename?, :strip
  end  
end

Whenever I call MMS2R::Media.new(email) I get the following with the latest version of the gem:

NoMethodError: undefined method return_path' for #<TMail::Mail:0x1068155b8> /Users/jlippiner/.rvm/gems/ree-1.8.7-2010.01@tdt/gems/mms2r-3.0.1/lib/mms2r/media.rb:169:indomain'
/Users/jlippiner/.rvm/gems/ree-1.8.7-2010.01@tdt/gems/mms2r-3.0.1/lib/mms2r/media.rb:153:in create' /Users/jlippiner/.rvm/gems/ree-1.8.7-2010.01@tdt/gems/mms2r-3.0.1/lib/mms2r/media.rb:111:innew'

New version is working a charm with Mail.

However I am having a challenge with a couple of messages with the @media={"text/html". What method do I use to print out the html version of the text? Or do I have to open the associated temp file to get at this?

Thanks
Paul

Hi,

I am using the example code off of the main project page. I have MMS2R 3.1 and Mail 2.2.5 installed. When I run this code:

mail = MMS2R::Media.new(Mail.read('./test.video-tmobile.eml'))

mail.default_media is nil and mail.media an empty Hash. I've been troubleshooting this for a couple of days and I'm not making much progress. Let me know if you need to see the .eml file, it's rather large and I'd rather not junk up this post with that content.

Thanks

Hello,

I am using mms2r/tmail with my rails application to process email received and I am very happy with it.
Today one of my user encountered a weird problem where 2 attached .pdf files were not process. I looked at the source of the mail and they both had a "application/octet-stream" type.

looking at the code in media.rb I noticed that on the process_media method (lines 404 to 426) if I comment the test on the 'application/octet-stream' to just keep the line 416
"type = part.content_type"
it works correctly and the 2 pdf files are processed

Do you have any idea on what's happened?

Thanks in advance
Nicolas

I've been having some strange things happen with some multipart messages.

The message is: http://gist.github.com/472080

and the stack trace is:

/usr/local/lib/ruby/gems/1.8/gems/mail-2.2.4/lib/mail/field.rb:122:in `send': undefined    method `filename' for #<Mail::UnstructuredField:0x1020051b0> (NoMethodError)
from /usr/local/lib/ruby/gems/1.8/gems/mail-2.2.4/lib/mail/field.rb:122:in `method_missing'
from /usr/local/lib/ruby/gems/1.8/gems/mail-2.2.4/lib/mail/message.rb:1849:in `find_attachment'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mail_ext.rb:33:in `filename'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:507:in `filename?'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:471:in `temp_file'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:405:in `process_media'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:358:in `process'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:355:in `each'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:355:in `process'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:233:in `initialize'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:113:in `orig_new'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:113:in `new'
from mmstest.rb:7

It seems like it doesn't like dealing with this kind of multipart message. Anyone else run into this issue, and have a fix for it?

Hey @monde,

Just ran into a small issue with Sprint (again). The issue is that the JSON returned from the Sprint Ajax is returning   instead of a regular space. This causes problems with using ruby string methods like strip as they don't expect the NBSP instead of utf8/ascii.

So, my question is: Should the code I submitted recently be ensuring that the text does not contain the NBSP? Or is this consistent with other text handling in mms2r i.e. are NBSP also retained/ignored in other parts of the code?

If it is the former let me know and I can send a pull request with a changes that would fix this.

Cheers,

James.

Double quotes and apostrophe's are getting mistranslated in mms2r 3.6. This was a persistent issue I had in ruby 1.8 but when upgrading to ruby 1.9 this was fixed. Again, after mms2r 3.6 and ruby 1.9.2 it is happening again.

Here are some screenshots detailing more what is going on.

I also have a mail fixture for this.

Note: Temporarily I have gone back to this change and it fixes this issue as well as the invalid byte sequence in UTF-8.

It seems a bit coincidental that Sprint has suddenly stopped working for me at the same time as Verizon (see issue #20) but I will report this anyway.

I am currently seeing a very different structure for sprint messages. Sprint is now including a text file attachment that contains the text of the message. Something that is also interesting is that messages received on June 13 are now expired on Sprint's servers when they should be accessible for 30-60 days.

Below is a mail message that exhibits the behavior:

Return-Path: [email protected]
Received: by 10.229.28.19 with SMTP id k19csp59935qcc; Tue, 26 Jun 2012 12:25:37 -0700
Received: by 10.68.213.234 with SMTP id nv10mr55008636pbc.56.1340738736898; Tue, 26 Jun 2012 12:25:36 -0700
Received: from smtp03.sun3.lightsurf.net (smtp.sun3.lightsurf.net. [63.215.195.131]) by mx.google.com with ESMTPS id qf1si16849177pbc.85.2012.06.26.12.25.36 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 26 Jun 2012 12:25:36 -0700
Received: from pcs-nams008 (pcs-mmsc.sun3.lightsurf.net [10.0.0.152]) by smtp03.sun3.lightsurf.net (8.12.11.20060308/8.12.11) with ESMTP id q5QJNHru005820 for [email protected]; Tue, 26 Jun 2012 12:25:36 -0700
Date: Tue, 26 Jun 2012 12:25:36 -0700
From: [email protected]
To: [email protected]
Message-ID: 31878575.1340738736299.JavaMail.lsadm@pcs-nams008
Subject: New Message
Mime-Version: 1.0
Content-Type: multipart/related;
boundary="----=_Part_321104_6797832.1340738736299";
charset=UTF-8
Content-Transfer-Encoding: 7bit
Delivered-To: [email protected]
Received-SPF: neutral (google.com: 63.215.195.131 is neither permitted nor
denied by best guess record for domain of [email protected])
client-ip=63.215.195.131;
Authentication-Results: mx.google.com; spf=neutral (google.com: 63.215.195.131
is neither permitted nor denied by best guess record for domain of
[email protected]) smtp.mail=[email protected]
X-Priority: 3
X-MSMail-Priority: Normal
Importance: Normal

------=_Part_321104_6797832.1340738736299
Date: Tue, 26 Jun 2012 19:28:32 +0000
Mime-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_Part_321103_10226250.1340738736298";
charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-ID: 4fea0d60abb12_2aa10adb8c768a8@cb6b861e-2b86-42be-be96-3e7f701daf10.mail

------=_Part_321103_10226250.1340738736298
Date: Tue, 26 Jun 2012 19:28:32 +0000
Mime-Version: 1.0
Content-Type: text/plain;
charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-ID: 4fea0d60a9b15_2aa10adb8c766ad@cb6b861e-2b86-42be-be96-3e7f701daf10.mail

You have received a Picture Mail from [email protected]

------=_Part_321103_10226250.1340738736298
Date: Tue, 26 Jun 2012 19:28:32 +0000
Mime-Version: 1.0
Content-Type: text/html;
charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-ID: 4fea0d60aac01_2aa10adb8c767a0@cb6b861e-2b86-42be-be96-3e7f701daf10.mail

<tr>
   <td colspan="2">
       <img src="http://pictures.sprintpcs.com/images/x.gif" width="5" height="45"/>
   </td>
</tr>
<tr>
   <td VALIGN="top" colspan="2" width="590">
      <img src="http://pictures.sprintpcs.com/images/x.gif" height="5"/>
   </td>
</tr>
<tr>
   <td VALIGN="top" colspan="2" width="100%">
       <table width="100%" border="0" cellspacing="0" cellpadding="0">
              <tr>
                  <td bgcolor="#ffffff" width="100%">
                        <div style="border-top:1px solid rgb(156, 154, 156)">&nbsp;</div>
                  </td>
              </tr>
                              <tr>
                                    <td>    
                                            <div style="float:right; padding-right:5px"><span style="font-family:trebuchet ms, Helvetica, Arial, Verdana; Font-size: 11px; Color: #000000">&#169; 2012 Sprint. All rights reserved.</span></div>
                                    </td>
                              </tr>
              <tr>
                  <td bgcolor="#ffffff" width="100%"><img src="http://pictures.sprintpcs.com/images/x.gif" border="0" height="10"/></td>
              </tr>
       </table>
   </td>
</tr>

You have a picture / video message from [email protected]

------=_Part_321103_10226250.1340738736298--

------=_Part_321104_6797832.1340738736299
Date: Tue, 26 Jun 2012 19:28:32 +0000
Mime-Version: 1.0
Content-Type: application/octet-stream;
charset=UTF-8;
name=textplain_2.txt
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=textplain_2.txt
Content-ID: 4fea0d60acc5f_2aa10adb8c76960@cb6b861e-2b86-42be-be96-3e7f701daf10.mail

@testweek "fail forward fast" - Testweek

------=_Part_321104_6797832.1340738736299--

This works fine:

ruby -rrubygems -Ilib -e "require 'mms2r'; require 'pp'; require 'logger'; mms = MMS2R::Media.new(Mail.read('/home/j/downloads/84.yml'), :logger => Logger.new(STDOUT)); pp mms.media; pp mms.number; pp mms.carrier; pp mms.mail.parts.first.body"

If we add "require rails;" to the mix:

ruby -rrubygems -Ilib -e "require 'rails'; require 'mms2r'; require 'pp'; require 'logger'; mms = MMS2R::Media.new(Mail.read('/home/j/downloads/84.yml'), :logger => Logger.new(STDOUT)); pp mms.media; pp mms.number; pp mms.carrier; pp mms.mail.parts.first.body"

we get:

MMS2R::Media created
MMS2R::Media processing
MMS2R::Media writing file /tmp/j/mms2r/332041621323016500006JavaMaillsadmpcs-nams003_6d05cddd835393daccfb871e7a40657a33e8dd64/1/041_0a3996dbcc9f0451_1-0.jpg
/home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:188:in write': "\xFF" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError) from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:188:inblock in sprint_write_file'
from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:188:in open' from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:188:insprint_write_file'
from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:167:in block in sprint_process_media' from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:130:ineach'
from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:130:in sprint_process_media' from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:61:inprocess'
from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media.rb:219:in initialize' from -e:1:innew'
from -e:1:in `

This was using a rvm gemset for ruby 1.9.2. The gems I installed were:
gem install mms2r
gem install rails

This resulted in the following gems and dependencies being installed:

*** LOCAL GEMS ***

actionmailer (3.1.3)
actionpack (3.1.3)
activemodel (3.1.3)
activerecord (3.1.3)
activeresource (3.1.3)
activesupport (3.1.3)
arel (2.2.1)
builder (3.0.0)
bundler (1.0.21)
erubis (2.7.0)
exifr (1.1.1)
hike (1.2.1)
i18n (0.6.0)
json (1.6.3)
mail (2.3.0)
mime-types (1.17.2)
mms2r (3.5.0)
multi_json (1.0.4)
nokogiri (1.5.0)
polyglot (0.3.3)
rack (1.3.5)
rack-cache (1.1)
rack-mount (0.8.3)
rack-ssl (1.3.2)
rack-test (0.6.1)
rails (3.1.3)
railties (3.1.3)
rake (0.9.2)
rdoc (3.11)
sprockets (2.0.3)
thor (0.14.6)
tilt (1.3.3)
treetop (1.4.10)
tzinfo (0.3.31)
uuidtools (2.1.2)

So something in there is stuffing things up.

I am trying to parse yahoo emails in order to extract their body of plain text and ignore the attachments. I first put the received email through the MAIL gem's receive function and then it goes through MMS2R after which I run regular expressions on mail.body but the problem is that after MMS2R parses the yahoo email, it somehow corrupts the body field so when I go mail.body it gives me

ArgumentError (invalid byte sequence in UTF-8):
app/controllers/emails_controller.rb:25:in `create'

mail.body works fine upto the point where I go "email = MMS2R::Media.new(mail)"

It seems to work fine with gmail, hotmail and every other web client except for yahoo. Is this a known issue? I am using
"ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2011.03"

I don't think any functionality is affected, just wanted to point out that iconv will be deprecated in ruby 1.9.3

/bundler/gems/mms2r-7a4d0fd419d7/lib/mms2r.rb:62:in 'block in <top (required)>': iconv will be deprecated in the future, use String#encode instead.

Thanks for a great gem

I am getting the following error when running tests from concurrent builds.

/usr/bin/ruby18 -I "/usr/lib64/ruby/gems/1.8/gems/cucumber-0.3.96/lib:lib" "/usr/lib64/ruby/gems/1.8/gems/cucumber-0.3.96/bin/cucumber" --profile firefox35 --format pretty --format html --out=/data/homedirs/cosmos/.hudson/jobs/firefox35/workspace/log/cucumber/html/cucumber_firefox35_results.html features/current
Using the firefox35 profile...
Unable to load thrift_native extension. Defaulting to pure Ruby libraries.
spawned dj pid 4556

• User {:username=>"demo1"}
  No such file or directory - /tmp/mms2r/9F8813E4DB0BB740A4E475B593C6E4D31E38928Fgsmbnop12esfirmwidecorpgscom/2/1253664813.98513.html (Errno::ENOENT)
  /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:418:in initialize' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:418:inopen'
  /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:418:in process_media' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:354:inprocess'
  (eval):3:in each_without_optional_block' (eval):3:ineach'
  /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:351:in process' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:233:ininitialize'
  /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:113:in orig_new' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:113:innew'
  /data/homedirs/cosmos/.hudson/jobs/firefox35/workspace/app/models/email/message.rb:113:in rfc822_attachments' /data/homedirs/cosmos/.hudson/jobs/firefox35/workspace/app/models/email/message.rb:99:inbuild_from_tmail'
  ./db/populate/02_demo_users.rb:21

It looks to me that this is because more than one process is reading the same email with the same message_id concurrently.

Looking at the code, all processes running as the same user share the same tmp directory.

def self.tmp_dir
  @@tmp_dir ||= File.join(Dir.tmpdir, (ENV['USER'].nil? ? '':ENV['USER']), 'mms2r')
end

And the media_dir is a product of the tmpdir using hte message id and the current time:

  @media_dir = File.join(self.tmp_dir(), 
                 self.safe_message_id(@mail.message_id))

All MMS2R versions from the beginning to the current version 2.4.1 are dependent on TMail. I will be releasing MMS2R 3.0.0 shortly, it will be dependent on the Mail gem, instead of TMail, and Rails 3 is using the Mail gem in ActionMailer as well.

I have noticed that MMS2R is no longer parsing verizon text messages correctly. I am getting a partial double up of content. I think that this is because Verizon have changed their email subject line to include a "..." at the end of the subject.

Here is a mail that shows the content of a recent message from vtext:

Return-Path: [email protected]
Received: by 10.229.28.19 with SMTP id k19csp54879qcc; Tue, 26 Jun 2012 11:25:41 -0700
Received: by 10.52.29.179 with SMTP id l19mr9987197vdh.73.1340735141504; Tue, 26 Jun 2012 11:25:41 -0700
Received: from njbrspamp10.vtext.com (njbrspamp11.vtext.com. [69.78.129.156]) by mx.google.com with ESMTP id dr7si10901941vdb.48.2012.06.26.11.25.41; Tue, 26 Jun 2012 11:25:41 -0700
Received: from unknown (HELO njbbvmaapp07-e1) ([10.134.225.138]) by njbrspamp10.vtext.com with ESMTP; Tue, 26 Jun 2012 18:25:41 +0000
Date: Tue, 26 Jun 2012 18:25:41 +0000
From: [email protected]
To: [email protected]
Message-ID: 25691599.454593.1340735141169.JavaMail.vmaprod@njbbvmaapp07
Subject: @nextweek great meeting with SocialXXXX, ready to ...
Mime-Version: 1.0
Content-Type: text/plain;
charset=us-ascii
Content-Transfer-Encoding: 7bit
Delivered-To: [email protected]
Received-SPF: pass (google.com: best guess record for domain of
[email protected] designates 69.78.129.156 as permitted sender)
client-ip=69.78.129.156;
Authentication-Results: mx.google.com; spf=pass (google.com: best guess record
for domain of [email protected] designates 69.78.129.156 as permitted
sender) smtp.mail=[email protected]

@nextweek great meeting with SocialXXXX, ready to kickass #fundayaway

I'm trying out the example script that comes with the code. It runs (with some appropriate modifications) in Ruby 1.8.7 and Rails2. However under Ruby 1.9.7 and rails 3 it fails. Here's the dump:
/Users/seanmccullough/.rvm/gems/ruby-1.9.2-preview3@rails3/gems/mms2r-3.0.1/lib/mms2r/media.rb:188:in domain': undefined methodany?' for "vzwpix.com":String (NoMethodError)
from /Users/seanmccullough/.rvm/gems/ruby-1.9.2-preview3@rails3/gems/mms2r-3.0.1/lib/mms2r/media.rb:153:in create' from /Users/seanmccullough/.rvm/gems/ruby-1.9.2-preview3@rails3/gems/mms2r-3.0.1/lib/mms2r/media.rb:111:innew'
from mmstest.rb:7:in `

The relevant code:

media.rb line 169
when mail.return_path
mail.return_path ? mail.return_path.split('@').last : ''
else
''
end
line 188
when return_path.any?

it seems that when the return path is nil, it tries to call
''.any?
which doesn't exist.

I'm not sure why this happens under one version of ruby and not another.