monde / mms2r Goto Github PK
View Code? Open in Web Editor NEWExtract user media from MMS (and not carrier cruft)
Home Page: http://mms2r.rubyforge.org/
License: MIT License
Extract user media from MMS (and not carrier cruft)
Home Page: http://mms2r.rubyforge.org/
License: MIT License
This works fine:
ruby -rrubygems -Ilib -e "require 'mms2r'; require 'pp'; require 'logger'; mms = MMS2R::Media.new(Mail.read('/home/j/downloads/84.yml'), :logger => Logger.new(STDOUT)); pp mms.media; pp mms.number; pp mms.carrier; pp mms.mail.parts.first.body"
If we add "require rails;" to the mix:
ruby -rrubygems -Ilib -e "require 'rails'; require 'mms2r'; require 'pp'; require 'logger'; mms = MMS2R::Media.new(Mail.read('/home/j/downloads/84.yml'), :logger => Logger.new(STDOUT)); pp mms.media; pp mms.number; pp mms.carrier; pp mms.mail.parts.first.body"
we get:
MMS2R::Media created
MMS2R::Media processing
MMS2R::Media writing file /tmp/j/mms2r/332041621323016500006JavaMaillsadmpcs-nams003_6d05cddd835393daccfb871e7a40657a33e8dd64/1/041_0a3996dbcc9f0451_1-0.jpg
/home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:188:in write': "\xFF" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError) from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:188:in
block in sprint_write_file'
from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:188:in open' from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:188:in
sprint_write_file'
from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:167:in block in sprint_process_media' from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:130:in
each'
from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:130:in sprint_process_media' from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media/sprint.rb:61:in
process'
from /home/j/.rvm/gems/ruby-1.9.2-p290@mms2r/gems/mms2r-3.5.0/lib/mms2r/media.rb:219:in initialize' from -e:1:in
new'
from -e:1:in `
This was using a rvm gemset for ruby 1.9.2. The gems I installed were:
gem install mms2r
gem install rails
This resulted in the following gems and dependencies being installed:
*** LOCAL GEMS ***
actionmailer (3.1.3)
actionpack (3.1.3)
activemodel (3.1.3)
activerecord (3.1.3)
activeresource (3.1.3)
activesupport (3.1.3)
arel (2.2.1)
builder (3.0.0)
bundler (1.0.21)
erubis (2.7.0)
exifr (1.1.1)
hike (1.2.1)
i18n (0.6.0)
json (1.6.3)
mail (2.3.0)
mime-types (1.17.2)
mms2r (3.5.0)
multi_json (1.0.4)
nokogiri (1.5.0)
polyglot (0.3.3)
rack (1.3.5)
rack-cache (1.1)
rack-mount (0.8.3)
rack-ssl (1.3.2)
rack-test (0.6.1)
rails (3.1.3)
railties (3.1.3)
rake (0.9.2)
rdoc (3.11)
sprockets (2.0.3)
thor (0.14.6)
tilt (1.3.3)
treetop (1.4.10)
tzinfo (0.3.31)
uuidtools (2.1.2)
So something in there is stuffing things up.
I don't think any functionality is affected, just wanted to point out that iconv will be deprecated in ruby 1.9.3
/bundler/gems/mms2r-7a4d0fd419d7/lib/mms2r.rb:62:in 'block in <top (required)>': iconv will be deprecated in the future, use String#encode instead.
Thanks for a great gem
Hey @monde,
Just ran into a small issue with Sprint (again). The issue is that the JSON returned from the Sprint Ajax is returning &NBSP; instead of a regular space. This causes problems with using ruby string methods like strip as they don't expect the NBSP instead of utf8/ascii.
So, my question is: Should the code I submitted recently be ensuring that the text does not contain the NBSP? Or is this consistent with other text handling in mms2r i.e. are NBSP also retained/ignored in other parts of the code?
If it is the former let me know and I can send a pull request with a changes that would fix this.
Cheers,
James.
I'm trying out the example script that comes with the code. It runs (with some appropriate modifications) in Ruby 1.8.7 and Rails2. However under Ruby 1.9.7 and rails 3 it fails. Here's the dump:
/Users/seanmccullough/.rvm/gems/ruby-1.9.2-preview3@rails3/gems/mms2r-3.0.1/lib/mms2r/media.rb:188:in domain': undefined method
any?' for "vzwpix.com":String (NoMethodError)
from /Users/seanmccullough/.rvm/gems/ruby-1.9.2-preview3@rails3/gems/mms2r-3.0.1/lib/mms2r/media.rb:153:in create' from /Users/seanmccullough/.rvm/gems/ruby-1.9.2-preview3@rails3/gems/mms2r-3.0.1/lib/mms2r/media.rb:111:in
new'
from mmstest.rb:7:in `
The relevant code:
media.rb line 169
when mail.return_path
mail.return_path ? mail.return_path.split('@').last : ''
else
''
end
line 188
when return_path.any?
it seems that when the return path is nil, it tries to call
''.any?
which doesn't exist.
I'm not sure why this happens under one version of ruby and not another.
I am trying to parse yahoo emails in order to extract their body of plain text and ignore the attachments. I first put the received email through the MAIL gem's receive function and then it goes through MMS2R after which I run regular expressions on mail.body but the problem is that after MMS2R parses the yahoo email, it somehow corrupts the body field so when I go mail.body it gives me
ArgumentError (invalid byte sequence in UTF-8):
app/controllers/emails_controller.rb:25:in `create'
mail.body works fine upto the point where I go "email = MMS2R::Media.new(mail)"
It seems to work fine with gmail, hotmail and every other web client except for yahoo. Is this a known issue? I am using
"ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2011.03"
Whenever I call MMS2R::Media.new(email) I get the following with the latest version of the gem:
NoMethodError: undefined method return_path' for #<TMail::Mail:0x1068155b8> /Users/jlippiner/.rvm/gems/ree-1.8.7-2010.01@tdt/gems/mms2r-3.0.1/lib/mms2r/media.rb:169:in
domain'
/Users/jlippiner/.rvm/gems/ree-1.8.7-2010.01@tdt/gems/mms2r-3.0.1/lib/mms2r/media.rb:153:in create' /Users/jlippiner/.rvm/gems/ree-1.8.7-2010.01@tdt/gems/mms2r-3.0.1/lib/mms2r/media.rb:111:in
new'
All MMS2R versions from the beginning to the current version 2.4.1 are dependent on TMail. I will be releasing MMS2R 3.0.0 shortly, it will be dependent on the Mail gem, instead of TMail, and Rails 3 is using the Mail gem in ActionMailer as well.
New version is working a charm with Mail.
However I am having a challenge with a couple of messages with the @media={"text/html". What method do I use to print out the html version of the text? Or do I have to open the associated temp file to get at this?
Thanks
Paul
I am getting the following error when running tests from concurrent builds.
/usr/bin/ruby18 -I "/usr/lib64/ruby/gems/1.8/gems/cucumber-0.3.96/lib:lib" "/usr/lib64/ruby/gems/1.8/gems/cucumber-0.3.96/bin/cucumber" --profile firefox35 --format pretty --format html --out=/data/homedirs/cosmos/.hudson/jobs/firefox35/workspace/log/cucumber/html/cucumber_firefox35_results.html features/current
Using the firefox35 profile...
Unable to load thrift_native extension. Defaulting to pure Ruby libraries.
spawned dj pid 4556
initialize' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:418:in
open'process_media' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:354:in
process'each_without_optional_block' (eval):3:in
each'process' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:233:in
initialize'orig_new' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:113:in
new'rfc822_attachments' /data/homedirs/cosmos/.hudson/jobs/firefox35/workspace/app/models/email/message.rb:99:in
build_from_tmail'It looks to me that this is because more than one process is reading the same email with the same message_id concurrently.
Looking at the code, all processes running as the same user share the same tmp directory.
def self.tmp_dir
@@tmp_dir ||= File.join(Dir.tmpdir, (ENV['USER'].nil? ? '':ENV['USER']), 'mms2r')
end
And the media_dir is a product of the tmpdir using hte message id and the current time:
@media_dir = File.join(self.tmp_dir(),
self.safe_message_id(@mail.message_id))
It seems a bit coincidental that Sprint has suddenly stopped working for me at the same time as Verizon (see issue #20) but I will report this anyway.
I am currently seeing a very different structure for sprint messages. Sprint is now including a text file attachment that contains the text of the message. Something that is also interesting is that messages received on June 13 are now expired on Sprint's servers when they should be accessible for 30-60 days.
Below is a mail message that exhibits the behavior:
Return-Path: [email protected]
Received: by 10.229.28.19 with SMTP id k19csp59935qcc; Tue, 26 Jun 2012 12:25:37 -0700
Received: by 10.68.213.234 with SMTP id nv10mr55008636pbc.56.1340738736898; Tue, 26 Jun 2012 12:25:36 -0700
Received: from smtp03.sun3.lightsurf.net (smtp.sun3.lightsurf.net. [63.215.195.131]) by mx.google.com with ESMTPS id qf1si16849177pbc.85.2012.06.26.12.25.36 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 26 Jun 2012 12:25:36 -0700
Received: from pcs-nams008 (pcs-mmsc.sun3.lightsurf.net [10.0.0.152]) by smtp03.sun3.lightsurf.net (8.12.11.20060308/8.12.11) with ESMTP id q5QJNHru005820 for [email protected]; Tue, 26 Jun 2012 12:25:36 -0700
Date: Tue, 26 Jun 2012 12:25:36 -0700
From: [email protected]
To: [email protected]
Message-ID: 31878575.1340738736299.JavaMail.lsadm@pcs-nams008
Subject: New Message
Mime-Version: 1.0
Content-Type: multipart/related;
boundary="----=_Part_321104_6797832.1340738736299";
charset=UTF-8
Content-Transfer-Encoding: 7bit
Delivered-To: [email protected]
Received-SPF: neutral (google.com: 63.215.195.131 is neither permitted nor
denied by best guess record for domain of [email protected])
client-ip=63.215.195.131;
Authentication-Results: mx.google.com; spf=neutral (google.com: 63.215.195.131
is neither permitted nor denied by best guess record for domain of
[email protected]) smtp.mail=[email protected]
X-Priority: 3
X-MSMail-Priority: Normal
Importance: Normal
------=_Part_321104_6797832.1340738736299
Date: Tue, 26 Jun 2012 19:28:32 +0000
Mime-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_Part_321103_10226250.1340738736298";
charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-ID: 4fea0d60abb12_2aa10adb8c768a8@cb6b861e-2b86-42be-be96-3e7f701daf10.mail
------=_Part_321103_10226250.1340738736298
Date: Tue, 26 Jun 2012 19:28:32 +0000
Mime-Version: 1.0
Content-Type: text/plain;
charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-ID: 4fea0d60a9b15_2aa10adb8c766ad@cb6b861e-2b86-42be-be96-3e7f701daf10.mail
You have received a Picture Mail from [email protected]
------=_Part_321103_10226250.1340738736298
Date: Tue, 26 Jun 2012 19:28:32 +0000
Mime-Version: 1.0
Content-Type: text/html;
charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-ID: 4fea0d60aac01_2aa10adb8c767a0@cb6b861e-2b86-42be-be96-3e7f701daf10.mail
<tr>
<td colspan="2">
<img src="http://pictures.sprintpcs.com/images/x.gif" width="5" height="45"/>
</td>
</tr>
<tr>
<td VALIGN="top" colspan="2" width="590">
<img src="http://pictures.sprintpcs.com/images/x.gif" height="5"/>
</td>
</tr>
<tr>
<td VALIGN="top" colspan="2" width="100%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr>
<td bgcolor="#ffffff" width="100%">
<div style="border-top:1px solid rgb(156, 154, 156)"> </div>
</td>
</tr>
<tr>
<td>
<div style="float:right; padding-right:5px"><span style="font-family:trebuchet ms, Helvetica, Arial, Verdana; Font-size: 11px; Color: #000000">© 2012 Sprint. All rights reserved.</span></div>
</td>
</tr>
<tr>
<td bgcolor="#ffffff" width="100%"><img src="http://pictures.sprintpcs.com/images/x.gif" border="0" height="10"/></td>
</tr>
</table>
</td>
</tr>
|
|||
|
------=_Part_321103_10226250.1340738736298--
------=_Part_321104_6797832.1340738736299
Date: Tue, 26 Jun 2012 19:28:32 +0000
Mime-Version: 1.0
Content-Type: application/octet-stream;
charset=UTF-8;
name=textplain_2.txt
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=textplain_2.txt
Content-ID: 4fea0d60acc5f_2aa10adb8c76960@cb6b861e-2b86-42be-be96-3e7f701daf10.mail
@testweek "fail forward fast" - Testweek
------=_Part_321104_6797832.1340738736299--
I've been having some strange things happen with some multipart messages.
The message is: http://gist.github.com/472080
and the stack trace is:
/usr/local/lib/ruby/gems/1.8/gems/mail-2.2.4/lib/mail/field.rb:122:in `send': undefined method `filename' for #<Mail::UnstructuredField:0x1020051b0> (NoMethodError)
from /usr/local/lib/ruby/gems/1.8/gems/mail-2.2.4/lib/mail/field.rb:122:in `method_missing'
from /usr/local/lib/ruby/gems/1.8/gems/mail-2.2.4/lib/mail/message.rb:1849:in `find_attachment'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mail_ext.rb:33:in `filename'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:507:in `filename?'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:471:in `temp_file'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:405:in `process_media'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:358:in `process'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:355:in `each'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:355:in `process'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:233:in `initialize'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:113:in `orig_new'
from /usr/local/lib/ruby/gems/1.8/gems/mms2r-3.0.1/lib/mms2r/media.rb:113:in `new'
from mmstest.rb:7
It seems like it doesn't like dealing with this kind of multipart message. Anyone else run into this issue, and have a fix for it?
Hi,
I am using the example code off of the main project page. I have MMS2R 3.1 and Mail 2.2.5 installed. When I run this code:
mail = MMS2R::Media.new(Mail.read('./test.video-tmobile.eml'))
mail.default_media is nil and mail.media an empty Hash. I've been troubleshooting this for a couple of days and I'm not making much progress. Let me know if you need to see the .eml file, it's rather large and I'd rather not junk up this post with that content.
Thanks
Hello,
I am using mms2r/tmail with my rails application to process email received and I am very happy with it.
Today one of my user encountered a weird problem where 2 attached .pdf files were not process. I looked at the source of the mail and they both had a "application/octet-stream" type.
looking at the code in media.rb I noticed that on the process_media method (lines 404 to 426) if I comment the test on the 'application/octet-stream' to just keep the line 416
"type = part.content_type"
it works correctly and the 2 pdf files are processed
Do you have any idea on what's happened?
Thanks in advance
Nicolas
I have noticed that MMS2R is no longer parsing verizon text messages correctly. I am getting a partial double up of content. I think that this is because Verizon have changed their email subject line to include a "..." at the end of the subject.
Here is a mail that shows the content of a recent message from vtext:
Return-Path: [email protected]
Received: by 10.229.28.19 with SMTP id k19csp54879qcc; Tue, 26 Jun 2012 11:25:41 -0700
Received: by 10.52.29.179 with SMTP id l19mr9987197vdh.73.1340735141504; Tue, 26 Jun 2012 11:25:41 -0700
Received: from njbrspamp10.vtext.com (njbrspamp11.vtext.com. [69.78.129.156]) by mx.google.com with ESMTP id dr7si10901941vdb.48.2012.06.26.11.25.41; Tue, 26 Jun 2012 11:25:41 -0700
Received: from unknown (HELO njbbvmaapp07-e1) ([10.134.225.138]) by njbrspamp10.vtext.com with ESMTP; Tue, 26 Jun 2012 18:25:41 +0000
Date: Tue, 26 Jun 2012 18:25:41 +0000
From: [email protected]
To: [email protected]
Message-ID: 25691599.454593.1340735141169.JavaMail.vmaprod@njbbvmaapp07
Subject: @nextweek great meeting with SocialXXXX, ready to ...
Mime-Version: 1.0
Content-Type: text/plain;
charset=us-ascii
Content-Transfer-Encoding: 7bit
Delivered-To: [email protected]
Received-SPF: pass (google.com: best guess record for domain of
[email protected] designates 69.78.129.156 as permitted sender)
client-ip=69.78.129.156;
Authentication-Results: mx.google.com; spf=pass (google.com: best guess record
for domain of [email protected] designates 69.78.129.156 as permitted
sender) smtp.mail=[email protected]
@nextweek great meeting with SocialXXXX, ready to kickass #fundayaway
Double quotes and apostrophe's are getting mistranslated in mms2r 3.6. This was a persistent issue I had in ruby 1.8 but when upgrading to ruby 1.9 this was fixed. Again, after mms2r 3.6 and ruby 1.9.2 it is happening again.
Here are some screenshots detailing more what is going on.
I also have a mail fixture for this.
Note: Temporarily I have gone back to this change and it fixes this issue as well as the invalid byte sequence in UTF-8.
Hi, just ran into very strange bug when media filename? returned filename with newline at the end. Email was sent from Zimbra system. Quick fix was to strip output of newline with adding following code as Rails initialializer:
module MMS2R
class MMS2R::Media
def filename_with_strip?(part)
filename_without_strip?(part).strip
end
alias_method_chain :filename?, :strip
end
end
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.