rawler fails on some URL about rawler

[oscardelben]

Hey Jetienne,

I'm not sure why that's happening. It looks like its happening inside net/http. What I can do is wrap the requests in an external library that handles all these exceptions (I'm planning to do this anyway).

One question: does this happen every time, or it's just a one case scenario? I was able to test your url with ruby 1.87 and ruby 1.9.2p136

[jetienne]

Hi,

Yes, this happens all the time. Strange we don't have the same behavior.

Regards,

[arkan]

Hey

Here are an other error :
ruby rawler http://www.google.fr
200 - http://www.google.fr/imghp?hl=fr&tab=wi
200 - http://www.google.fr/webhp?hl=fr&tab=iw
200 - http://video.google.fr/?hl=fr&tab=wv
200 - http://maps.google.fr/maps?hl=fr&tab=wl
200 - http://news.google.fr/nwshp?hl=fr&tab=wn
200 - http://www.google.fr/prdhp?hl=fr&tab=wf
/Users/Arkan/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/uri/generic.rb:746:in rescue in merge': bad URI(is not URI?): /products?hl=fr&gl=fr&holiday=l10&q=sèche-linge (URI::InvalidURIError) from /Users/Arkan/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/uri/generic.rb:743:inmerge'
from /Users/Arkan/Code/rawler/lib/rawler/crawler.rb:28:in absolute_url' from /Users/Arkan/Code/rawler/lib/rawler/crawler.rb:19:inblock in links'
from /Users/Arkan/.rvm/gems/ruby-1.9.2-p0@messy/gems/nokogiri-1.4.4/lib/nokogiri/xml/node_set.rb:239:in block in each' from /Users/Arkan/.rvm/gems/ruby-1.9.2-p0@messy/gems/nokogiri-1.4.4/lib/nokogiri/xml/node_set.rb:238:inupto'
from /Users/Arkan/.rvm/gems/ruby-1.9.2-p0@messy/gems/nokogiri-1.4.4/lib/nokogiri/xml/node_set.rb:238:in each' from /Users/Arkan/Code/rawler/lib/rawler/crawler.rb:19:inmap'
from /Users/Arkan/Code/rawler/lib/rawler/crawler.rb:19:in links' from /Users/Arkan/Code/rawler/lib/rawler/base.rb:23:invalidate_links_in_page'
from /Users/Arkan/Code/rawler/lib/rawler/base.rb:31:in validate_page' from /Users/Arkan/Code/rawler/lib/rawler/base.rb:24:inblock in validate_links_in_page'
from /Users/Arkan/Code/rawler/lib/rawler/base.rb:23:in each' from /Users/Arkan/Code/rawler/lib/rawler/base.rb:23:invalidate_links_in_page'
from /Users/Arkan/Code/rawler/lib/rawler/base.rb:31:in validate_page' from /Users/Arkan/Code/rawler/lib/rawler/base.rb:24:inblock in validate_links_in_page'
from /Users/Arkan/Code/rawler/lib/rawler/base.rb:23:in each' from /Users/Arkan/Code/rawler/lib/rawler/base.rb:23:invalidate_links_in_page'
from /Users/Arkan/Code/rawler/lib/rawler/base.rb:31:in validate_page' from /Users/Arkan/Code/rawler/lib/rawler/base.rb:24:inblock in validate_links_in_page'
from /Users/Arkan/Code/rawler/lib/rawler/base.rb:23:in each' from /Users/Arkan/Code/rawler/lib/rawler/base.rb:23:invalidate_links_in_page'
from /Users/Arkan/Code/rawler/lib/rawler/base.rb:17:in validate' from rawler:32:in

'
zsh: exit 1 ruby rawler http://www.google.fr

[oscardelben]

Thanks guys,

I should have fixed the problem. Please do a gem update rawler (make sure it's 0.0.4). The second problem should have gone away, the first one I wasn't able to reproduce unfortunately.

[jney]

hi oscar,
i reproduced the bug even in 0.0.4 :

>>> rawler http://fr.yahoo.com/
/Users/jney/.rvm/rubies/jruby-1.5.6/lib/ruby/1.8/uri/generic.rb:732:in `merge': bad URI(is not URI?): %20http://fr.docs.yahoo.com/mail/fast/index.html%20 (URI::InvalidURIError)
        from /Users/jney/.rvm/gems/jruby-1.5.6/gems/rawler-0.0.4/lib/rawler/crawler.rb:28:in `absolute_url'
        from /Users/jney/.rvm/gems/jruby-1.5.6/gems/rawler-0.0.4/lib/rawler/crawler.rb:19:in `links'
        from /Users/jney/.rvm/gems/jruby-1.5.6/gems/nokogiri-1.4.4.2-java/lib/nokogiri/xml/node_set.rb:239:in `each'
        from /Users/jney/.rvm/gems/jruby-1.5.6/gems/nokogiri-1.4.4.2-java/lib/nokogiri/xml/node_set.rb:238:in `upto'
        from /Users/jney/.rvm/gems/jruby-1.5.6/gems/nokogiri-1.4.4.2-java/lib/nokogiri/xml/node_set.rb:238:in `each'
        from /Users/jney/.rvm/gems/jruby-1.5.6/gems/rawler-0.0.4/lib/rawler/crawler.rb:19:in `map'
        from /Users/jney/.rvm/gems/jruby-1.5.6/gems/rawler-0.0.4/lib/rawler/crawler.rb:19:in `links'
        from /Users/jney/.rvm/gems/jruby-1.5.6/gems/rawler-0.0.4/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/jney/.rvm/gems/jruby-1.5.6/gems/rawler-0.0.4/lib/rawler/base.rb:17:in `validate'
        from /Users/jney/.rvm/gems/jruby-1.5.6/gems/rawler-0.0.4/bin/rawler:32
        from /Users/jney/.rvm/gems/jruby-1.5.6/gems/rawler-0.0.4/bin/rawler:19:in `load'
        from /usr/local/bin/rawler:19
zsh: exit 1     rawler http://fr.yahoo.com/

[PabloC]

hi oscar! I'mhaving the same problem on 0.0.5

":in `rescue in merge': bad URI(is not URI?): http://www.cardif.com.ar/Website Cardif/Argentina/site/movistar/paginaseguros.html (URI::InvalidURIError)"

[oscardelben]

PabloC, thsnks. These issues should be solved as soon as I switch to Mechanize. Sorry for the inconvenience.

[oscardelben]

Hi guys, I'm closing this, I've tested most urls here and it should work in 0.6. If you find other problems please open a new ticket so I can start over. Thanks again for ALL the help.