GithubHelp home page GithubHelp logo

rawler's People

Contributors

afeld avatar avonderluft avatar bcobb avatar cr0t avatar danelbert avatar dkapadia avatar enricogenauck avatar kenegozi avatar lnxbil avatar oscardelben avatar sos4nt avatar triggernz avatar vesan avatar youngbrioche avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

vesan jerryk0001 kenegozi metade youngbrioche sos4nt avonderluft jefffis dkapadia flyabroadkit sce thiyagarajan newbray solsystems afeld clare312 ejhayes danelbert st63jun castledoor enricogenauck djdarkbeat mhatherly agross n-studio iq-scm

rawler's Issues

--ignore_fragments is not supported (but documented) 

$ bundle exec rawler --ignore_fragments http://google.com
Error: unknown argument '--ignore_fragments'.
Try --help for help.

$ bundle exec rawler --ignore_fragments http://google.com
Error: unknown argument '--ignore_fragments'.
Try --help for help.

--help doesn't print the option, despite it being documented on GitHub.

$ bundle exec rawler --help
Rawler is a command line utility for parsing links on a website

Usage:
      rawler http://example.com [options]

where [options] are:
  --username, -u <s>:   HTTP Basic Username
  --password, -p <s>:   HTTP Basic Password
      --wait, -w <f>:   Seconds to wait between requests, may be fractional
                        e.g. '1.5' (default: 3.0)
           --log, -l:   Log results to file rawler_log.txt
   --logfile, -o <s>:   Specify logfile, implies --log (default:
                        rawler_log.txt)
           --css, -c:   Check CSS links
      --skip, -s <s>:   Skip URLs that match a pattern
     --iskip, -i <s>:   Skip URLs that match a case insensitive pattern
   --include, -n <s>:   Only include URLS that match a pattern
  --iinclude, -d <s>:   Only include URLS that match a case insensitive
                        pattern. Equivalent to '--include
                        ^http://mysite.com/*'.
         --local, -a:   Restrict to the given URL and below
       --version, -v:   Print version and exit
          --help, -h:   Show this message

"rawler http://geeks.aretotally.in/" - `initialize': the scheme http does not accept registry part

fo@li175-103:~/tmp/geeks$ rawler http://geeks.aretotally.in/
/usr/lib/ruby/1.8/uri/generic.rb:195:in initialize': the scheme http does not accept registry part: AppDemoingIntegrationBetweenGrailswithFacebook'sGraph'sAPIusingGroovy (or bad hostname?) (URI::InvalidURIError) from /usr/lib/ruby/1.8/uri/http.rb:78:ininitialize'
from /usr/lib/ruby/1.8/uri/common.rb:488:in new' from /usr/lib/ruby/1.8/uri/common.rb:488:inparse'

net/http.rb:1266:in `addr_port': undefined method `+' for nil:NilClass (NoMethodError)

Got an exception while crawling my site. I'm not sure which link confused it by I figured I'd pass along the stack trace:

[rvm 1.9.2] ~ $ rawler http://jimmycuadra.com/
I, [2011-03-29T08:38:26.407541 #28817]  INFO -- : 200 - http://jimmycuadra.com/
I, [2011-03-29T08:38:30.208174 #28817]  INFO -- : 200 - http://jimmycuadra.com/projects
I, [2011-03-29T08:38:36.770008 #28817]  INFO -- : 200 - http://jimmycuadra.com/about
I, [2011-03-29T08:38:55.920711 #28817]  INFO -- : 200 - https://github.com/jimmycuadra
I, [2011-03-29T08:38:59.744387 #28817]  INFO -- : 200 - http://twitter.com/jimmycuadra
I, [2011-03-29T08:39:06.076651 #28817]  INFO -- : 200 - http://en.wikipedia.org/wiki/San_Diego
I, [2011-03-29T08:39:10.441370 #28817]  INFO -- : 200 - http://en.wikipedia.org/wiki/Los_Angeles
I, [2011-03-29T08:39:13.625704 #28817]  INFO -- : 200 - http://ucsd.edu/
I, [2011-03-29T08:39:18.099691 #28817]  INFO -- : 200 - http://www.ruby-lang.org/en/
W, [2011-03-29T08:39:18.100729 #28817]  WARN -- : 302 - http://www.ruby-lang.org/ - Called from: http://jimmycuadra.com/about - Following redirection to: http://www.ruby-lang.org/en/
I, [2011-03-29T08:39:21.508748 #28817]  INFO -- : 200 - http://rubyonrails.org/
I, [2011-03-29T08:39:25.031023 #28817]  INFO -- : 200 - http://git-scm.com/
I, [2011-03-29T08:39:34.318888 #28817]  INFO -- : 200 - http://www.linkedin.com/in/jimmycuadra
I, [2011-03-29T08:39:37.616162 #28817]  INFO -- : 200 - http://jimmycuadra.com/login
I, [2011-03-29T08:40:05.612382 #28817]  INFO -- : 200 - http://www.apple.com/
W, [2011-03-29T08:40:08.952060 #28817]  WARN -- : 301 - http://www.rubyonrails.org/ - Called from: http://jimmycuadra.com/login - Following redirection to: http://rubyonrails.org/
I, [2011-03-29T08:40:12.328434 #28817]  INFO -- : 200 - http://www.webkit.org/
W, [2011-03-29T08:40:12.329675 #28817]  WARN -- : 301 - http://webkit.org/ - Called from: http://jimmycuadra.com/login - Following redirection to: http://www.webkit.org/
I, [2011-03-29T08:40:45.985793 #28817]  INFO -- : 200 - http://morethingsneed.to/
I, [2011-03-29T08:40:53.375976 #28817]  INFO -- : 200 - http://eventful.com/
I, [2011-03-29T08:40:59.556434 #28817]  INFO -- : 200 - http://rady.ucsd.edu/
I, [2011-03-29T08:41:09.337865 #28817]  INFO -- : 200 - https://github.com/jimmycuadra/to_lang
I, [2011-03-29T08:41:12.993974 #28817]  INFO -- : 200 - http://ruby5.envylabs.com/episodes/141-episode-138-january-4-2011/stories/1229-to_lang-a-ruby-interface-to-google-translate
I, [2011-03-29T08:41:16.124937 #28817]  INFO -- : 200 - http://news.ycombinator.com/item?id=2095006
I, [2011-03-29T08:41:19.907478 #28817]  INFO -- : 200 - https://github.com/jimmycuadra/changesets
I, [2011-03-29T08:41:23.646278 #28817]  INFO -- : 200 - https://github.com/jimmycuadra/jimmycuadra.com
I, [2011-03-29T08:41:59.920392 #28817]  INFO -- : 200 - http://jimmycuadra.com/?page=2
I, [2011-03-29T08:42:27.815614 #28817]  INFO -- : 200 - http://jimmycuadra.com/?page=3
I, [2011-03-29T08:42:58.577409 #28817]  INFO -- : 200 - http://jimmycuadra.com/?page=4
I, [2011-03-29T08:43:32.515108 #28817]  INFO -- : 200 - http://jimmycuadra.com/?page=5
I, [2011-03-29T08:44:09.485068 #28817]  INFO -- : 200 - http://jimmycuadra.com/?page=6
I, [2011-03-29T08:44:49.289188 #28817]  INFO -- : 200 - http://jimmycuadra.com/?page=7
I, [2011-03-29T08:45:32.111589 #28817]  INFO -- : 200 - http://jimmycuadra.com/?page=8
I, [2011-03-29T08:46:18.082790 #28817]  INFO -- : 200 - http://jimmycuadra.com/?page=9
I, [2011-03-29T08:47:07.046778 #28817]  INFO -- : 200 - http://jimmycuadra.com/?page=10
I, [2011-03-29T08:47:58.922460 #28817]  INFO -- : 200 - http://jimmycuadra.com/?page=11
I, [2011-03-29T08:48:53.794334 #28817]  INFO -- : 200 - http://jimmycuadra.com/?page=12
I, [2011-03-29T08:49:54.517105 #28817]  INFO -- : 200 - http://jimmycuadra.com/posts/to-lang-030-featuring-batch-translations-and-a-command-line-utility
I, [2011-03-29T08:50:22.255492 #28817]  INFO -- : 200 - http://rubydoc.info/gems/to_lang/0.3.0/frames
I, [2011-03-29T08:50:25.932320 #28817]  INFO -- : 200 - https://rubygems.org/gems/to_lang
I, [2011-03-29T08:50:29.171260 #28817]  INFO -- : 200 - http://jimmycuadra.com/tags/to-lang
I, [2011-03-29T08:50:53.879056 #28817]  INFO -- : 200 - http://jimmycuadra.com/posts/to-lang-a-ruby-library-for-translating-strings
I, [2011-03-29T08:51:18.556974 #28817]  INFO -- : 200 - http://rubydoc.info/github/jimmycuadra/to_lang/master/frames
I, [2011-03-29T08:51:27.870143 #28817]  INFO -- : 200 - http://jimmycuadra.com/tags/ruby
I, [2011-03-29T08:51:55.811730 #28817]  INFO -- : 200 - http://jimmycuadra.com/posts/contributing-to-guard-rspec
I, [2011-03-29T08:52:20.622668 #28817]  INFO -- : 200 - http://jimmycuadra.com/posts/the-challenge-of-asset-packaging-on-heroku
I, [2011-03-29T08:52:45.913574 #28817]  INFO -- : 200 - https://github.com/sbecker/asset_packager
I, [2011-03-29T08:52:49.638110 #28817]  INFO -- : 200 - https://github.com/amasses/heroku_asset_packager
I, [2011-03-29T08:52:53.635611 #28817]  INFO -- : 200 - https://github.com/mgomes/heroku_asset_cacher
I, [2011-03-29T08:52:57.383869 #28817]  INFO -- : 200 - https://github.com/documentcloud/jammit/
I, [2011-03-29T08:53:01.171662 #28817]  INFO -- : 200 - https://github.com/mishoo/UglifyJS
I, [2011-03-29T08:53:06.370768 #28817]  INFO -- : 200 - https://github.com/joyent/node
W, [2011-03-29T08:53:06.371782 #28817]  WARN -- : 301 - https://github.com/ry/node - Called from: http://jimmycuadra.com/posts/the-challenge-of-asset-packaging-on-heroku - Following redirection to: https://github.com/joyent/node
I, [2011-03-29T08:53:10.147776 #28817]  INFO -- : 200 - https://github.com/aler/therubyracer-heroku
I, [2011-03-29T08:53:13.922657 #28817]  INFO -- : 200 - https://github.com/lautis/uglifier
I, [2011-03-29T08:53:17.118068 #28817]  INFO -- : 200 - http://jimmycuadra.com/tags/performance
I, [2011-03-29T08:53:59.705884 #28817]  INFO -- : 200 - http://jimmycuadra.com/tags/heroku
I, [2011-03-29T08:54:42.404713 #28817]  INFO -- : 200 - http://jimmycuadra.com/tags/ruby-on-rails
I, [2011-03-29T08:55:10.179105 #28817]  INFO -- : 200 - http://jimmycuadra.com/posts/screencast-actionmailer-and-gmail
I, [2011-03-29T08:55:35.676673 #28817]  INFO -- : 200 - https://github.com/collectiveidea/action_mailer_optional_tls/
W, [2011-03-29T08:55:35.677953 #28817]  WARN -- : 301 - http://github.com/collectiveidea/action_mailer_optional_tls/ - Called from: http://jimmycuadra.com/posts/screencast-actionmailer-and-gmail - Following redirection to: https://github.com/collectiveidea/action_mailer_optional_tls/
I, [2011-03-29T08:55:39.261090 #28817]  INFO -- : 200 - http://nebyoolae.com/
I, [2011-03-29T08:55:42.480881 #28817]  INFO -- : 200 - http://jimmycuadra.com/tags/screencast
I, [2011-03-29T08:56:10.091286 #28817]  INFO -- : 200 - http://jimmycuadra.com/posts/screencast-default-form-values-with-jquery
I, [2011-03-29T08:56:37.786933 #28817]  INFO -- : 200 - http://jimmycuadra.com/tags/jquery
I, [2011-03-29T08:57:02.442540 #28817]  INFO -- : 200 - http://jimmycuadra.com/posts/understanding-jquery-14s-proxy-method
I, [2011-03-29T08:57:27.337585 #28817]  INFO -- : 200 - http://www.prototypejs.org/
I, [2011-03-29T08:57:30.861312 #28817]  INFO -- : 200 - http://jquery.com/
W, [2011-03-29T08:57:30.862656 #28817]  WARN -- : 301 - http://www.jquery.com/ - Called from: http://jimmycuadra.com/posts/understanding-jquery-14s-proxy-method - Following redirection to: http://jquery.com/
I, [2011-03-29T08:57:34.562546 #28817]  INFO -- : 200 - http://jquery14.com/day-01/jquery-14
I, [2011-03-29T08:57:40.841179 #28817]  INFO -- : 200 - http://jimmycuadra.com/tags/javascript
I, [2011-03-29T08:58:05.691616 #28817]  INFO -- : 200 - http://jimmycuadra.com/posts/keeping-track-of-javascript-event-handlers
I, [2011-03-29T08:58:31.669789 #28817]  INFO -- : 200 - http://www.w3.org/TR/2001/WD-DOM-Level-3-Events-20010823/events.html#Events-EventListenerList
I, [2011-03-29T08:58:37.873656 #28817]  INFO -- : 200 - http://jimmycuadra.com/tags
I, [2011-03-29T08:59:02.459848 #28817]  INFO -- : 200 - http://jimmycuadra.com/tags/browsers
I, [2011-03-29T08:59:27.149907 #28817]  INFO -- : 200 - http://jimmycuadra.com/posts/do-websites-need-to-look-the-same-in-every-browser
I, [2011-03-29T08:59:52.408214 #28817]  INFO -- : 200 - http://www.crockford.com/
I, [2011-03-29T08:59:55.652296 #28817]  INFO -- : 200 - http://dowebsitesneedtolookexactlythesameineverybrowser.com/
I, [2011-03-29T09:00:05.531665 #28817]  INFO -- : 200 - http://colchambers.blogspot.com/
I, [2011-03-29T09:00:08.785263 #28817]  INFO -- : 200 - http://twitter.com/vitaminjeff
I, [2011-03-29T09:00:36.043545 #28817]  INFO -- : 200 - http://jimmycuadra.com/posts/can-google-chrome-frame-really-help
I, [2011-03-29T09:01:01.650887 #28817]  INFO -- : 200 - http://blog.chromium.org/2009/09/introducing-google-chrome-frame.html
I, [2011-03-29T09:01:05.417239 #28817]  INFO -- : 200 - http://www.bringdownie6.com/
I, [2011-03-29T09:01:08.754120 #28817]  INFO -- : 200 - http://ie6update.com/
I, [2011-03-29T09:01:12.350804 #28817]  INFO -- : 200 - http://arstechnica.com/microsoft/news/2009/09/microsoft-google-chrome-frame-makes-ie-less-secure.ars
I, [2011-03-29T09:01:16.592030 #28817]  INFO -- : 200 - http://almost.done21.com/2009/04/announcing-ie6-update-help-kill-internet-explorer-6/
I, [2011-03-29T09:01:19.901794 #28817]  INFO -- : 200 - http://www.youtube.com/watch?v=o4MwTvtyrUQ
/Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1266:in `addr_port': undefined method `+' for nil:NilClass (NoMethodError)
    from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1201:in `begin_transport'
    from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1180:in `transport_request'
    from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1169:in `request'
    from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1162:in `block in request'
    from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:627:in `start'
    from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1160:in `request'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/request.rb:28:in `perform_request'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/request.rb:8:in `get'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:38:in `add_status_code'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:32:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:40:in `add_status_code'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:32:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:40:in `add_status_code'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:32:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:18:in `validate'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/bin/rawler:32:in `<top (required)>'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/bin/rawler:19:in `load'
    from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/bin/rawler:19:in `<main>'

Authorization header is sent even without --username and --password 

rawler https://nativescript.org/

fails because the request looks like this:

GET https://www.nativescript.org/ HTTP/1.1
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
User-Agent: Ruby
Authorization: Basic Og==
Connection: close
Host: www.nativescript.org

The server returns HTTP/500 only because of Authorization: Basic Og==.

Exception thrown when server doesn't answer

I got an exception thrown when checking this (old) link (where the server doesn't answer):
http://www.lcia-arbitration.com/

The error:

/Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:581:in `initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError)
        from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:581:in `open'
        from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:581:in `block in connect'
        from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/timeout.rb:44:in `timeout'
        from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/timeout.rb:82:in `timeout'
        from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:581:in `connect'
        from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:574:in `do_start'
        from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:563:in `start'
        from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:1094:in `request'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/request.rb:28:in `perform_request'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/request.rb:8:in `get'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:38:in `add_status_code'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:32:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:17:in `validate'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/bin/rawler:33:in `<top (required)>'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/bin/rawler:19:in `load'
        from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/bin/rawler:19:in `<main>'

Base tag is not taken into consideration

I was rawling http://www.dermatix.gr and found that while there is a <base href="http://www.dermatix.gr" /> it was not taken into consideration when it tested <a href="ta-proionta-dermatix/">DERMATIX</a>.

The result was that it was looking for http://www.dermatix.gr/ta-proionta-dermatix/ta-proionta-dermatix which of course is wrong.

URLs with percent are parsed incorrectly

Hi,

we have some HTML that contains links like these:

<a href="http://link.springer.com/article/10.1007%2Fs12064-012-0164-1">foo</a>

This link is found by rawler, but it is re-encoded to http://link.springer.com/article/10.1007%252Fs12064-012-0164-1. Note: %2 before vs. %252 after.

The check is then performed using the incorrect URL containing %252. Of course, it yields a 404 and reports the link as broken, while browsers and the W3C validator consider it valid.

I traced it down to this line: https://github.com/oscardelben/rawler/blob/master/lib/rawler/crawler.rb#L44

Running Ruby 2.3.1.

http_proxy?

Is there any way to use this behind a firewall, with an http_proxy?
Thanks.

Fails on JavaScript URLs

Hey,

Rawler fails on href="javascript:someFunction();" type of URLs, see:

http://lothringer13.de/halle/archiv/
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/uri/generic.rb:732:in `merge': bad URI(is not URI?): javascript:linkTo_UnCryptMailto('nbjmup;jogpAmpuisjohfs.esfj{fio/dpn'); (URI::InvalidURIError)

Include page being checked in rawler's output

Oscar,

Thank you for the fixes! There's only one other thing I'm missing: When there's, say, a broken link, currently there is no way of knowing what page of the site the link is on...

...So, it would be great to include this in rawler's output. Don't you agree?

The app can not rescue Errno::ETIMEDOUT exception

The app can not rescue Errno::ETIMEDOUT exception

I got this exception first with this installed gem, then I test the current latest code, still got this issue.

Why can't the code rescue Errno::ETIMEDOUT exception ? Should we add code for method perform_request in request.rb file ?

usermatoMacBook-Pro:rawler qichunren$ ./bin/rawler http://twitter.com
/Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:644:in `initialize': Operation timed out - connect(2) (Errno::ETIMEDOUT)
    from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:644:in `open'
    from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:644:in `block in connect'
    from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/timeout.rb:44:in `timeout'
    from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/timeout.rb:87:in `timeout'
    from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:644:in `connect'
    from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:637:in `do_start'
    from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:626:in `start'
    from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:1160:in `request'
    from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/request.rb:28:in `perform_request'
    from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/request.rb:12:in `head'
    from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/crawler.rb:40:in `not_html?'
    from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/crawler.rb:12:in `links'
    from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/base.rb:23:in `validate_links_in_page'
    from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/base.rb:17:in `validate'
    from ./bin/rawler:32:in `'

rawler fails on some URL

rawler http://www.wopata.com will fail at:

200 - http://www.biogaran.fr/kelmed/moteur
/Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:38:in block in add_status_code': undefined methodsize' for nil:NilClass (NoMethodError)
from /Users/jetienne/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:627:in start' from /Users/jetienne/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:490:instart'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:37:in add_status_code' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:27:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:invalidate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:ineach'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:14:invalidate'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/bin/rawler:11:in <top (required)>' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/bin/rawler:19:inload'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/bin/rawler:19:in `

'

Bug when page is redirected?

It's happening for me with:
$ rawler http://arbitragem.pt/links/index.php

...rawler says that lots of links are broken.

You see http://arbitragem.pt/index.php changes the location to http://arbitragem.pt/noticias/index.php
...I think that's what's causing the problem.

Let me know if you need more info.

You should not have to specify http://

[piotr@vaygr] ~ ruby-1.9.2-p136 $ rawler layer22.com
/Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1266:in addr_port': undefined method+' for nil:NilClass (NoMethodError)
from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1201:in begin_transport' from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1180:intransport_request'
from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1169:in request' from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1162:inblock in request'
from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:627:in start' from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1160:inrequest'
from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/request.rb:28:in perform_request' from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/request.rb:12:inhead'
from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/crawler.rb:40:in not_html?' from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/crawler.rb:12:inlinks'
from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/base.rb:23:in validate_links_in_page' from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/base.rb:17:invalidate'
from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/bin/rawler:32:in <top (required)>' from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/bin/rawler:19:inload'
from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/bin/rawler:19:in `

'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.