oscardelben / rawler Goto Github PK
View Code? Open in Web Editor NEWRawler is a tool that crawls the links of your website
Home Page: http://oscardelben.com
License: MIT License
Rawler is a tool that crawls the links of your website
Home Page: http://oscardelben.com
License: MIT License
[piotr@vaygr] ~ ruby-1.9.2-p136 $ rawler layer22.com
/Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1266:in addr_port': undefined method
+' for nil:NilClass (NoMethodError)
from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1201:in begin_transport' from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1180:in
transport_request'
from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1169:in request' from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1162:in
block in request'
from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:627:in start' from /Users/piotr/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1160:in
request'
from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/request.rb:28:in perform_request' from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/request.rb:12:in
head'
from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/crawler.rb:40:in not_html?' from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/crawler.rb:12:in
links'
from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/base.rb:23:in validate_links_in_page' from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/lib/rawler/base.rb:17:in
validate'
from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.0.4/bin/rawler:32:in <top (required)>' from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/bin/rawler:19:in
load'
from /Users/piotr/.rvm/gems/ruby-1.9.2-p136/bin/rawler:19:in `
If the page link is http://example.com/somepage and it has anchor links like: http://example.com/somepage#anchor1 and http://example.com/somepage#anchor2 both links are considered as 404
I got this exception first with this installed gem, then I test the current latest code, still got this issue.
Why can't the code rescue Errno::ETIMEDOUT exception ? Should we add code for method perform_request in request.rb file ?
usermatoMacBook-Pro:rawler qichunren$ ./bin/rawler http://twitter.com /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:644:in `initialize': Operation timed out - connect(2) (Errno::ETIMEDOUT) from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:644:in `open' from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:644:in `block in connect' from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/timeout.rb:44:in `timeout' from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/timeout.rb:87:in `timeout' from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:644:in `connect' from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:637:in `do_start' from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:626:in `start' from /Users/qichunren/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:1160:in `request' from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/request.rb:28:in `perform_request' from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/request.rb:12:in `head' from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/crawler.rb:40:in `not_html?' from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/crawler.rb:12:in `links' from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/base.rb:23:in `validate_links_in_page' from /Users/qichunren/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.4/lib/rawler/base.rb:17:in `validate' from ./bin/rawler:32:in `'
Is there any way to use this behind a firewall, with an http_proxy?
Thanks.
What the title says :)
I got an exception thrown when checking this (old) link (where the server doesn't answer):
http://www.lcia-arbitration.com/
The error:
/Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:581:in `initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError)
from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:581:in `open'
from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:581:in `block in connect'
from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/timeout.rb:44:in `timeout'
from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/timeout.rb:82:in `timeout'
from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:581:in `connect'
from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:574:in `do_start'
from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:563:in `start'
from /Users/andy/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/1.9.1/net/http.rb:1094:in `request'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/request.rb:28:in `perform_request'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/request.rb:8:in `get'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:38:in `add_status_code'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:32:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:33:in `validate_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:24:in `block in validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `each'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:23:in `validate_links_in_page'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/lib/rawler/base.rb:17:in `validate'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/gems/rawler-0.0.7/bin/rawler:33:in `<top (required)>'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/bin/rawler:19:in `load'
from /Users/andy/.rvm/gems/ruby-1.9.1-p378%cur/bin/rawler:19:in `<main>'
rawler https://nativescript.org/
fails because the request looks like this:
GET https://www.nativescript.org/ HTTP/1.1
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
User-Agent: Ruby
Authorization: Basic Og==
Connection: close
Host: www.nativescript.org
The server returns HTTP/500 only because of Authorization: Basic Og==
.
I was rawling http://www.dermatix.gr and found that while there is a <base href="http://www.dermatix.gr" />
it was not taken into consideration when it tested <a href="ta-proionta-dermatix/">DERMATIX</a>
.
The result was that it was looking for http://www.dermatix.gr/ta-proionta-dermatix/ta-proionta-dermatix
which of course is wrong.
$ bundle exec rawler --ignore_fragments http://google.com
Error: unknown argument '--ignore_fragments'.
Try --help for help.
$ bundle exec rawler --ignore_fragments http://google.com
Error: unknown argument '--ignore_fragments'.
Try --help for help.
--help
doesn't print the option, despite it being documented on GitHub.
$ bundle exec rawler --help
Rawler is a command line utility for parsing links on a website
Usage:
rawler http://example.com [options]
where [options] are:
--username, -u <s>: HTTP Basic Username
--password, -p <s>: HTTP Basic Password
--wait, -w <f>: Seconds to wait between requests, may be fractional
e.g. '1.5' (default: 3.0)
--log, -l: Log results to file rawler_log.txt
--logfile, -o <s>: Specify logfile, implies --log (default:
rawler_log.txt)
--css, -c: Check CSS links
--skip, -s <s>: Skip URLs that match a pattern
--iskip, -i <s>: Skip URLs that match a case insensitive pattern
--include, -n <s>: Only include URLS that match a pattern
--iinclude, -d <s>: Only include URLS that match a case insensitive
pattern. Equivalent to '--include
^http://mysite.com/*'.
--local, -a: Restrict to the given URL and below
--version, -v: Print version and exit
--help, -h: Show this message
It's happening for me with:
$ rawler http://arbitragem.pt/links/index.php
...rawler says that lots of links are broken.
You see http://arbitragem.pt/index.php
changes the location to http://arbitragem.pt/noticias/index.php
...I think that's what's causing the problem.
Let me know if you need more info.
Oscar,
Thank you for the fixes! There's only one other thing I'm missing: When there's, say, a broken link, currently there is no way of knowing what page of the site the link is on...
...So, it would be great to include this in rawler's output. Don't you agree?
Got an exception while crawling my site. I'm not sure which link confused it by I figured I'd pass along the stack trace:
[rvm 1.9.2] ~ $ rawler http://jimmycuadra.com/
I, [2011-03-29T08:38:26.407541 #28817] INFO -- : 200 - http://jimmycuadra.com/
I, [2011-03-29T08:38:30.208174 #28817] INFO -- : 200 - http://jimmycuadra.com/projects
I, [2011-03-29T08:38:36.770008 #28817] INFO -- : 200 - http://jimmycuadra.com/about
I, [2011-03-29T08:38:55.920711 #28817] INFO -- : 200 - https://github.com/jimmycuadra
I, [2011-03-29T08:38:59.744387 #28817] INFO -- : 200 - http://twitter.com/jimmycuadra
I, [2011-03-29T08:39:06.076651 #28817] INFO -- : 200 - http://en.wikipedia.org/wiki/San_Diego
I, [2011-03-29T08:39:10.441370 #28817] INFO -- : 200 - http://en.wikipedia.org/wiki/Los_Angeles
I, [2011-03-29T08:39:13.625704 #28817] INFO -- : 200 - http://ucsd.edu/
I, [2011-03-29T08:39:18.099691 #28817] INFO -- : 200 - http://www.ruby-lang.org/en/
W, [2011-03-29T08:39:18.100729 #28817] WARN -- : 302 - http://www.ruby-lang.org/ - Called from: http://jimmycuadra.com/about - Following redirection to: http://www.ruby-lang.org/en/
I, [2011-03-29T08:39:21.508748 #28817] INFO -- : 200 - http://rubyonrails.org/
I, [2011-03-29T08:39:25.031023 #28817] INFO -- : 200 - http://git-scm.com/
I, [2011-03-29T08:39:34.318888 #28817] INFO -- : 200 - http://www.linkedin.com/in/jimmycuadra
I, [2011-03-29T08:39:37.616162 #28817] INFO -- : 200 - http://jimmycuadra.com/login
I, [2011-03-29T08:40:05.612382 #28817] INFO -- : 200 - http://www.apple.com/
W, [2011-03-29T08:40:08.952060 #28817] WARN -- : 301 - http://www.rubyonrails.org/ - Called from: http://jimmycuadra.com/login - Following redirection to: http://rubyonrails.org/
I, [2011-03-29T08:40:12.328434 #28817] INFO -- : 200 - http://www.webkit.org/
W, [2011-03-29T08:40:12.329675 #28817] WARN -- : 301 - http://webkit.org/ - Called from: http://jimmycuadra.com/login - Following redirection to: http://www.webkit.org/
I, [2011-03-29T08:40:45.985793 #28817] INFO -- : 200 - http://morethingsneed.to/
I, [2011-03-29T08:40:53.375976 #28817] INFO -- : 200 - http://eventful.com/
I, [2011-03-29T08:40:59.556434 #28817] INFO -- : 200 - http://rady.ucsd.edu/
I, [2011-03-29T08:41:09.337865 #28817] INFO -- : 200 - https://github.com/jimmycuadra/to_lang
I, [2011-03-29T08:41:12.993974 #28817] INFO -- : 200 - http://ruby5.envylabs.com/episodes/141-episode-138-january-4-2011/stories/1229-to_lang-a-ruby-interface-to-google-translate
I, [2011-03-29T08:41:16.124937 #28817] INFO -- : 200 - http://news.ycombinator.com/item?id=2095006
I, [2011-03-29T08:41:19.907478 #28817] INFO -- : 200 - https://github.com/jimmycuadra/changesets
I, [2011-03-29T08:41:23.646278 #28817] INFO -- : 200 - https://github.com/jimmycuadra/jimmycuadra.com
I, [2011-03-29T08:41:59.920392 #28817] INFO -- : 200 - http://jimmycuadra.com/?page=2
I, [2011-03-29T08:42:27.815614 #28817] INFO -- : 200 - http://jimmycuadra.com/?page=3
I, [2011-03-29T08:42:58.577409 #28817] INFO -- : 200 - http://jimmycuadra.com/?page=4
I, [2011-03-29T08:43:32.515108 #28817] INFO -- : 200 - http://jimmycuadra.com/?page=5
I, [2011-03-29T08:44:09.485068 #28817] INFO -- : 200 - http://jimmycuadra.com/?page=6
I, [2011-03-29T08:44:49.289188 #28817] INFO -- : 200 - http://jimmycuadra.com/?page=7
I, [2011-03-29T08:45:32.111589 #28817] INFO -- : 200 - http://jimmycuadra.com/?page=8
I, [2011-03-29T08:46:18.082790 #28817] INFO -- : 200 - http://jimmycuadra.com/?page=9
I, [2011-03-29T08:47:07.046778 #28817] INFO -- : 200 - http://jimmycuadra.com/?page=10
I, [2011-03-29T08:47:58.922460 #28817] INFO -- : 200 - http://jimmycuadra.com/?page=11
I, [2011-03-29T08:48:53.794334 #28817] INFO -- : 200 - http://jimmycuadra.com/?page=12
I, [2011-03-29T08:49:54.517105 #28817] INFO -- : 200 - http://jimmycuadra.com/posts/to-lang-030-featuring-batch-translations-and-a-command-line-utility
I, [2011-03-29T08:50:22.255492 #28817] INFO -- : 200 - http://rubydoc.info/gems/to_lang/0.3.0/frames
I, [2011-03-29T08:50:25.932320 #28817] INFO -- : 200 - https://rubygems.org/gems/to_lang
I, [2011-03-29T08:50:29.171260 #28817] INFO -- : 200 - http://jimmycuadra.com/tags/to-lang
I, [2011-03-29T08:50:53.879056 #28817] INFO -- : 200 - http://jimmycuadra.com/posts/to-lang-a-ruby-library-for-translating-strings
I, [2011-03-29T08:51:18.556974 #28817] INFO -- : 200 - http://rubydoc.info/github/jimmycuadra/to_lang/master/frames
I, [2011-03-29T08:51:27.870143 #28817] INFO -- : 200 - http://jimmycuadra.com/tags/ruby
I, [2011-03-29T08:51:55.811730 #28817] INFO -- : 200 - http://jimmycuadra.com/posts/contributing-to-guard-rspec
I, [2011-03-29T08:52:20.622668 #28817] INFO -- : 200 - http://jimmycuadra.com/posts/the-challenge-of-asset-packaging-on-heroku
I, [2011-03-29T08:52:45.913574 #28817] INFO -- : 200 - https://github.com/sbecker/asset_packager
I, [2011-03-29T08:52:49.638110 #28817] INFO -- : 200 - https://github.com/amasses/heroku_asset_packager
I, [2011-03-29T08:52:53.635611 #28817] INFO -- : 200 - https://github.com/mgomes/heroku_asset_cacher
I, [2011-03-29T08:52:57.383869 #28817] INFO -- : 200 - https://github.com/documentcloud/jammit/
I, [2011-03-29T08:53:01.171662 #28817] INFO -- : 200 - https://github.com/mishoo/UglifyJS
I, [2011-03-29T08:53:06.370768 #28817] INFO -- : 200 - https://github.com/joyent/node
W, [2011-03-29T08:53:06.371782 #28817] WARN -- : 301 - https://github.com/ry/node - Called from: http://jimmycuadra.com/posts/the-challenge-of-asset-packaging-on-heroku - Following redirection to: https://github.com/joyent/node
I, [2011-03-29T08:53:10.147776 #28817] INFO -- : 200 - https://github.com/aler/therubyracer-heroku
I, [2011-03-29T08:53:13.922657 #28817] INFO -- : 200 - https://github.com/lautis/uglifier
I, [2011-03-29T08:53:17.118068 #28817] INFO -- : 200 - http://jimmycuadra.com/tags/performance
I, [2011-03-29T08:53:59.705884 #28817] INFO -- : 200 - http://jimmycuadra.com/tags/heroku
I, [2011-03-29T08:54:42.404713 #28817] INFO -- : 200 - http://jimmycuadra.com/tags/ruby-on-rails
I, [2011-03-29T08:55:10.179105 #28817] INFO -- : 200 - http://jimmycuadra.com/posts/screencast-actionmailer-and-gmail
I, [2011-03-29T08:55:35.676673 #28817] INFO -- : 200 - https://github.com/collectiveidea/action_mailer_optional_tls/
W, [2011-03-29T08:55:35.677953 #28817] WARN -- : 301 - http://github.com/collectiveidea/action_mailer_optional_tls/ - Called from: http://jimmycuadra.com/posts/screencast-actionmailer-and-gmail - Following redirection to: https://github.com/collectiveidea/action_mailer_optional_tls/
I, [2011-03-29T08:55:39.261090 #28817] INFO -- : 200 - http://nebyoolae.com/
I, [2011-03-29T08:55:42.480881 #28817] INFO -- : 200 - http://jimmycuadra.com/tags/screencast
I, [2011-03-29T08:56:10.091286 #28817] INFO -- : 200 - http://jimmycuadra.com/posts/screencast-default-form-values-with-jquery
I, [2011-03-29T08:56:37.786933 #28817] INFO -- : 200 - http://jimmycuadra.com/tags/jquery
I, [2011-03-29T08:57:02.442540 #28817] INFO -- : 200 - http://jimmycuadra.com/posts/understanding-jquery-14s-proxy-method
I, [2011-03-29T08:57:27.337585 #28817] INFO -- : 200 - http://www.prototypejs.org/
I, [2011-03-29T08:57:30.861312 #28817] INFO -- : 200 - http://jquery.com/
W, [2011-03-29T08:57:30.862656 #28817] WARN -- : 301 - http://www.jquery.com/ - Called from: http://jimmycuadra.com/posts/understanding-jquery-14s-proxy-method - Following redirection to: http://jquery.com/
I, [2011-03-29T08:57:34.562546 #28817] INFO -- : 200 - http://jquery14.com/day-01/jquery-14
I, [2011-03-29T08:57:40.841179 #28817] INFO -- : 200 - http://jimmycuadra.com/tags/javascript
I, [2011-03-29T08:58:05.691616 #28817] INFO -- : 200 - http://jimmycuadra.com/posts/keeping-track-of-javascript-event-handlers
I, [2011-03-29T08:58:31.669789 #28817] INFO -- : 200 - http://www.w3.org/TR/2001/WD-DOM-Level-3-Events-20010823/events.html#Events-EventListenerList
I, [2011-03-29T08:58:37.873656 #28817] INFO -- : 200 - http://jimmycuadra.com/tags
I, [2011-03-29T08:59:02.459848 #28817] INFO -- : 200 - http://jimmycuadra.com/tags/browsers
I, [2011-03-29T08:59:27.149907 #28817] INFO -- : 200 - http://jimmycuadra.com/posts/do-websites-need-to-look-the-same-in-every-browser
I, [2011-03-29T08:59:52.408214 #28817] INFO -- : 200 - http://www.crockford.com/
I, [2011-03-29T08:59:55.652296 #28817] INFO -- : 200 - http://dowebsitesneedtolookexactlythesameineverybrowser.com/
I, [2011-03-29T09:00:05.531665 #28817] INFO -- : 200 - http://colchambers.blogspot.com/
I, [2011-03-29T09:00:08.785263 #28817] INFO -- : 200 - http://twitter.com/vitaminjeff
I, [2011-03-29T09:00:36.043545 #28817] INFO -- : 200 - http://jimmycuadra.com/posts/can-google-chrome-frame-really-help
I, [2011-03-29T09:01:01.650887 #28817] INFO -- : 200 - http://blog.chromium.org/2009/09/introducing-google-chrome-frame.html
I, [2011-03-29T09:01:05.417239 #28817] INFO -- : 200 - http://www.bringdownie6.com/
I, [2011-03-29T09:01:08.754120 #28817] INFO -- : 200 - http://ie6update.com/
I, [2011-03-29T09:01:12.350804 #28817] INFO -- : 200 - http://arstechnica.com/microsoft/news/2009/09/microsoft-google-chrome-frame-makes-ie-less-secure.ars
I, [2011-03-29T09:01:16.592030 #28817] INFO -- : 200 - http://almost.done21.com/2009/04/announcing-ie6-update-help-kill-internet-explorer-6/
I, [2011-03-29T09:01:19.901794 #28817] INFO -- : 200 - http://www.youtube.com/watch?v=o4MwTvtyrUQ
/Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1266:in `addr_port': undefined method `+' for nil:NilClass (NoMethodError)
from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1201:in `begin_transport'
from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1180:in `transport_request'
from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1169:in `request'
from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1162:in `block in request'
from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:627:in `start'
from /Users/jimmy/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/net/http.rb:1160:in `request'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/request.rb:28:in `perform_request'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/request.rb:8:in `get'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:38:in `add_status_code'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:32:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:40:in `add_status_code'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:32:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:40:in `add_status_code'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:32:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:33:in `validate_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:25:in `block in validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `each'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:24:in `validate_links_in_page'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/lib/rawler/base.rb:18:in `validate'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/gems/rawler-0.1.0/bin/rawler:32:in `<top (required)>'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/bin/rawler:19:in `load'
from /Users/jimmy/.rvm/gems/ruby-1.9.2-p136/bin/rawler:19:in `<main>'
fo@li175-103:~/tmp/geeks$ rawler http://geeks.aretotally.in/
/usr/lib/ruby/1.8/uri/generic.rb:195:in initialize': the scheme http does not accept registry part: AppDemoingIntegrationBetweenGrailswithFacebook'sGraph'sAPIusingGroovy (or bad hostname?) (URI::InvalidURIError) from /usr/lib/ruby/1.8/uri/http.rb:78:in
initialize'
from /usr/lib/ruby/1.8/uri/common.rb:488:in new' from /usr/lib/ruby/1.8/uri/common.rb:488:in
parse'
why let a good .travis.yml go to waste?
http://about.travis-ci.org/docs/user/getting-started/#Step-two%3A-Activate-GitHub-Service-Hook
rawler http://www.wopata.com will fail at:
200 - http://www.biogaran.fr/kelmed/moteur
/Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:38:in block in add_status_code': undefined method
size' for nil:NilClass (NoMethodError)
from /Users/jetienne/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:627:in start' from /Users/jetienne/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/net/http.rb:490:in
start'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:37:in add_status_code' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:27:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:28:in
validate_page'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:21:in block in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in
each'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:20:in validate_links_in_page' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/lib/rawler/base.rb:14:in
validate'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/gems/rawler-0.0.2/bin/rawler:11:in <top (required)>' from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/bin/rawler:19:in
load'
from /Users/jetienne/.rvm/gems/ruby-1.9.2-p0/bin/rawler:19:in `
Hey,
Rawler fails on href="javascript:someFunction();" type of URLs, see:
http://lothringer13.de/halle/archiv/
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/uri/generic.rb:732:in `merge': bad URI(is not URI?): javascript:linkTo_UnCryptMailto('nbjmup;jogpAmpuisjohfs.esfj{fio/dpn'); (URI::InvalidURIError)
Hi,
we have some HTML that contains links like these:
<a href="http://link.springer.com/article/10.1007%2Fs12064-012-0164-1">foo</a>
This link is found by rawler, but it is re-encoded to http://link.springer.com/article/10.1007%252Fs12064-012-0164-1
. Note: %2
before vs. %252
after.
The check is then performed using the incorrect URL containing %252
. Of course, it yields a 404 and reports the link as broken, while browsers and the W3C validator consider it valid.
I traced it down to this line: https://github.com/oscardelben/rawler/blob/master/lib/rawler/crawler.rb#L44
Running Ruby 2.3.1.
/var/lib/gems/2.7.0/gems/rawler-0.1.8/lib/rawler/crawler.rb:51: warning: URI.escape is obsolete
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.