chriskite / robotex Goto Github PK

Bug Steps:
Attempt to extract the delay for a user agent from a robots.txt file where the crawl-delay for a specific user agent appears after the rule for *. Example:
https://www.archives.gov/robots.txt

User-agent: *
Crawl-delay: 10
...
User-agent: usasearch
Crawl-delay: 2

Expected results:

> Robotex.new('usasearch').delay('https://www.archives.gov')
#=> 2

Actual results:

> Robotex.new('usasearch').delay('https://www.archives.gov')
#=> 10

See https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-user-agents for precedence rules.

If Ruby is in warnings mode, two warnings are output. Here's a script to reproduce the problem. This script is straight out of the examples, except it's run in warnings mode:

#!/usr/bin/ruby -w
require 'robotex'
robotex = Robotex.new "My User Agent"
robotex.allowed?("http://www.example.com/foo")

These are the warnings given:

/var/lib/gems/2.3.0/gems/robotex-1.0.0/lib/robotex.rb:103: warning: assigned but unused variable - io
/var/lib/gems/2.3.0/gems/robotex-1.0.0/lib/robotex.rb:115: warning: instance variable @timeout not initialized

@chriskite , is this repo being maintained? Any interest in adding a maintainer or transferring maintenance to someone else?

Rules with trailing comments are not being applied correctly.

Example:
https://ask.fmcsa.dot.gov/robots.txt

User-agent: *                    # ADDED BY HMS
Disallow: /                      # ADDED BY HMS

The above rule should disallow everything on the site. Instead:

> robotex = Robotex.new('testing')
> robotex.allowed?('https://ask.fmcsa.dot.gov/anything')
true

PR: #3

Reference: https://developers.google.com/search/reference/robots_txt#file-format

It would be very useful to have a method to extract the sitemaps listed in a robots.txt file, per the sitemaps specification: https://www.sitemaps.org/protocol.html#submit_robots

Example usage:
http://www.nytimes.com/robots.txt

Sitemap: http://spiderbites.nytimes.com/sitemaps/www.nytimes.com/sitemap.xml.gz
Sitemap: http://www.nytimes.com/sitemaps/sitemap_news/sitemap.xml.gz
Sitemap: http://spiderbites.nytimes.com/sitemaps/sitemap_video/sitemap.xml.gz
Sitemap: http://spiderbites.nytimes.com/sitemaps/www.nytimes.com_realestate/sitemap.xml.gz
Sitemap: http://spiderbites.nytimes.com/sitemaps/www.nytimes.com/2016_election_sitemap.xml.gz

> robotex = Robotex.new
> robotex.sitemaps('http://www.nytimes.com')
=> ["http://spiderbites.nytimes.com/sitemaps/www.nytimes.com/sitemap.xml.gz",
 "http://www.nytimes.com/sitemaps/sitemap_news/sitemap.xml.gz",
 "http://spiderbites.nytimes.com/sitemaps/sitemap_video/sitemap.xml.gz",
 "http://spiderbites.nytimes.com/sitemaps/www.nytimes.com_realestate/sitemap.xml.gz",
 "http://spiderbites.nytimes.com/sitemaps/www.nytimes.com/2016_election_sitemap.xml.gz"]

Robotex does not follow redirected robots.txt pages, which can result in pages erroneously appearing to be allowed?. Example:

In https://www.yelp.com/robots.txt:
Disallow: /biz_link

> robotex = Robotex.new "My User Agent"

> robotex.allowed?("https://www.yelp.com/biz_link")
false #correct

> robotex = Robotex.new "My User Agent"

> robotex.allowed?("http://www.yelp.com/biz_link")
true #incorrect