m1n0 / middleman-robots

create robots.txt when do 'build'.

Middleman::Robots

middleman-robots is an extension of Middleman. This can create robots.txt when build.

Installation

Add this line to your application's Gemfile:

gem 'middleman-robots'

And then execute:

$ bundle

Or install it yourself as:

$ gem install middleman-robots

Usage

Important - if you set the :build_dir variable, make sure you set it before the middleman-robots settings code, otherwise default /build directory is used. Basic usage:

# config.rb
configure :build do
  set :build_dir, 'some-path/' #optional
  
  activate :robots, :rules => [
    {:user_agent => '*', :allow => %w(/)}
  ],
  :sitemap => "http://example.com/sitemap.xml"
end

Created robots.txt:

User-Agent: *
Allow: /

Sitemap: http://example.com/sitemap.xml

You can use options, :rules {[:user_agent(string), :allow(array), :disallow(array)]} and :sitemap. Like this:

# config.rb
configure :build do
  activate :robots,
    :rules => [
      {
        :user_agent => 'Googlebot',
        :disallow =>  %w(tmp/ /something/dir/file_disallow.html),
        :allow =>  %w(allow/ /something/dir/file_allow.html)
      },
      {
        :user_agent => 'Googlebot-Image',
        :disallow =>  %w(tmp/ /something/dir/file_disallow.html),
        :allow =>  %w(allow/ /something/dir/file_allow.html)
      }
    ],
    :sitemap => "http://example.com/sitemap.xml"
end

Created robots.txt:

User-Agent: Googlebot
Disallow: /tmp/
Disallow: /something/dir/file_disallow.html
Allow: /allow/
Allow: /something/dir/file_allow.html

User-Agent: Googlebot-Image
Disallow: /tmp/
Disallow: /something/dir/file_disallow.html
Allow: /allow/
Allow: /something/dir/file_allow.html

Sitemap: http://example.com/sitemap.xml

Contributing

1. Fork it ( https://github.com/yterajima/middleman-robots/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request