laurilehmijoki / s3_website Goto Github PK

Hi,

I am having concurrnecy issues, tried this in yam
concurrency_level: 1
disable_parallel_processing: true

but still get the error, only way around this is:

disable_parallel_processing=true s3_website push

Fetching: filey-diff-1.4.0.gem (100%)
Fetching: s3_website-1.3.2.gem (100%)
Successfully installed filey-diff-1.4.0
Successfully installed s3_website-1.3.2
Gems updated: filey-diff, s3_website

Hi,

Would it be feasible to add a configuration parameter so html files with location in the format

dir/index.html

Are uploaded just as

dir

with text/html as the mime type? My reason is to avoid the 301 redirect generated by S3 from "dir" to "dir/" when visiting such a URL in the browser.

Would any of the authors have any guidance on what to do to add this enhancement? (Or even explanations why this isn't possible or a good idea) I'm not familiar with Ruby, but happy to learn.

Thanks,

Michal.

On mac I often get this error when s3_website push. Anybody has any idea how I could fix this?

Deploying _site/* to webrc.co
Calculating diff //Users/sabondano/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/openssl/buffering.rb:145:in sysread_nonblock': end of file reached (EOFError) from /Users/sabondano/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/openssl/buffering.rb:145:inread_nonblock'
from /Users/sabondano/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/protocol.rb:135:in rbuf_fill' from /Users/sabondano/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/protocol.rb:116:inreaduntil'
from /Users/sabondano/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/protocol.rb:126:in readline' from /Users/sabondano/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/http.rb:2219:inread_status_line'
from /Users/sabondano/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/http.rb:2208:in read_new' from /Users/sabondano/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/http.rb:1191:intransport_request'
from /Users/sabondano/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/http.rb:1177:in request' from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/net/http/connection_pool/session.rb:64:inrequest'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:173:in block in request' from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:194:insession_for'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:171:in request' from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/net/http/connection_pool/connection.rb:173:inrequest'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/aws/core/http/net_http_handler.rb:66:in handle' from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:231:inblock in make_sync_request'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:267:in retry_server_errors' from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:227:inmake_sync_request'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:472:in block (2 levels) in client_request' from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:355:inlog_client_request'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:441:in block in client_request' from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:337:inreturn_or_raise'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:440:in client_request' from (eval):3:inhead_object'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/aws/s3/s3_object.rb:294:in head' from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/aws-sdk-1.8.5/lib/aws/s3/s3_object.rb:313:inlast_modified'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/filey-diff-1.4.1/lib/filey-diff/data-sources/aws_sdk_s3.rb:77:in last_modified_and_md5' from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/filey-diff-1.4.1/lib/filey-diff/data-sources/aws_sdk_s3.rb:62:inmap_s3_object_to_filey'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/filey-diff-1.4.1/lib/filey-diff/data-sources/aws_sdk_s3.rb:17:in block in do_internal_load' from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/filey-diff-1.4.1/lib/filey-diff/data-sources/aws_sdk_s3.rb:26:incall'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/filey-diff-1.4.1/lib/filey-diff/data-sources/aws_sdk_s3.rb:26:in block (2 levels) in in_parallel_or_sequentially' from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/filey-diff-1.4.1/lib/filey-diff/data-sources/aws_sdk_s3.rb:35:incall'
from /Users/sabondano/.rvm/gems/ruby-1.9.2-p320/gems/filey-diff-1.4.1/lib/filey-diff/data-sources/aws_sdk_s3.rb:35:in `block (3 levels) in in_parallel_or_sequentially'

I'm testing this out with Cloudfront in front of S3. Based on my testing, I can't set subdirectory indices, e.g.

example.com/pretty/index.html -> example.com/pretty/

It works on the S3 source bucket, and your documentation of cloudfront_invalidate_root implies that setting these indices is possible. However, I still receive a "No Such Key" error:

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>pretty/</Key>
<RequestId>--</RequestId>
<HostId>

---
</HostId>
</Error>

I've tried invalidations to fix this. How are you able to set the cloudfront folder indices?

Hello There,

I've added s3_endpoint to my s3_website.yml and set it to eu-west-1 which is where my bucket is located. However when applying the config I get the following error:

Applying the configurations in s3_website.yml on the AWS services ...
/Users/robertrawlins/.rvm/gems/ruby-2.0.0-p353@global/gems/configure-s3-website-1.5.1/lib/configure-s3-website/s3_client.rb:136:in `initialize': InvalidS3LocationConstraintError (InvalidS3LocationConstraintError)
    from /Users/robertrawlins/.rvm/gems/ruby-2.0.0-p353@global/gems/configure-s3-website-1.5.1/lib/configure-s3-website/s3_client.rb:159:in `new'
    from /Users/robertrawlins/.rvm/gems/ruby-2.0.0-p353@global/gems/configure-s3-website-1.5.1/lib/configure-s3-website/s3_client.rb:159:in `by_config_source'
    from /Users/robertrawlins/.rvm/gems/ruby-2.0.0-p353@global/gems/configure-s3-website-1.5.1/lib/configure-s3-website/http_helper.rb:4:in `call_s3_api'
    from /Users/robertrawlins/.rvm/gems/ruby-2.0.0-p353@global/gems/configure-s3-website-1.5.1/lib/configure-s3-website/s3_client.rb:34:in `enable_website_configuration'
    from /Users/robertrawlins/.rvm/gems/ruby-2.0.0-p353@global/gems/configure-s3-website-1.5.1/lib/configure-s3-website/s3_client.rb:12:in `configure_website'
    from /Users/robertrawlins/.rvm/gems/ruby-2.0.0-p353@global/gems/configure-s3-website-1.5.1/lib/configure-s3-website/runner.rb:4:in `run'
    from /Users/robertrawlins/.rvm/gems/ruby-2.0.0-p353@global/gems/configure-s3-website-1.5.1/bin/configure-s3-website:12:in `<top (required)>'
    from /Users/robertrawlins/.rvm/rubies/ruby-2.0.0-p353/bin/configure-s3-website:23:in `load'
    from /Users/robertrawlins/.rvm/rubies/ruby-2.0.0-p353/bin/configure-s3-website:23:in `<main>'
    from /Users/robertrawlins/.rvm/gems/ruby-2.0.0-p353@global/bin/ruby_executable_hooks:15:in `eval'
    from /Users/robertrawlins/.rvm/gems/ruby-2.0.0-p353@global/bin/ruby_executable_hooks:15:in `<main>'

Am I doing something wrong here? Is that the wrong identifier for that location?

Thanks,

Robert

Just set it all up - ran for a while trying to push to a bucket and then BOOM...

Here's the error:

/Users/rwboyer/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:878:in `initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError)

Here it is in context

Deploying _site/* to fineart.lesterpickerphoto.com
/Users/rwboyer/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:878:in initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError) from /Users/rwboyer/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:878:inopen'
from /Users/rwboyer/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:878:in block in connect' from /Users/rwboyer/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/timeout.rb:66:intimeout'
from /Users/rwboyer/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:877:in connect' from /Users/rwboyer/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:862:indo_start'
from /Users/rwboyer/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:857:in start' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/net/http/connection_pool/session.rb:118:instart'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:208:in _create_session' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:193:insession_for'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:171:in request' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/net/http/connection_pool/connection.rb:173:inrequest'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/http/net_http_handler.rb:66:in handle' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:231:inblock in make_sync_request'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:267:in retry_server_errors' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:227:inmake_sync_request'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:472:in block (2 levels) in client_request' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:355:inlog_client_request'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:441:in block in client_request' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:337:inreturn_or_raise'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:440:in client_request' from (eval):3:inlist_objects'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/s3/object_collection.rb:297:in list_request' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/s3/paginated_collection.rb:29:in_each_item'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/collection/with_limit_and_next_token.rb:54:in _each_batch' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/collection.rb:82:ineach_batch'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/core/collection.rb:49:in each' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/aws-sdk-1.8.5/lib/aws/s3/object_collection.rb:282:ineach'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/filey-diff-1.2.1/lib/filey-diff/data-sources/aws_sdk_s3.rb:25:in map' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/filey-diff-1.2.1/lib/filey-diff/data-sources/aws_sdk_s3.rb:25:inin_parallel_or_sequentially'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/filey-diff-1.2.1/lib/filey-diff/data-sources/aws_sdk_s3.rb:15:in do_internal_load' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/filey-diff-1.2.1/lib/filey-diff/data-sources/data_source.rb:8:inget_fileys'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/filey-diff-1.2.1/lib/filey-diff/comparison.rb:28:in select_in_outer_array' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/filey-diff-1.2.1/lib/filey-diff/comparison.rb:11:inlist_changed'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/s3_website-1.0.1/lib/s3_website/diff_helper.rb:7:in resolve_files_to_upload' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/s3_website-1.0.1/lib/s3_website/uploader.rb:43:inupload_files'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/s3_website-1.0.1/lib/s3_website/uploader.rb:14:in run' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/s3_website-1.0.1/lib/s3_website/tasks.rb:8:inpush'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/s3_website-1.0.1/bin/s3_website:46:in push' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/thor-0.18.1/lib/thor/command.rb:27:inrun'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/thor-0.18.1/lib/thor/invocation.rb:120:in invoke_command' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/thor-0.18.1/lib/thor.rb:363:indispatch'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/thor-0.18.1/lib/thor/base.rb:439:in start' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/gems/s3_website-1.0.1/bin/s3_website:53:in<top (required)>'
from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/bin/s3_website:23:in load' from /Users/rwboyer/.rvm/gems/ruby-2.0.0-p247/bin/s3_website:23:in

Any ideas or known issues with the versions of ruby/gems that you can see that I use?

Thanks

RB

Many users still use Ruby 1.8.7, because it ships as the system Ruby in many operating systems.

This is where 1.8.7 fails:

$ rvm use 1.8.7
$ bundle install
Fetching gem metadata from http://rubygems.org/........
Fetching gem metadata from http://rubygems.org/..
Resolving dependencies...
Using rake (10.1.0) 
Installing addressable (2.3.4) 
Installing ffi (1.9.0) 
Installing childprocess (0.3.9) 
Installing builder (3.2.2) 
Installing diff-lcs (1.2.4) 
Installing multi_json (1.7.7) 
Installing gherkin (2.12.0) 
Installing cucumber (1.3.2) 
Installing rspec-expectations (2.13.0) 
Installing aruba (0.5.3) 
Installing json (1.8.0) 
Installing mini_portile (0.5.0) 
Installing nokogiri (1.6.0) 
Gem::InstallError: nokogiri requires Ruby version >= 1.9.2.
An error occurred while installing nokogiri (1.6.0), and Bundler cannot continue.
Make sure that `gem install nokogiri -v '1.6.0'` succeeds before bundling.

First time user here. After the first s3 cfg apply command I got the following message and after a long time nothing happened at all:

Applying the configurations in s3_website.yml on the AWS services ...

After, I was tired of waiting I was able to continue after clicking enter but didnt get a chance to say yes to delivering via cloudfront which would've been nice.

Just wanted to share incase it can be fixed for future first time users.

First things first...this is the most useful thing I've found in a long time. Much thanks.

Gzipping is great, but it would be incredible to have my JS files combined prior to the gzip process.

Some companies will only use gems with a certain license.
The canonical and easy way to check is via the gemspec,

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Even for projects that already specify a license, including a license in your gemspec is a good practice, since it is easily
discoverable there without having to check the readme or for a license file.

For example, there is a License Finder gem to help companies ensure all gems they use
meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough
issue that even Bundler now generates gems with a default 'MIT' license.

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file),
github has created a license picker tool.

In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally
looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :).

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue and let me know. In either case, I'll follow up. Thanks!

p.s. I've written a blog post about this project

I'm using cactus (python static website generator) and the static files are not copied into the build folder, and are instead symlinked to the folder. As a result, s3_website push --site .build doesn't follow the symlinks and thus doesn't upload the static files.

Is there anyway to include symlinks? I'm not familiar with ruby.

Let the user define the level of concurrency.

In s3_website.yml:

concurrency_level: <integer>

Also decrease the default concurrency level to 25, since the current default 100 is too high.

Hi,

I realized that S3 Website invalidate all files to CloudFront, instead of the only the ones I changed (while there is a "changed_files" parameter in the CloudFront invalidator). This can be problematic because it invalidate every time I post a new article, and I may quite easily reached the free 1000 invalidations per month (the blog contains nearly 200 files).

Is there an option to enable this behavior ?

EDIT : my mistake, it seems to work in fact. I inaccurately thought that "/" was equal to invalidate all items… Sorry :D

Hi, this gem is really helpful and has really helped me cut down on time to deploy to s3.

I had an idea regarding adding an option for Route 53 in the config file that would auto configure your dns were you using Route 53 (which I suspect most people doing this are).

I could use the ruby_route_53 library for implementation.

Notes

• Disabled by default (commented out sections in s3_website.yml much like the gzip options)
• Option to create additional buckets that point to the s3_bucket specified in the config.
  • Ex: s3_bucket: www.example.com with an option that would create a bucket for example.com set to redirect
• Will run with s3_website apply cfg with error messages for:
  • Optional bucket already exists
  • DNS not on route53 (won't create a new hosted zone if not already on there)
  • Ask the user (with another [Y/n]) if they want to override an existing dns entry if one exists for either the optional buckets (those that redirect) and the main bucket.

Let me know what you think and I can get started on it. The only problem that I can see with it is that it does add another dependency.

I'm quite new to Ruby and AWS, so maybe I'm doing something wrong, but I've just tried this out and noticed that when you run s3_website push the s3_website.yml file is uploaded into your site's home directory.

So anyone who then goes to your URL, eg bucketname.s3-website-us-west-2.amazonaws.com/s3_website.yml can see these keys, which obviously is a security concern.

(PS, I should add that I'm using Jekyll to build my site. I've just checked out the older repo jekyll-s3, and the configuration file there is preceded by an underscore which means it is not included in the _site directory. So, perhaps s3_website needs to generate a config file with an underscore too?)

On certain files I want to disable caching by setting this header: "Cache-Control: no-cache, max-age=0". If I set max_age to 0, I only get "Cache-Control: max-age=0". Would be great if when max_age is 0, then no-cache is added as well.

Problem

When the s3_website cfg apply creates a CloudFront distribution, it marks the origin as an S3 origin. While it is true that in our case the origin is always an S3 bucket, this causes a problem that @bakura10 describes in #29 (comment).

Solution

Modify the s3_website cfg apply such that it creates the distribution with a generic origin.

Edit: we need to apply this fix into the configure-s3-website, since that gem is responsible for the implementation of the s3_website cfg apply command.

Benefit

New users benefit from this feature, because they will be able to invalidate CloudFront resources like /article/.

I.e., without this feature, they have to stick to URLs like http://mysite.com/article/index.html.

When I push my site, I get this error:

Calculating diff / invalid distance too far back (Zlib::DataError)

I'm running jekyll 1.4.3, ruby 1.9.3, s3_website 1.6.6.

Any thoughts on the issue here? Thanks

I believe the docs say gzip compression is only supported for .css, .js, .html, and .txt files. Will you support .json and other popular data formats as well? Is it primarily a matter of setting the Content-Type headers on S3?

I have some serialized data I'd like to use in a Jekyll blog post. The datasets are fairly large so compression would be nice. I realize I can work around this manually by uploading those files to S3, but if I could configure it in s3_website.yml, that'd be really great to have.

I noticed that s3_website (via filey-diff, in aws_sdk_s3.rb) makes a HEAD call for every file it is attempting to sync. (In fact does it do multiple HEAD request, I noticed several calls to s3_object.head).

On a slower connection, and with hundreds of files, these multiple HEAD requests can take a few minutes.

Is it possible to have an optional mode where s3_website uses the metadata from the earlier call to Get Bucket. The ETag in the metadata is a MD5 hash of the object, perhaps that could be used? This mode probably could not be used with gzip: mode, I don't think the content-encoding is in the metadata.

Hi,

When updating files in the S3 bucket, is it possible to add the cache control header to support optimised search engine crawling. The header in the HTTP response we need is Last-Modified

If I deploy my site and then change options, such as max_age or gzip extensions, I can't re-deploy the site with the new settings without getting "No new or changed files to upload." Maybe a s3_website push --force option would help.

Hey

I am setting s3_website up into my grunt task and for this I cannot give human input.
You do have a headless mode which works great for the push command but seems not to be supported for the s3_website cfg apply command.

For consistency and using this for automated task it would help to have a headless mode for all stages of the script.

I guess either default back to "N" for the cloudfront question or make a setting in the s3_website.yml for cloudfront: true or cloudfront: false for this option.

Thanks

It would be nice to override default site root in s3_website.yml so one does not have to invoke s3_website push --site public.

Alternatively, s3_website can fallback to public if _site does not exist. Since it's positioned to work with Octopress it looks weird that it does not find public by default... am I missing something?

Perhaps I'm doing it wrong, but I keep getting this error when I run s3_website cfg apply:

jed:~/Dropbox/Code/Sites/jedfoster.com [git:master+?]  
→ s3_website cfg apply
Applying the configurations in s3_website.yml on the AWS services ...
/Users/jed/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/configure-s3-website-1.5.0/lib/configure-s3-website/http_helper.rb:54:in `call_api': AWS API call failed: (GenericError)
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><RequestId>7B02DC9DEF00EFB1</RequestId><Bucket>assets.jedfoster.com</Bucket><HostId>ZU2boFGltJbUIQE0IGgJkFnpMaLyOmFh7bBJcznJjm9e6aGKHW29RrK6jgUlKQRi</HostId><Endpoint>assets.jedfoster.com.s3.amazonaws.com</Endpoint></Error>
    from /Users/jed/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/configure-s3-website-1.5.0/lib/configure-s3-website/http_helper.rb:7:in `call_s3_api'
    from /Users/jed/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/configure-s3-website-1.5.0/lib/configure-s3-website/s3_client.rb:34:in `enable_website_configuration'
    from /Users/jed/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/configure-s3-website-1.5.0/lib/configure-s3-website/s3_client.rb:12:in `configure_website'
    from /Users/jed/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/configure-s3-website-1.5.0/lib/configure-s3-website/runner.rb:4:in `run'
    from /Users/jed/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/configure-s3-website-1.5.0/bin/configure-s3-website:12:in `<top (required)>'
    from /Users/jed/.rbenv/versions/1.9.3-p194/bin/configure-s3-website:23:in `load'
    from /Users/jed/.rbenv/versions/1.9.3-p194/bin/configure-s3-website:23:in `<main>'

Here's my s3_website.yml file:

s3_id: XXX
s3_secret: XXX
s3_bucket: assets.jedfoster.com

# Below are examples of all the available configurations.
# See README for more detailed info on each of them.

# max_age:
#   "assets/*": 6000
#   "*": 300

gzip:
  - .html
  - .css
  - .md
  - .js

# s3_endpoint:  

# ignore_on_server: that_folder_of_stuff_i_dont_keep_locally

# s3_reduced_redundancy: true

# cloudfront_distribution_id: your-dist-id

# cloudfront_distribution_config:
#   default_cache_behavior:
#     min_TTL: <%= 60 * 60 * 24 %>
#   aliases:
#     quantity: 1
#     items:
#       CNAME: your.website.com

# cloudfront_invalidate_root: true

# concurrency_level: 100

redirects:
  index.php: /
#   about.php: about.html
#   music-files/promo.mp4: http://www.youtube.com/watch?v=dQw4w9WgXcQ

# routing_rules:
#   - condition:
#       key_prefix_equals: blog/some_path
#     redirect:
#       host_name: blog.example.com
#       replace_key_prefix_with: some_new_path/
#       http_redirect_code: 301

s3_website cfg apply "works" if I set s3_bucket to the endpoint listed on my AWS console (under "Static Website Hosting"), but the resulting bucket is assets.jedfoster.com.s3-website-us-west-2.amazonaws.com which I don't think is correct.

Is this a bug, or am I doing it wrong?

Hi,

I may be configuring my project badly, but here is the problem: I set up a max age of 86400 seconds (1 day) for all my HTML documents.

When I use s3_website push command, it automatically invalidates the item, but it invalidates, for instance: "/blog/some-key/article/index.html". However, most pages are instead accessed using the URL WITHOUT the index.html. It seems that CloudFront has two copies of the object, so that if I go to "/blog/some-key/article" I'm still being served with the old article, while if I had /article/index.html I am being served with the updated version.

Is there a way to force s3_jekyll to invalidate the URL WITHOUT the /index.html suffix ? Or does it come from a bad configuration of my S3+CloudFront ?

Thanks!

Cloudfront invalidation seems to fail with the following error:

/Users/sudonim/.rbenv/versions/1.9.3-p194-perf/lib/ruby/gems/1.9.1/gems/simple-cloudfront-invalidator-1.0.0/lib/simple-cloudfront-invalidator.rb:57:in `sign_and_call': AWS API call failed. Reason: (RuntimeError)`

The full response is in this Gist:
https://gist.github.com/sud0n1m/aceecb10f0e48cfd435a

It seems to suggest that I'm sending a malformed response. Is that so?

When I try to push my blog to s3, I get thousands and thousands of error messages in my rails console. Here's a small snippet:

Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6671.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6675.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6679.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6682.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6685.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6689.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6691.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6693.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6695.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6696.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6700.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6703.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6704.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6707.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/serve/images/2011/Canada/IMG_6715.jpg (Errno::EMFILE) Retrying in 3 seconds...
Upload serve/2011/mount-evans-and-the-highest-paved-road-in-north-america-part-1/index.html: Success!Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2195.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2196.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2198.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2201.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8321.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8313.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8349.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8314.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8353.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8360.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8374.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8390.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8399.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8402.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8407.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8415.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8412.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8455.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8451.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8453.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8460.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8476.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8517.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8582.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8522.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8529.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8586.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8534.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8556.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8594.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8569.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8579.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8595.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8584.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8588.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8612.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8599.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8342.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8616.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2204.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2354.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2363.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2362.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2367.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2369.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2374.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2383.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2396.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2411.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2206.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2210.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2205.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2219.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2224.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2208.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2235.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2245.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2213.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2222.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2259.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2276.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2244.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2252.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2291.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2272.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2289.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2292.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2296.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2303.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2307.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2298.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2293.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2311.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2338.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2301.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2347.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2305.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2349.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2309.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2312.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2326.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2324.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2345.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2348.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2350.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2297.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2387.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2401.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2468.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2512.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2429.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2493.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2529.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2549.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2567.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2558.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2578.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2583.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2588.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2603.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2618.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2590.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2694.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2710.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2750.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2760.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2747-2.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2824.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2850.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2829.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8794.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2861.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2862.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2872.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8846.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2879.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2887.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2902.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8881.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2942.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2956.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2959.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8897.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_3093.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8642.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8647.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8905.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8651.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8668.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8672.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8908.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8673.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8675.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8677.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8953.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8686.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8688.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8692.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8962.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8695.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8696.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8697.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8968.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8703.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8721.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8712.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8984.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8728.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8761.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8775.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8998.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8817.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8867.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8888.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9016.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_8987.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9004.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9022.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9065.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9074.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9084.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9077.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9089.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9108.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9087.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7335.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9093.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_2727.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9110.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7344.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7365.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7369.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_9114.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7351.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7345.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7353.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7354.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7366.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7371.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7374.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7390.jpg (Errno::EMFILE) Retrying in 3 seconds...Exception Occurred: Too many open files - /Users/Guillaume/Projects/Jekyll/_site/server/images/2011/US/IMG_7396.jpg (Errno::EMFILE) Retrying in 3 seconds...

I also get the occasional:

Exception Occurred: getaddrinfo: nodename nor servname provided, or not known (SocketError) Retrying in 3 seconds...

Then eventually some of my images are successfully uploaded (though I can't confirm all of them are) but I end up with the message "Operation failed even though we tried to recover from it"

When i try to redeploy my bucket it fails. For example, if i do a change to some file and if i need to redeploy the site with this change, s3_website fails. First I have to delete manually the files on that bucket, and only in that case s3_website works again. The specific error is:

Calculating diff //usr/lib/ruby/1.9.1/openssl/buffering.rb:174:in sysread_nonblock': end of file reached (EOFError) from /usr/lib/ruby/1.9.1/openssl/buffering.rb:174:inread_nonblock'
from /usr/lib/ruby/1.9.1/net/protocol.rb:141:in rbuf_fill' from /usr/lib/ruby/1.9.1/net/protocol.rb:122:inreaduntil'
from /usr/lib/ruby/1.9.1/net/protocol.rb:132:in readline' from /usr/lib/ruby/1.9.1/net/http.rb:2562:inread_status_line'
from /usr/lib/ruby/1.9.1/net/http.rb:2551:in read_new' from /usr/lib/ruby/1.9.1/net/http.rb:1319:inblock in transport_request'
from /usr/lib/ruby/1.9.1/net/http.rb:1316:in catch' from /usr/lib/ruby/1.9.1/net/http.rb:1316:intransport_request'
from /usr/lib/ruby/1.9.1/net/http.rb:1293:in request' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/net/http/connection_pool/session.rb:64:inrequest'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:173:in block in request' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:194:insession_for'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:171:in request' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/net/http/connection_pool/connection.rb:173:inrequest'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/http/net_http_handler.rb:66:in handle' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:231:inblock in make_sync_request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:267:in retry_server_errors' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:227:inmake_sync_request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:472:in block (2 levels) in client_request' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:355:inlog_client_request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:441:in block in client_request' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:337:inreturn_or_raise'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:440:in client_request' from (eval):3:inhead_object'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/s3/s3_object.rb:294:in head' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/s3/s3_object.rb:313:inlast_modified'
from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:77:in last_modified_and_md5' from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:62:inmap_s3_object_to_filey'
from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:17:in block in do_internal_load' from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:26:incall'
from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:26:in block (2 levels) in in_parallel_or_sequentially' from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:35:incall'
from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:35:in block (3 levels) in in_parallel_or_sequentially' felipe@excalibur:~/aeRepositories/aeHome$ s3_website push --site dist Deploying dist/* to aehome-staging Calculating diff |/usr/lib/ruby/1.9.1/openssl/buffering.rb:174:insysread_nonblock': end of file reached (EOFError)
from /usr/lib/ruby/1.9.1/openssl/buffering.rb:174:in read_nonblock' from /usr/lib/ruby/1.9.1/net/protocol.rb:141:inrbuf_fill'
from /usr/lib/ruby/1.9.1/net/protocol.rb:122:in readuntil' from /usr/lib/ruby/1.9.1/net/protocol.rb:132:inreadline'
from /usr/lib/ruby/1.9.1/net/http.rb:2562:in read_status_line' from /usr/lib/ruby/1.9.1/net/http.rb:2551:inread_new'
from /usr/lib/ruby/1.9.1/net/http.rb:1319:in block in transport_request' from /usr/lib/ruby/1.9.1/net/http.rb:1316:incatch'
from /usr/lib/ruby/1.9.1/net/http.rb:1316:in transport_request' from /usr/lib/ruby/1.9.1/net/http.rb:1293:inrequest'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/net/http/connection_pool/session.rb:64:in request' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:173:inblock in request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:194:in session_for' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:171:inrequest'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/net/http/connection_pool/connection.rb:173:in request' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/http/net_http_handler.rb:66:inhandle'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:231:in block in make_sync_request' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:267:inretry_server_errors'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:227:in make_sync_request' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:472:inblock (2 levels) in client_request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:355:in log_client_request' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:441:inblock in client_request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:337:in return_or_raise' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:440:inclient_request'
from (eval):3:in head_object' from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/s3/s3_object.rb:294:inhead'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/s3/s3_object.rb:313:in last_modified' from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:77:inlast_modified_and_md5'
from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:62:in map_s3_object_to_filey' from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:17:inblock in do_internal_load'
from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:26:in call' from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:26:inblock (2 levels) in in_parallel_or_sequentially'
from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:35:in call' from /usr/lib/ruby/gems/1.9.1/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:35:inblock (3 levels) in in_parallel_or_sequentially

Hi,

I use the draft feature in Jekyll, however, the push is also pushing draft content, which is not required. Is it possible to add folder filtering?

e.g.
ignore_folders:
"rafts"

The documentation for IAM users states "create API credentials that have sufficient permissions." The documentation does not list what specific permissions are required on a bucket to use the gem.

How may I change the GZIP method or options?
For instance, specifying a different compression ratio, or using Google's Zopfli library?

Is it possible to push the output to multiple buckets?

I have about 1200 posts on my jekyll site. Compiling it takes about 22 seconds on my Macbook Air (for reference), but the "calculating diff" portion of s3_website push takes over a minute. Is this normal?

Thanks - loving the gem! Just deployed my site to CF with it.

Sorry for all the questions but just tried the gzip option and it doesn't' seem to do anything.. do I have to gzip the files myself? I know that's probably the answer just wanted to make sure.

At the moment s3_website push does two GET Bucket operations:

1. in the filey-diff gem, when calculating diff
2. in uploader.rb when determining files-to-remove-from-s3

One of two requests is unnecessary. Remove either of them.

At the moment s3_website invokes the AWS SDK gem's upload API in parallel, within Thread.new. Obviously, this is considerably faster than performing the uploads sequentially.

I assume that most s3_website users are using an RMI Ruby. As far as I know, the thread schedulers in RMI Rubies are not optimally fair to our use case of making parallel HTTP connections. As a result, I suggest that we try replacing the usage AWS SDK with http://rubyeventmachine.com. It should be able to speed up the HTTP operations of s3_website.

There is, however, one significant hindrance to adopting EventMachine: the Cucumber+VCR integration tests. I don't yet know how to transform them into the evented world. Does anyone have ideas on how to solve this problem? More specifically: how to test evented HTTP requests against fixture data?

Hi,

I'm a new user. I'm having trouble getting s3_website to work. Here is the traceback:

Carry-Tiger:Website andrewlahser$ s3_website push
Deploying _site/* to howconceptual.com
Calculating diff /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:339:in return_or_raise': No Such Key (AWS::S3::Errors::NoSuchKey) from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:440:inclient_request'
from (eval):3:in head_object' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/aws-sdk-1.8.5/lib/aws/s3/s3_object.rb:294:inhead'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:57:in map_s3_object_to_filey' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:17:inblock in do_internal_load'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:26:in call' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:26:inblock (2 levels) in in_parallel_or_sequentially'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:35:in call' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:35:inblock (3 levels) in in_parallel_or_sequentially'

So, I also tried disabling parallel processing.

Carry-Tiger:Website andrewlahser$ disable_parallel_processing=true s3_website push
Deploying _site/* to howconceptual.com
Calculating diff /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:339:in return_or_raise': No Such Key (AWS::S3::Errors::NoSuchKey) from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:440:inclient_request'
from (eval):3:in head_object' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/aws-sdk-1.8.5/lib/aws/s3/s3_object.rb:294:inhead'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:57:in map_s3_object_to_filey' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:17:inblock in do_internal_load'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:26:in call' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:26:inblock (2 levels) in in_parallel_or_sequentially'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:30:in call' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:30:ineach'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:30:in in_parallel_or_sequentially' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:19:indo_internal_load'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/data-sources/data_source.rb:8:in get_fileys' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/comparison.rb:28:inselect_in_outer_array'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/filey-diff-1.4.0/lib/filey-diff/comparison.rb:11:in list_changed' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/s3_website-1.4.0/lib/s3_website/diff_helper.rb:11:inblock in resolve_files_to_upload'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/s3_website-1.4.0/lib/s3_website/diff_helper.rb:34:in with_progress_indicator' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/s3_website-1.4.0/lib/s3_website/diff_helper.rb:4:inresolve_files_to_upload'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/s3_website-1.4.0/lib/s3_website/uploader.rb:49:in upload_files' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/s3_website-1.4.0/lib/s3_website/uploader.rb:15:inrun'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/s3_website-1.4.0/lib/s3_website/tasks.rb:8:in push' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/s3_website-1.4.0/bin/s3_website:52:inpush'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/command.rb:27:in run' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/invocation.rb:120:ininvoke_command'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor.rb:363:in dispatch' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/base.rb:439:instart'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/gems/s3_website-1.4.0/bin/s3_website:59:in <top (required)>' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/bin/s3_website:23:inload'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/bin/s3_website:23:in <main>' from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:ineval'
from /Users/andrewlahser/.rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:in `

I apologize if this is a silly question. I'm not sure what to do next. I have a medium sized jekyll site (about 500 pages).

It would be cool if s3_website supported the dotenv gem. Then I could define environment variables in a .env file that I don't check into version control while s3_website.yml IS checked in.

This would allow me to deploy from both my local machine as well as Travis CI (via GitHub hooks).

If one tries to s3_website push while not connected to the internet, the program will just crash and burn without providing any proper error message.

Deploying _site/*
Calculating diff //System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http.rb:878:in `initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError)
    from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http.rb:878:in `open'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http.rb:878:in `block in connect'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/timeout.rb:66:in `timeout'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http.rb:877:in `connect'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http.rb:862:in `do_start'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http.rb:857:in `start'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/net/http/connection_pool/session.rb:118:in `start'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:208:in `_create_session'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:193:in `session_for'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/net/http/connection_pool.rb:171:in `request'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/net/http/connection_pool/connection.rb:173:in `request'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/http/net_http_handler.rb:66:in `handle'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:231:in `block in make_sync_request'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:267:in `retry_server_errors'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:227:in `make_sync_request'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:472:in `block (2 levels) in client_request'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:355:in `log_client_request'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:441:in `block in client_request'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:337:in `return_or_raise'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:440:in `client_request'
    from (eval):3:in `list_objects'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/s3/object_collection.rb:297:in `list_request'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/s3/paginated_collection.rb:29:in `_each_item'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/collection/with_limit_and_next_token.rb:54:in `_each_batch'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/collection.rb:82:in `each_batch'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/core/collection.rb:49:in `each'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-1.8.5/lib/aws/s3/object_collection.rb:282:in `each'
    from /Library/Ruby/Gems/2.0.0/gems/filey-diff-1.4.2/lib/filey-diff/data-sources/aws_sdk_s3.rb:24:in `map'
    from /Library/Ruby/Gems/2.0.0/gems/filey-diff-1.4.2/lib/filey-diff/data-sources/aws_sdk_s3.rb:24:in `in_parallel_or_sequentially'
    from /Library/Ruby/Gems/2.0.0/gems/filey-diff-1.4.2/lib/filey-diff/data-sources/aws_sdk_s3.rb:19:in `do_internal_load'
    from /Library/Ruby/Gems/2.0.0/gems/filey-diff-1.4.2/lib/filey-diff/data-sources/data_source.rb:8:in `get_fileys'
    from /Library/Ruby/Gems/2.0.0/gems/filey-diff-1.4.2/lib/filey-diff/comparison.rb:28:in `select_in_outer_array'
    from /Library/Ruby/Gems/2.0.0/gems/filey-diff-1.4.2/lib/filey-diff/comparison.rb:11:in `list_changed'
    from /Library/Ruby/Gems/2.0.0/gems/s3_website-1.6.1/lib/s3_website/diff_helper.rb:11:in `block in resolve_files_to_upload'
    from /Library/Ruby/Gems/2.0.0/gems/s3_website-1.6.1/lib/s3_website/diff_helper.rb:34:in `with_progress_indicator'
    from /Library/Ruby/Gems/2.0.0/gems/s3_website-1.6.1/lib/s3_website/diff_helper.rb:4:in `resolve_files_to_upload'
    from /Library/Ruby/Gems/2.0.0/gems/s3_website-1.6.1/lib/s3_website/uploader.rb:49:in `upload_files'
    from /Library/Ruby/Gems/2.0.0/gems/s3_website-1.6.1/lib/s3_website/uploader.rb:14:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/s3_website-1.6.1/lib/s3_website/tasks.rb:8:in `push'
    from /Library/Ruby/Gems/2.0.0/gems/s3_website-1.6.1/bin/s3_website:52:in `push'
    from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
    from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
    from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
    from /Library/Ruby/Gems/2.0.0/gems/s3_website-1.6.1/bin/s3_website:59:in `<top (required)>'
    from /usr/bin/s3_website:23:in `load'
    from /usr/bin/s3_website:23:in `<main>'

/Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/s3_website-1.3.2/lib/s3_website/uploader.rb:151:in initialize': no implicit conversion of Array into String (TypeError) from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/s3_website-1.3.2/lib/s3_website/uploader.rb:151:innew'
from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/s3_website-1.3.2/lib/s3_website/uploader.rb:151:in build_list_of_files_to_delete' from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/s3_website-1.3.2/lib/s3_website/uploader.rb:133:inremove_superfluous_files'
from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/s3_website-1.3.2/lib/s3_website/uploader.rb:21:in run' from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/s3_website-1.3.2/lib/s3_website/tasks.rb:8:inpush'
from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/s3_website-1.3.2/bin/s3_website:52:in push' from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/thor-0.18.1/lib/thor/command.rb:27:inrun'
from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/thor-0.18.1/lib/thor/invocation.rb:120:in invoke_command' from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/thor-0.18.1/lib/thor.rb:363:indispatch'
from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/thor-0.18.1/lib/thor/base.rb:439:in start' from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/gems/s3_website-1.3.2/bin/s3_website:59:in<top (required)>'
from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/bin/s3_website:23:in load' from /Users/ulises/.rvm/gems/ruby-2.0.0-p0/bin/s3_website:23:in

I'm getting some very odd behaviour with my push command... It seems that if I have any sub-folders named the same as my --site directory (in this case "dist"), it uploads the files, then precedes to tell me that that don't exist and would I like to delete them!?

siyfion@virtualMint ~/Git/LabelLogicLive_Client $ s3_website push --site dist
Deploying dist/* to labellogiclive.com
Calculating diff ... done
Uploading 11 new file(s)
Upload bower_components/angularytics/dist/angularytics.zip [max-age=3600]: Success!
Upload bower_components/angularytics/dist/angularytics.min.js [gzipped] [max-age=3600]: Success!
Upload bower_components/lodash/dist/lodash.underscore.min.js [gzipped] [max-age=3600]: Success!
Upload bower_components/lodash/dist/lodash.min.js [gzipped] [max-age=3600]: Success!
Upload bower_components/angularytics/dist/angularytics.js [gzipped] [max-age=3600]: Success!
Upload bower_components/lodash/dist/lodash.compat.min.js [gzipped] [max-age=3600]: Success!
Upload bower_components/lodash/dist/lodash.js [gzipped] [max-age=3600]: Success!
Upload bower_components/lodash/dist/lodash.compat.js [gzipped] [max-age=3600]: Success!
Upload bower_components/angularytics/dist/dependencies/lodash.js [gzipped] [max-age=3600]: Success!
Upload bower_components/lodash/dist/lodash.underscore.js [gzipped] [max-age=3600]: Success!
Upload bower_components/angularytics/dist/dependencies/angular.js [gzipped] [max-age=3600]: Success!
bower_components/angularytics/dist/angularytics.js is on S3 but not in your website directory anymore. Do you want to [d]elete, [D]elete all, [k]eep, [K]eep all?
K
Done! Go visit: http://labellogiclive.com.s3-website-eu-west-1.amazonaws.com/index.html
Invalidating Cloudfront items...
  /
succeeded

Currently s3_website prints the follow output:

Deploying _site/* to site.com
No new or changed files to upload
Done! Go visit: http://site.com.s3-website-us-east-1.amazonaws.com/index.html

Change it to this:

Deploying _site/* to site.com
Calculating diff... done
No new or changed files to upload
Done! Go visit: http://site.com.s3-website-us-east-1.amazonaws.com/index.html

The "calculating diff" state might take several seconds or even minutes. As a consequence, it's important to indicate to the user what's going on.

On ruby-2.1.0, running push outputs "Digest::Digest is deprecated; use Digest" dozens of times.

This is coming from the amazon aws sdk. It's fixed in master, but I don't think it has been released yet. I just wanted to make you aware of this because I don't think the way the current gemspec is configured will catch that update (they're currently on something like 1.31.x, this project is speced to 1.8.x).

It seems that "invalidations" exceed API limit during calculating diff. It says, "Processing your request will cause you to exceed the maximum number of in-progress invalidations".

This error occurs when I run s3_website push several times in several minutes. Before this error occured, the same operations (s3_website push) succeed. Waiting for tens of minutes, it recovers so that I can run s3_website push and get the expected result.

Is this a bug or my wrong configuration?

$ s3_website push
Deploying _site/* to ishibashihideto.net
Calculating diff ... done
No new or changed files to upload
Done! Go visit: http://ishibashihideto.net.s3-website-ap-northeast-1.amazonaws.com/index.html
/Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/gems/simple-cloudfront-invalidator-1.0.0/lib/simple-cloudfront-invalidator.rb:57:in `sign_and_call': AWS API call failed. Reason: (RuntimeError)
<?xml version="1.0"?>
<ErrorResponse xmlns="http://cloudfront.amazonaws.com/doc/2012-05-05/"><Error><Type>Sender</Type><Code>TooManyInvalidationsInProgress</Code><Message>Processing your request will cause you to exceed the maximum number of in-progress invalidations.</Message></Error><RequestId>********-****-****-****-************</RequestId></ErrorResponse>
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/gems/simple-cloudfront-invalidator-1.0.0/lib/simple-cloudfront-invalidator.rb:30:in `invalidate'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/gems/s3_website-1.3.2/lib/cloudfront/invalidator.rb:14:in `invalidate'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/gems/s3_website-1.3.2/lib/s3_website/tasks.rb:36:in `invalidate_cf_dist_if_configured'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/gems/s3_website-1.3.2/lib/s3_website/tasks.rb:10:in `push'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/gems/s3_website-1.3.2/bin/s3_website:52:in `push'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/gems/s3_website-1.3.2/bin/s3_website:59:in `<top (required)>'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/bin/s3_website:23:in `load'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/bin/s3_website:23:in `<main>'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/bin/ruby_noexec_wrapper:14:in `eval'
    from /Users/ishibashi/.rvm/gems/ruby-1.9.3-p392/bin/ruby_noexec_wrapper:14:in `<main>'

Hi, I'm getting an "Access Denied" error when running 'cfg apply.'

I've already deployed my site successfully and have issued numerous updates using s3_website, and have no problems with the 'push' command--it's specifically when trying to apply my changed configuration (with gzip enabled) that I'm seeing the error.

Is there a specific permission that is required for the cfg apply operation, that is different from those used by 'push'?

I currently have the ListAllMyBuckets, ListBucket, CreateBucket, PutObject, GetObject, and DeleteObject permissions set for my AWS IAM user.

Some simple debugging shows that a PUT request is being generated, and I can see that the local config file is being properly read and used to help generate the request with the appropriate id, secret, and bucket.

Any thoughts? In summary, 'push' works but 'cfg apply' doesn't.

Background: I want to enable gzip. If I change the config file, I need to run 'cfg apply.' Is that correct?

Just pushing the site with the config change hasn't seemed to cause the Content-Encoding header to be added to responses, so I assume I have to apply the change with 'cfg apply.'

This makes it easier to use s3_website with travis, because you'll get notified if it didn't succeed.

Hi there,

I am sure this is something easy that I just can't see right now but I seem to be unable to set up the correct API credentials.

What I got so far:
I added a user in IAM with the group policy and user policy as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}

And still get AllAccessDisabled when I do s3_website cfg apply.

The very same credentials work fine when using an FTP client like Transmit. Are there any other settings I'm not aware of that have to be set for this to work?

Any help is appreciated.