rubygems / bundler-api

Bundler 1.1+ API

RubyGems

RubyGems is a package management framework for Ruby.

A package (also known as a library) contains a set of functionality that can be invoked by a Ruby program, such as reading and parsing an XML file. We call these packages "gems" and RubyGems is a tool to install, create, manage and load these packages in your Ruby environment.

RubyGems is also a client for RubyGems.org, a public repository of Gems that allows you to publish a Gem that can be shared and used by other developers. See our guide on publishing a Gem at guides.rubygems.org

Getting Started

Installing and managing a Gem is done through the gem command. To install a Gem such as Nokogiri which lets you read and parse XML in Ruby:

$ gem install nokogiri

RubyGems will download the Nokogiri Gem from RubyGems.org and install it into your Ruby environment.

Finally, inside your Ruby program, load the Nokogiri gem and start parsing your XML:

require 'nokogiri'

Nokogiri.XML('<h1>Hello World</h1>')

For more information about how to use RubyGems, see our RubyGems basics guide at guides.rubygems.org

Requirements

• RubyGems supports Ruby 3.0 or later.

Installation

RubyGems is already installed in your Ruby environment, you can check the version you have installed by running gem --version in your terminal emulator.

In some cases Ruby & RubyGems may be provided as OS packages. This is not a recommended way to use Ruby & RubyGems. It's better to use a Ruby Version Manager, such as rbenv or chruby. If you still want to use the version provided by your OS package manager, please also use your OS package manager to upgrade rubygems, and disregard any other installation instructions given below.

If you would like to manually install RubyGems:

• Download from https://rubygems.org/pages/download, unpack, and cd into RubyGems' src
• OR clone this repository and cd into the repository

Install RubyGems by running:

$ ruby setup.rb

For more details and other options, see:

$ ruby setup.rb --help

Upgrading RubyGems

To upgrade to the latest RubyGems, run:

$ gem update --system

See UPGRADING for more details and alternative instructions.

Release policy

RubyGems and Bundler are released in sync, although they do not share their major version number. It is planned that also their major version numbers will be sync'ed in the future.

The release policy is somewhat similar to the release policy of Ruby itself:

• Frequent patch releases (every 2-4 weeks) including bug fixes, minor enhancements, small features, or even medium sized features declared as experimental for battle testing.
• Yearly minor releases including bigger features, and minor breaking changes (affecting only edge cases and a very small set of users).
• Occasional major releases (replacing yearly minors) including major breaking changes.

Documentation

RubyGems uses rdoc for documentation. A compiled set of the docs can be viewed online at rubydoc.

RubyGems also provides a comprehensive set of guides which covers numerous topics such as creating a new gem, security practices and other resources at https://guides.rubygems.org

Getting Help

Filing Tickets

Got a bug and you're not sure? You're sure you have a bug, but don't know what to do next? In any case, let us know about it! The best place for letting the RubyGems team know about bugs or problems you're having is on the RubyGems issues page at GitHub.

Bundler Compatibility

See https://bundler.io/compatibility for known issues.

Supporting

RubyGems is managed by Ruby Central, a non-profit organization that supports the Ruby community through projects like this one, as well as RubyConf, RailsConf, and RubyGems.org. You can support Ruby Central by attending or sponsoring a conference, or by joining as a supporting member.

Contributing

If you'd like to contribute to RubyGems, that's awesome, and we <3 you. Check out our guide to contributing for more information.

Code of Conduct

Everyone interacting in the RubyGems project’s codebases, issue trackers, chat rooms, and mailing lists is expected to follow the contributor code of conduct.

bundler-api's Issues

Add gem webhook

There's either a bug in the webhook sent from rubygems.org, the code which parses the webhook in bundler-api, or a bad gemspec that's causing the following error:

Jul 14 08:28:29 webhook request: {"name"=>"kafka-jars", "version"=>"0.8.0.pre1", "platform"=>{"cpu"=>nil, "os"=>"java", "version"=>nil}, "prerelease"=>true, "rubygems_token"=>"..."} 
Jul 14 08:28:29 Sequel::DatabaseError - PG::Error: ERROR:  operator does not exist: character varying = boolean 
Jul 14 08:28:29 LINE 6:       AND versions.platform = (('cpu' IS NULL) AND ('os' = '... 
Jul 14 08:28:29                                     ^ 
Jul 14 08:28:29 HINT:  No operator matches the given name and argument type(s). You might need to add explicit type casts.

The bug is the value of platform. For comparison, here are a few successful webhooks.

Jul 14 06:12:21 webhook request: {"name"=>"insulate", "version"=>"0.0.1", "platform"=>"ruby", "prerelease"=>false, "rubygems_token"=>"..."} 
Jul 14 06:12:21 Adding: insulate-0.0.1 

Jul 14 06:37:09 webhook request: {"name"=>"ruby-srp", "version"=>"0.2.0", "platform"=>"ruby", "prerelease"=>false, "rubygems_token"=>"..."} 
Jul 14 06:37:10 Adding: ruby-srp-0.2.0 

Jul 14 08:15:07 webhook request: {"name"=>"spud_banners", "version"=>"0.9.2", "platform"=>"ruby", "prerelease"=>false, "rubygems_token"=>"..."} 
Jul 14 08:15:07 Adding: spud_banners-0.9.2 

Jul 14 08:19:38 webhook request: {"name"=>"driftrock-service", "version"=>"0.4.17", "platform"=>"ruby", "prerelease"=>false, "rubygems_token"=>"..."} 
Jul 14 08:19:38 Adding: driftrock-service-0.4.17

Thoughts, @indirect?

Rescue Sinatra::NotFound

Pretty sure we don't actually need to log an exception whenever we send a 404. :)

View full backtrace and more info at honeybadger.io

Download stats missing for dates before 2012-08-20 in the API

Hi!

I am using the API to get download statistics for a gem. It was working before, but now I only get download numbers from 2012-08-20 and later. Earlier dates always give a zero download count.

Here is an example:

curl "https://rubygems.org/api/v1/versions/rails-3.2.0/downloads/search.json?from=2012-08-01&to=2012-08-31"

{"2012-08-01":0,"2012-08-02":0,"2012-08-03":0,"2012-08-04":0,"2012-08-05":0,"2012-08-06":0,"2012-08-07":0,"2012-08-08":0,"2012-08-09":0,"2012-08-10":0,"2012-08-11":0,"2012-08-12":0,"2012-08-13":0,"2012-08-14":0,"2012-08-15":0,"2012-08-16":0,"2012-08-17":0,"2012-08-18":0,"2012-08-19":0,"2012-08-20":175,"2012-08-21":178,"2012-08-22":280,"2012-08-23":183,"2012-08-24":153,"2012-08-25":56,"2012-08-26":57,"2012-08-27":169,"2012-08-28":169,"2012-08-29":141,"2012-08-30":198,"2012-08-31":150}

Am I using the API wrongly or are the download stats not available?

java-platform gems not appearing in the dependencies API

I maintain the websocket-driver gem. If you check that page, you'll see there are java versions of every release of the library.

In the past, it has often taken Bundler up to an hour to be able to find the java version. With the latest release, I've been waiting 3 days and Bundler still cannot find it, so all my downstream projects are broken on JRuby. Turns out the gem is not listed in this API response:

http://rubygems.org/api/v1/dependencies.json?gems=websocket-driver

Please let me know when this is fixed since it's blocking me working on libraries downstream of this project.

Getting an Array where we expect a Gem::Dependency

$ bundle exec rake 'update[5]' --trace
rake aborted!
undefined method `requirement' for ["ruby-ajp", ">= 0.2.0"]:Array
/Users/dan/Projects/ruby/bundler-api/lib/bundler_api/update/job.rb:110:in `block (2 levels) in insert_spec'
/Users/dan/Projects/ruby/bundler-api/lib/bundler_api/update/job.rb:109:in `each'
/Users/dan/Projects/ruby/bundler-api/lib/bundler_api/update/job.rb:109:in `block in insert_spec'

dep here is an Array instead of a Dependency object for some reason. Haven't had a chance to dig into this yet.

Enable configuration for an enterprise private gem server

We have this local gem server in our enterprise for private company projects. Until very recently we were using geminabox, which kinda-sorta implements bundler API, and we lived with the performance issues and lack of AAA features.

We have just started migrating most of the service supply to the web server (Apache) itself, in that the gems are served directly from it, and the push/ endpoint is now served using a ruby cgi script, which will perform some ldap/authorization validations before uploading the gem and reindexing. With that, we have effectively migrated the Rubygems API.

However, the bundler API support is gone, which is no significant problem with bundler 1.x, as it falls back to Rubygems API. It would be nice to have it though. I was wondering whether the endpoint could be also easily cgi-scripted, in that we could just mirror what we did with the rubygems endpoints, and the bundler-api gem would just provide the necessary public APi around it. It seems that this is just proxying to rubygems, but is there a way to configure it with an internal endpoint?

Thx anyway for any sort of reply. I understand this is not an issue and will not be offend if it's project policy to just close it and redirect me to a proper debate forum like a google group or anything similar.

Check for cache compliance with the caching RFC

The relevant cache line: https://github.com/bundler/bundler-api/blob/2cbbd3df69bce6cdec3c1cf9ad8a6bfe9b4a8cd5/lib/bundler_api/web.rb#L211

RFC: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

Action items from outage on 3/22

After the outage on 3/22, we came up with some action items. Please add the PR number to the todo when you check it off.

• Add ENV switch for new index endpoints (#142)
• Set up pgbouncer on each dyno and increase total dynos (#143)
• Use memcached for all new index responses (#149)
• Add back purging new index on Fastly, but update the memcached value beforehand (#152)
• Review all of the configuration changes and commits made to bundler-api on 3/22 (#?)

Gain better app visibility w/ New Relic

Sinatra + Sequel = Pain?

Graph Fastly response in Librato

Drain Fastly logs to Librato's l2met service to create throughput and response (success, error, etc.) graphs.

Retry PG timeouts rather than crashing update task

If a postgres query times out, the entire update task crashes:

Oct 24 08:01:01 bundler-api app/scheduler.4810:  ** Invoke update (first_time) 
Oct 24 08:01:01 bundler-api app/scheduler.4810:  ** Execute update 
Oct 24 08:01:02 bundler-api app/scheduler.4810:  # of specs from indexes: 629167 
Oct 24 08:01:08 bundler-api app/scheduler.4810:  # of non yanked local gem versions: 629166 
Oct 24 08:01:16 bundler-api app/scheduler.4810:  Finished Enqueuing Jobs! 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  Sequel::DatabaseConnectionError: PG::ConnectionBad: timeout expired 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/adapters/postgres.rb:236:in `initialize' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/adapters/postgres.rb:236:in `new' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/adapters/postgres.rb:236:in `connect' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/connection_pool.rb:101:in `make_new' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/connection_pool/threaded.rb:160:in `make_new' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/connection_pool/threaded.rb:137:in `available' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/connection_pool/threaded.rb:127:in `block in acquire' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/connection_pool/threaded.rb:201:in `block in sync' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/connection_pool/threaded.rb:201:in `synchronize' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/connection_pool/threaded.rb:201:in `sync' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/connection_pool/threaded.rb:126:in `acquire' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/connection_pool/threaded.rb:88:in `hold' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/database/connecting.rb:250:in `synchronize' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/adapters/postgres.rb:866:in `literal_string_append' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/sql.rb:78:in `literal_append' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/sql.rb:671:in `block in placeholder_literal_string_sql_append' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/sql.rb:668:in `loop' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/sql.rb:668:in `placeholder_literal_string_sql_append' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/sql.rb:107:in `to_s_append' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/sql.rb:1192:in `literal_expression_append' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/sql.rb:85:in `literal_append' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/sql.rb:343:in `literal' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/sql.rb:1508:in `static_sql' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/sql.rb:222:in `select_sql' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/actions.rb:139:in `each' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel_pg-1.6.11/lib/sequel_pg/sequel_pg.rb:58:in `each' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/actions.rb:640:in `single_record' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/vendor/bundle/ruby/2.2.0/gems/sequel-4.18.0/lib/sequel/dataset/actions.rb:192:in `first' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/lib/bundler_api/update/gem_db_helper.rb:30:in `exists?' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/lib/bundler_api/update/job.rb:17:in `run' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/lib/bundler_api/update/consumer_pool.rb:36:in `block (2 levels) in create_thread' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/lib/bundler_api/update/consumer_pool.rb:32:in `loop' 
Oct 24 08:01:37 bundler-api app/scheduler.4810:  /app/lib/bundler_api/update/consumer_pool.rb:32:in `block in create_thread' 

It would be better to catch the timeout exception and retry at least 2-3x before crashing.

Need schema/migration info

In general the repo doesn't have any info on getting this running locally or otherwise. More specifically a big thing that's missing is codified DB schema/migration info. I have the start of something locally, will work on packaging it up for a PR.

Periodically synchronize gems

We need a way to verify that the gems in bundler's database match those on rubygems.org and to pull down any changes.

`bundle update` update `haml` to a beta version

I am using haml-rails, which has dependency on haml >= 3.1, < 5.0, which has been locked at 4.0.5 in Gemfile.lock
After I run bundle update, it always bump it to 4.1.0.beta.1
Is it normal? Or the version format is parse incorrectly?

Note: If I run bundle update several times after bumped to 4.1.0.beta.1, it will sometimes go back to 4.0.5

Generating the full dependency graph

Hey,

I've been looking at generating the full dependency graph to be stored locally. I initially thought it would be too big, but I got it down to 2.3MB. See details here: http://www.tail-f.org/2013-07-22#post-6.

What do you think about this approach? How would we integrate it in Bundler API? This is more a cronjob, not an API, and thus should probably be stored in S3.

Increase Sequel pool size now that we have more threads

Seeing a lot of Sequel::PoolTimeout errors, which (I think) means that threads are waiting to serve responses but aren't able to because they can't get a database connection out of the pool. Probably we just need to give each Puma process a bigger pool of DB connections now that they have so many threads between them.

View full backtrace and more info at honeybadger.io

Set up alerts & New Relic

Not sure why we don't have this on already. Dealing with downtime right now and I heard of it offhand, no alerts from Pingdom or NewRelic.

Produce new index format

The API needs to produce the new index files using its database.

[Wishlist] Differentiate type of dependency

Right now, the first entry on http://bundler.rubygems.org/api/v1/dependencies.json?gems=rexical lists hoe as a dependency. But, we can't know if that is a Development Dependency (which it is) or a Runtime Dependency. It would be great to have that information too.

PS : We are planning to use this API for our Debian Packaging progressbar to get dependency information and we intend to prioritize Runtime dependencies over development dependencies. And, getting this information from the API would be a great help

ArgumentError

Message

  comparison of String with 400 failed

Backtrace (last 10 lines)

/app/lib/bundler_api/cache.rb:34:in `>='
/app/lib/bundler_api/cache.rb:34:in `block in verify_responses!'
/app/lib/bundler_api/cache.rb:34:in `select'
/app/lib/bundler_api/cache.rb:34:in `verify_responses!'
/app/lib/bundler_api/cache.rb:68:in `purge_gem'
/app/lib/bundler_api/web.rb:127:in `block (3 levels) in <class:Web>'
/app/lib/bundler_api/web.rb:277:in `block in in_background'

View on AppSignal: https://appsignal.com/bundler/sites/54dcc62a776f726d1b61e503/web/exceptions/ArgumentError

only adds first platform type for a gem in sync code

API isn't inserting all of the platforms during the sync/update code. See rubygems/bundler#2158

Create database setup script

There needs to be an automated process for setting up the database for testing and local development.

Exception when gems is an array

Error detected in: Bundler API ()

Full report at https://www.honeybadger.io/inspector/p/349/4426817

NoMethodError: undefined method `split' for ["rails"]:Array

Summary

http://bundler.rubygems.org/api/v1/dependencies?gems[]=rails
Trace

/app/lib/bundler_api/web.rb:48:in gems
/app/lib/bundler_api/web.rb:53:in get_deps
/app/lib/bundler_api/web.rb:98:in block in <class:Web>

Place nicely with foreman

We're using dotenv to manage the environment loaded from .env and .env.local. dotenv will ignore setting an environment variable if it exists. foreman will load .env itself when starting a process or running a command. Any environment overrides in .env.local will have no effect when the app boots and dotenv loads it because all the values it overrides will already be defined.

For example, set MAX_THREADS in both .env and .env.local. The value in .env will be used when starting the site using foreman start web whereas the value from .env.local will be used when using script/web.

One solution is to submit a patch to dotenv that changes the behavior to override instead of ignore existing environment variables.

Set up Airbrake or Honeybadger for exception tracking and emails

Doing a full initial sync will miss some dependencies

Due to this condition an initial sync will miss dependencies on gems that have not been synced yet.

For example, version 3.0.1 of the aasm gem depends on activerecord. Since the sync is done in seemingly alphabetical order, when version 3.0.1 of aasm is synced its dependency on activerecord will not be recorded since that gem has not been synced yet.

Guessing bundler-api as deployed to bundler-api.herokuapp.com was seeded with data extracted from rubygems.org so it skipped this step.

Librato is rejecting data with values of `nil`

As long as reporting data to Librato fails with this error, our dashboard is going to be very inaccurate. We should remove keys that have nil values, or set those values to zero.

[Metriks][Librato] Metriks::LibratoMetricsReporter::RequestFailedError raised during metric submission: 400 "Bad Request" 
Aug 22 17:35:00 bundler-api app/web.3:    Response body: {"errors":{"params":{"value":["is not a valid number"]}},"request_time":1471912500} 
Aug 22 17:35:00 bundler-api app/web.3:    Submitted data: {"gauges[0][name]"=>"thread_pool.backlog.median", "gauges[0][source]"=>"web.3", "gauges[0][measure_time]"=>1471912500, "gauges[0][value]"=>nil, "gauges[4][name]"=>"thread_pool.backlog.95th_percentile", "gauges[4][source]"=>"web.3", "gauges[4][measure_time]"=>1471912500, "gauges[4][value]"=>nil, "gauges[8][name]"=>"thread_pool.backlog.999th_percentile", "gauges[8][source]"=>"web.3", "gauges[8][measure_time]"=>1471912500, "gauges[8][value]"=>nil, "gauges[12][name]"=>"thread_pool.running.median", "gauges[12][source]"=>"web.3", "gauges[12][measure_time]"=>1471912500, "gauges[12][value]"=>0.0, "gauges[16][name]"=>"thread_pool.running.95th_percentile", "gauges[16][source]"=>"web.3", "gauges[16][measure_time]"=>1471912500, "gauges[16][value]"=>0.0, "gauges[20][name]"=>"thread_pool.running.999th_percentile", "gauges[20][source]"=>"web.3", "gauges[20][measure_time]"=>1471912500, "gauges[20][value]"=>0.0} 

Possible CDN issue for "inspec" gem

Moved from rubygems/bundler#4497

Given a simple gemfile like this:

source "https://rubygems.org"

gem "inspec", "~> 0.20.1"

I'm getting:

bundle install       
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Could not find inspec-0.20.1 in any of the sources

The same gemfile works fine with 1.11.x of bundler. I originally thought this had something to do with windows/not windows, but my windows machines are in Azure and my machines that do not see this problem are elsewhere, so it could be a CDN issue instead. I can repro the issue on a Windows VM and on my host OS (OS X) on my laptop.

Also, I've found that version 0.20.1 is not included if I navigate to https://index.rubygems.org/info/inspec in a browser (tried chrome and safari), but for whatever reason, I do see it if I fetch the data with curl.

Track request/response times in librato

Right now librato tracks internal response time (the time it takes to generate the response), but it doesn't track the amount of time that an HTTP request takes to go through the routing mesh blah blah blah. We should also track and graph the time it takes to make a request against the api at bundler-api.herokuapp.com and the time it takes to make a request against the api reverse proxied through rubygems.org.

Move the entire Bundler API into rubygems/rubygems.org

Then we can monitor, debug, and maintain in one place. 👍

Expire Fastly cache on new index files

Today, we tell Fastly to cache compact index files for a very long time, via HTTP headers:

Surrogate-Control: max-age=2592000, stale-while-revalidate=60

We need to tell Fastly when the cache is invalidated (by, say, a new gem being released), so that it will re-fetch an up-to-date version of the file. Resolving this should also fix rubygems/bundler#4371.

API Not Serving Dependencies to Bundler Properly?

Yo!

Got 2 projects using bundler to install jekyll & its runtime dependencies:

• https://travis-ci.org/gjtorikian/jekyll-last-modified-at/jobs/30090765#L62
• https://travis-ci.org/octopress/render-tag/jobs/30087040#L26

Looks like none of the runtime deps are installed unless explicitly in the local gemfile, which is super weird. I would expect all of the runtime dependencies to also be installed. Any idea what's up?

/cc @lmarburger @hone

/cc @rhk @henrikhodne @svenfuchs to see if it's a travis bundler issue.

Build read-through rubygems caching and mirroring

Once we support the new index format, we want the bundler-api sinatra app to also provide read-through caching for rubygems.org. Then, you can run one of those and point to it using bundle config mirror.https://rubygems.org https://mything.whatever.

• When gem metadata is requested, fetch and cache it from rubygems.org
• Provide .gem files and cache them when they are requested
• Allow stale cached gems to be removed(?)
• Add a flag to cache for longer than a few seconds
• Provide a command to actively fetch a set of gems or all gems immediately
• Allow pushing "local" gems to this server that are not sent to rubygems.org

Bundler::HTTPError during bundle

I'm not sure this is actually an issue with bundler-api. However what I'm seeing is the following:

Fetching gem metadata from http://rubygems.org/.
Error Bundler::HTTPError during request to dependency API
Fetching full source index from http://rubygems.org/
Using rake (10.0.3)
...
Using activesupport (2.3.17)
Using rack (1.1.6)

Bundler::HTTPError: Network error while fetching http://production.cf.rubygems.org/quick/Marshal.4.8/actionpack-2.3.17.gemspec.rz
An error occurred while installing actionpack (2.3.17), and Bundler cannot continue.
Make sure that gem install actionpack -v '2.3.17' succeeds before bundling.

gem i actionpack -v '2.3.17'
Successfully installed actionpack-2.3.17
1 gem installed

I can curl the actionpack-2.3.17.gemspec.rz file without any problems:
curl -O http://production.cf.rubygems.org/quick/Marshal.4.8/actionpack-2.3.17.gemspec.rz

And the gemspec seems fine:

ruby -e "puts Marshal.load(Gem.inflate(File.read('actionpack-2.3.17.gemspec.rz'))).summary"
Web-flow and rendering framework putting the VC in MVC.

I'm also wondering if there isn't something strange going on with my ISP or the hops between me and the rubygems servers because I wasn't having any issues when connecting from a different location.

Thanks for any help you can provide in troubleshooting this or please redirect me if there is a better forum for troubleshooting this issue.

Thanks

Capture version information from user agent header

Bundler 1.4 and 1.5 ship with a user agent header that returns ruby, rubygems, bundler, engine, and version information. We should store it! Maybe roll it up into redis? Maybe just send it to librato so we can graph it. Somewhere.

/api/v1/dependencies?gems=jpmobile always returns 500

Recently, https://bundler.rubygems.org/api/v1/dependencies?gems=jpmobile always returns 500.
As far as I know, this problem seems to have started around December 30.

Add ruby and rubygems requirements to database

New index format needs ruby and rubygems requirements. Add the columns to the database and backfill all existing rows in the versions table.

Instrument database response times

We report on the throughput of the database but what we really need is how long each database query takes. Since presumably each database query blocks a dyno, it'd be helpful to know how much time Ruby is sitting idle waiting for database I/O.

Build breaks on RubyGems 2.0

There are 2 specs which fail on RubyGems 2.0. They're both in gem_db_helper_spec.rb.

s.add_runtime_dependency(true, "> 0")
s.add_runtime_dependency(:baz, "> 0")

I'm not sure why these specs exist. @indirect or @hone, do you have any insight into this?

Vagrant file?

Hello,

Are you guys interested in a vagrant file? I can provide it.

Bundler freezes on dependencies

Sorry if this isn't the right place for this. But I saw this issue ( #17) and it resembled my issue, so I'll give it a shot.

I'm a contributor to https://github.com/mperham/sidekiq and a few days ago the Travis CI tests started erroring out (ie. https://travis-ci.org/mperham/sidekiq/jobs/4360052). As you can see, it makes the call to bundle install, prints the Fetching gem metadata from http://rubygems.org/ twice, and then freezes. Here's the verbose output:

[11:48:42] bhilkert [~/Dropbox/code/sidekiq] (master) $ bundle outdated --verbose
Fetching gem metadata from http://rubygems.org/
Query List: ["sidekiq", "minitest", "sinatra", "slim", "rake", "actionmailer", "activerecord", "celluloid", "sqlite3", "simplecov", "pry", "shotgun", "redis", "redis-namespace", "connection_pool", "multi_json"]
Query Gemcutter Dependency Endpoint API: sidekiq minitest sinatra slim rake actionmailer activerecord celluloid sqlite3 simplecov pry shotgun redis redis-namespace connection_pool multi_json
Fetching from: http://rubygems.org/api/v1/dependencies?gems=sidekiq,minitest,sinatra,slim,rake,actionmailer,activerecord,celluloid,sqlite3,simplecov,pry,shotgun,redis,redis-namespace,connection_pool,multi_json
HTTP Success
Query List: ["mail", "actionpack", "text-format", "tzinfo", "arel", "activemodel", "activesupport", "facter", "timers", "hoe", "win32console", "method_source", "slop", "coderay", "spoon", "ruby_parser", "yard", "rspec", "rack", "launchy", "compass", "sass", "sprockets-sass", "simplecov-html", "tilt", "rack-protection", "mongrel", "rack-test", "temple", "escape_utils", "ffi"]
Query Gemcutter Dependency Endpoint API: mail actionpack text-format tzinfo arel activemodel activesupport facter timers hoe win32console method_source slop coderay spoon ruby_parser yard rspec rack launchy compass sass sprockets-sass simplecov-html tilt rack-protection mongrel rack-test temple escape_utils ffi
Fetching from: http://rubygems.org/api/v1/dependencies?gems=mail,actionpack,text-format,tzinfo,arel,activemodel,activesupport,facter,timers,hoe,win32console,method_source,slop,coderay,spoon,ruby_parser,yard,rspec,rack,launchy,compass,sass,sprockets-sass,simplecov-html,tilt,rack-protection,mongrel,rack-test,temple,escape_utils,ffi
HTTP Success
Query List: ["erubis", "sprockets", "journey", "builder", "rack-cache", "rack-mount", "i18n", "bcrypt-ruby", "memcache-client", "haml", "chunky_png", "fssm", "bones", "RubyInline", "rubyforge", "gemcutter", "configuration", "addressable", "treetop", "mime-types", "tlsmail", "daemons", "fastthread", "gem_plugin", "cgi_multipart_eof_fix", "test-spec", "camping", "fcgi", "ruby-openid", "thin", "rspec-mocks", "rspec-expectations", "rspec-core", "cucumber", "diff-lcs", "syntax", "spicycode-rcov", "sexp_processor", "ParseTree", "sprockets-helpers", "text-hyphen"]
Query Gemcutter Dependency Endpoint API: erubis sprockets journey builder rack-cache rack-mount i18n bcrypt-ruby memcache-client haml chunky_png fssm bones RubyInline rubyforge gemcutter configuration addressable treetop mime-types tlsmail daemons fastthread gem_plugin cgi_multipart_eof_fix test-spec camping fcgi ruby-openid thin rspec-mocks rspec-expectations rspec-core cucumber diff-lcs syntax spicycode-rcov sexp_processor ParseTree sprockets-helpers text-hyphen
Fetching from: http://rubygems.org/api/v1/dependencies?gems=erubis,sprockets,journey,builder,rack-cache,rack-mount,i18n,bcrypt-ruby,memcache-client,haml,chunky_png,fssm,bones,RubyInline,rubyforge,gemcutter,configuration,addressable,treetop,mime-types,tlsmail,daemons,fastthread,gem_plugin,cgi_multipart_eof_fix,test-spec,camping,fcgi,ruby-openid,thin,rspec-mocks,rspec-expectations,rspec-core,cucumber,diff-lcs,syntax,spicycode-rcov,sexp_processor,ParseTree,sprockets-helpers,text-hyphen
HTTP Success
Query List: ["little-plugger", "loquacious", "bones-git", "bones-extras", "markaby", "metaid", "mab", "term-ansicolor", "polyglot", "gherkin", "json_pure", "json", "abstract", "net-scp", "maruku", "ZenTest", "archive-tar-minitar", "nokogiri", "rcov", "SexpProcessor", "multimap", "ruby-yadis", "hike", "flexmock", "mocha", "eventmachine", "eventmachine-le", "http_parser.rb", "preforker", "facets"]
Query Gemcutter Dependency Endpoint API: little-plugger loquacious bones-git bones-extras markaby metaid mab term-ansicolor polyglot gherkin json_pure json abstract net-scp maruku ZenTest archive-tar-minitar nokogiri rcov SexpProcessor multimap ruby-yadis hike flexmock mocha eventmachine eventmachine-le http_parser.rb preforker facets
Fetching from: http://rubygems.org/api/v1/dependencies?gems=little-plugger,loquacious,bones-git,bones-extras,markaby,metaid,mab,term-ansicolor,polyglot,gherkin,json_pure,json,abstract,net-scp,maruku,ZenTest,archive-tar-minitar,nokogiri,rcov,SexpProcessor,multimap,ruby-yadis,hike,flexmock,mocha,eventmachine,eventmachine-le,http_parser.rb,preforker,facets
HTTP Success
Query List: ["bones-rcov", "bones-rubyforge", "bones-rspec", "bones-zentest", "git", "trollop", "spruz", "metaclass", "net-ssh", "racc", "tenderlove-frex", "rake-compiler", "rexical", "weakling"]
Query Gemcutter Dependency Endpoint API: bones-rcov bones-rubyforge bones-rspec bones-zentest git trollop spruz metaclass net-ssh racc tenderlove-frex rake-compiler rexical weakling
Fetching from: http://rubygems.org/api/v1/dependencies?gems=bones-rcov,bones-rubyforge,bones-rspec,bones-zentest,git,trollop,spruz,metaclass,net-ssh,racc,tenderlove-frex,rake-compiler,rexical,weakling
HTTP Success
Query List: ["needle", "jruby-pageant", "echoe"]
Query Gemcutter Dependency Endpoint API: needle jruby-pageant echoe
Fetching from: http://rubygems.org/api/v1/dependencies?gems=needle,jruby-pageant,echoe
HTTP Success
Query List: ["highline", "allison", "rdoc"]
Query Gemcutter Dependency Endpoint API: highline allison rdoc
Fetching from: http://rubygems.org/api/v1/dependencies?gems=highline,allison,rdoc
HTTP Success
Query List: ["termios"]
Query Gemcutter Dependency Endpoint API: termios
Fetching from: http://rubygems.org/api/v1/dependencies?gems=termios
HTTP Success
Query List: []
Unmet Dependencies: ["tenderlove-frex", "spicycode-rcov"]
Fetching gem metadata from http://rubygems.org/
Query List: ["tenderlove-frex", "spicycode-rcov"]
Query Gemcutter Dependency Endpoint API: tenderlove-frex spicycode-rcov
Fetching from: http://rubygems.org/api/v1/dependencies?gems=tenderlove-frex,spicycode-rcov
HTTP Success
Query List: []

I should also mention that I didn't originally notice this b/c I had all the dependencies installed on my machine already, so bundle install wasn't doing any more. Only when I ran bundle outdated did it actually go out and hit this. https://github.com/mperham doesn't have this issue. He is using rvm. I have using Rbenv, but I fail to understand why that would change anything.

I also tried to update bundler, but to no avail:

[12:03:53] bhilkert [~/Dropbox/code/sidekiq] (master) $ gem update bundle
Updating installed gems
Updating bundler
Fetching: bundler-1.2.3.gem (100%)
Successfully installed bundler-1.2.3
Gems updated: bundler
Installing ri documentation for bundler-1.2.3...
Installing RDoc documentation for bundler-1.2.3...

I found a few other cases involving tenderlove-frex:

https://groups.google.com/forum/?fromgroups=#!topic/rubygems-org/sgpO6ed_kAM

http://stackoverflow.com/questions/13443332/bundle-update-stuck

http://proxy.obd2tool.com/index.php?url=141L19y91q1HyOyr0I1CyX1s0UyGyFyMyb1v121t1lyd0Kyj1tyN1xya

http://iqwen.net/question/72831

Any advice?

Proposal serving new index format

Basing on André's overview of new index format in EY blog (https://blog.engineyard.com/2014/new-rubygems-index-format) and quick look at the code. I've revamped my comment so we don't rely on nginx:

How many roundtrips a usual rails bundle install call makes?

1. [not used?] Downloads /names.list to validate names from Gemfile
2. Downloads /versions.list (though we need, say, 50 out of 100k version numbers from that file). The file is updated very frequently, and therefore not cached.
3. 50 deps requests
4. Even more requests on dependent gems
5. Even more requests on dependencies of dependencies (btw it's interesting, how deep this could possibly be?)

TOO much requests (I've counted 1 + 50 + 70 calls for an app I'm currently working on). Expensive HTTP requests, header parsing, traffic overhead et c.

Proposal:

Once a new gem version is pushed, you have to update version.list file, corresponding dependencies/gem_name.file and never go to the database issuing an expensive query with JOINS, never concatenate or split strings ever again. A modern file system should handle 100k files in a single directory with no slowdown.

BTW how does bundler api knows that a new gem has been pushed to rubygems?

Side note: As far as I can see, you are sorting in ruby, twice, gem_info.rb:81, and web.rb:171. This could be done on DB side. But I would go for pre-sorted files.

Further improvement:
Get accumulated request, similar to Gemfile format, but probably in JSON: {'rails': '~> 3.2.16', 'rake': '~> 0.9.2', ... }. Send accumulated response for 'versions':

{
  'rails': '3.2.16, 3.2.17, 4.0.4, 4.1.0',
  'rake': '0.9.2, 10.2.0, 10.3.0, ...',
  ...
}

And second request, /deps:

{
  'rails': '3.2.16 rake:>= 0.7.1,activesupport:= 1.3.1|ruby:> 1.8.7,checksum:bcd234\n
             4.0.4 ...',
  'rack': '0.9.2 ...',
  ...
}

This limits the number of requests to just two (probably three if we add a pre-request to /names just to validate that gem names are ok) plus requests to dependencies, should be equal to the deepest dependencies (rails -> activerecord -> arel -> hoe).

Advantage: significantly less HTTP requests. No SQL, just read and concatenate file contents into request. Less traffic, no need to send over versions of all existing gems.

composite_primary_keys gem v 5.0.10 missing dependencies in dependencies API

composite_primary_keys 5.0.10 depends on 'activerecord', '>= 3.2.9', '~> 3.2.0'. This dependency is visible in the rubygems.org UI, in metadata.gz in the .gem package, and when using the gem dependency command after installing.

However, the dependencies API omits the dependency for that version of the gem:

[{:name=>"composite_primary_keys",
  :number=>"5.0.10",
  :platform=>"ruby",
  :dependencies=>[]},
 {:name=>"composite_primary_keys",
  :number=>"5.0.9",
  :platform=>"ruby",
  :dependencies=>[["activerecord", ">= 3.2.8, ~> 3.2.0"]]},
 {:name=>"composite_primary_keys",
  :number=>"5.0.8",
  :platform=>"ruby",
  :dependencies=>[["activerecord", "~> 3.2.0"]]},
# ...

(Output from this script.)

A practical result of this omission is that Bundler will always choose version 5.0.10 when doing a clean resolve even when a project is using a version of activerecord that is not compatible with version 5.0.10.

(/cc @cfis, the current maintainer of composite_primary_keys.)

Update Appsignal

In addition, we probably need to use send_exception instead of add_exception