shopify / toxiproxy-ruby

A ruby api for Toxiproxy that can be used for resiliency testing.

toxiproxy-ruby

toxiproxy-ruby >= 1.x is compatible with the Toxiproxy 2.x series. toxiproxy-ruby 0.x is compatible with the Toxiproxy 1.x series.

Toxiproxy is a proxy to simulate network and system conditions. The Ruby API aims to make it simple to write tests that ensure your application behaves appropriately under harsh conditions. Before you can use the Ruby library, you need to read the Usage section of the Toxiproxy README.

$ gem install toxiproxy

Make sure the Toxiproxy server is already running.

For more information about Toxiproxy and the available toxics, see the Toxiproxy documentation

Usage

The Ruby client communicates with the Toxiproxy daemon via HTTP. By default it connects to http://127.0.0.1:8474, but you can point to any host:

Toxiproxy.host = 'http://toxiproxy.local:5665'

For example, to simulate 1000ms latency on a database server you can use the latency toxic with the latency argument (see the Toxiproxy project for a list of all toxics):

Toxiproxy[:mysql_master].toxic(:latency, latency: 1000).apply do
  Shop.first # this took at least 1s
end

You can also take an endpoint down for the duration of a block at the TCP level:

Toxiproxy[:mysql_master].down do
  Shop.first # this'll raise
end

If you want to simulate all your Redis instances being down:

Toxiproxy[/redis/].down do
  # any redis call will fail
end

If you want to simulate that your cache server is slow at incoming network (upstream), but fast at outgoing (downstream), you can apply a toxic to just the upstream:

Toxiproxy[:cache].upstream(:latency, latency: 1000).apply do
  Cache.get(:omg) # will take at least a second
end

By default the toxic is applied to the downstream connection, you can be explicit and chain them:

Toxiproxy[/redis/].upstream(:slow_close, delay: 100).downstream(:latency, jitter: 300).apply do
  # all redises are now slow at responding and closing
end

See the Toxiproxy README for a list of toxics.

Populate

To populate Toxiproxy pass the proxy configurations to Toxiproxy#populate:

Toxiproxy.populate([{
  name: "mysql_master",
  listen: "localhost:21212",
  upstream: "localhost:3306",
},{
  name: "mysql_read_only",
  listen: "localhost:21213",
  upstream: "localhost:3306",
}])

This will create the proxies passed, or replace the proxies if they already exist in Toxiproxy. It's recommended to do this early as early in boot as possible, see the Toxiproxy README. If you have many proxies, we recommend storing the Toxiproxy configs in a configuration file and deserializing it into Toxiproxy.populate.

If you're doing this in Rails, you may have to do this in config/boot.rb (as early in boot as possible) as older versions of ActiveRecord establish a database connection as soon as it's loaded.

toxiproxy-ruby's Issues

MySQL client is not connected

Hi team, during the testing I found a very unusual behaviour of Toxiproxy#down method. The following snippet causes the MySQL client is not connected error:

test "#record_create throws a SQL exception if mysql_master database is down" do
  record = Record.new(attributes)
  Toxiproxy[:mysql_master].down do
    assert_raises(ActiveRecord::ConnectionNotEstablished, ActiveRecord::StatementInvalid) do
       create_record(record)
    end
  end
  Record.first # => raising an error, i.e. "ActiveRecord::ConnectionNotEstablished: MySQL client is not connected"
end

Override toxyproxy host uri

Would there be any negative side-effect to allowing the host uri to be changed from the default: 127.0.0.1:8474?

Referencing this

I'm working with a multi-container docker environment where it would be ideal to keep the toxyproxy server in a separate container from my source code. I would like to configure the Toxyproxy client to send requests to the separate container, rather than the home IP.

I'm also volunteering myself to making the change if you guys approve.

Thanks

#17 (comment)

Asserting for DB failovers

I'm using Toxiproxy for our external services and we're now getting ready to do a bunch of DB failover work. To better handle our failovers without dropping queries, we've patched ActiveRecord to catch any MySQL errors, perform a reconnect and then try the query again. I can manually confirm this works by kicking off this script and either toggling the availability of the toxiproxy or DB server manually during the execution.

ATTEMPT_COUNT = 300
puts "==> Truncating the users test_db table"
ActiveRecord::Base.connection.execute("truncate test_db.user")

puts "==> Starting to send MySQL queries"
require 'securerandom'
ATTEMPT_COUNT.times do
  sleep 0.1
  hash = SecureRandom.uuid
  begin
    puts "    [#{Time.now.strftime("%T.%L")}] Inserting #{hash}"
    ActiveRecord::Base.connection.execute("INSERT INTO user (first_name, last_name) VALUES ('test', '#{hash}')")
    puts "    Success!"
  rescue Exception => e
    puts "    [#{Time.now.strftime("%T.%L")}] #{e}"
  end
end

row_count = ActiveRecord::Base.connection.execute("select * from user").count
puts
puts "Attempted writes: #{ATTEMPT_COUNT}"
puts "DB row count:     #{row_count}"
puts "Variance:         #{ATTEMPT_COUNT - row_count}"

However, I'm getting a little stuck when it comes to using Toxiproxy to emulate the failover completing. I first tried:

Toxiproxy[:mysql_master].down do
  User.first
end

It seems our patch works a little too well because it sits here waiting for the MySQL server to come back but it never does as the yield is still running. I then tried to split the enable/disable but still had the same results with the following:

Toxiproxy[:mysql_master].disable
User.first
Toxiproxy[:mysql_master].enable

Which leads me to the following questions:

• Could you share how your using Toxiproxy with things like DB failovers? Is this something you're able to test similarly to my intention or do you handle it on a per model basis? I essentially need the proxy to only be present for a short period of time but re-enable after the time has passed.
• The only way I could think of having this work would be to pass another argument to down (and later disable) which would only disable the proxy for a period of time. Is applying a non-blocking timeout to that functionality something you'd consider useful for the library?

Thanks!

Action required: CircleCI v1.0 is EOL, this repo is impacted

This repository relies on CircleCI v1.0 which will be shut down on Aug 31st.

cc @fw42

Play nice with Webmock, VCR, ...

It'd be nice if Toxiproxy by default disabled Webmock complaining about it. A test helper in general would be favourable for Toxiproxy to avoid common mistakes.

Migrate trunk branch from master to main

This issue is created to use less-loaded and more-inclusive language that other repos and projects have adopted.

For toxiproxy-ruby:
There is no buildkite pipeline to update

• update default branch
• update protection rule
• update shipit
• update GH test action workflow, readme, gemspec's change log link
• Delete master branch