🍁 A/B testing for Rails
- Designed for web and email
- Comes with a dashboard to view results and update variants
- Uses your database for storage
- Seamlessly handles the transition from anonymous visitor to logged in user
Uses Bayesian statistics to evaluate results so you don’t need to choose a sample size ahead of time.
Add this line to your application’s Gemfile:
gem "field_test"Run:
rails generate field_test:install
rails db:migrateAnd mount the dashboard in your config/routes.rb:
mount FieldTest::Engine, at: "field_test"Be sure to secure the dashboard in production.
Add an experiment to config/field_test.yml.
experiments:
button_color:
variants:
- red
- green
- blueRefer to it in controllers, views, and mailers.
button_color = field_test(:button_color)To make testing easier, you can specify a variant with query parameters
http://localhost:3000/?field_test[button_color]=green
When someone converts, record it with:
field_test_converted(:button_color)When an experiment is over, specify a winner:
experiments:
button_color:
winner: greenAll calls to field_test will now return the winner, and metrics will stop being recorded.
You can keep returning the variant for existing participants after a winner is declared:
experiments:
button_color:
winner: green
keep_variant: trueYou can also close an experiment to new participants without declaring a winner while still recording metrics for existing participants:
experiments:
button_color:
closed: trueCalls to field_test for new participants will return the control, and they won’t be added to the experiment.
You can get the list of experiments and variants for a user with:
field_test_experimentsFor JavaScript and native apps, add calls to your normal endpoints.
class CheckoutController < ActionController::API
def start
render json: {button_color: field_test(:button_color)}
end
def finish
field_test_converted(:button_color)
# ...
end
endFor anonymous visitors in native apps, pass a Field-Test-Visitor header with a unique identifier.
Any model or string can be a participant in an experiment.
For web requests, it uses current_user (if it exists) and an anonymous visitor id to determine the participant. Set your own with:
class ApplicationController < ActionController::Base
def field_test_participant
current_company
end
endFor mailers, it tries @user then params[:user] to determine the participant. Set your own with:
class ApplicationMailer < ActionMailer::Base
def field_test_participant
@company
end
endYou can also manually pass a participant with:
field_test(:button_color, participant: company)To get variants in jobs, models, and other contexts, use:
experiment = FieldTest::Experiment.find(:button_color)
button_color = experiment.variant(user)By default, bots are returned the first variant and excluded from metrics. Change this with:
exclude:
bots: falseExclude certain IP addresses with:
exclude:
ips:
- 127.0.0.1
- 10.0.0.0/8You can also use custom logic:
field_test(:button_color, exclude: request.user_agent == "Test")Keep track of when experiments started and ended. Use any format Time.parse accepts. Variants assigned outside this window are not included in metrics.
experiments:
button_color:
started_at: Dec 1, 2016 8 am PST
ended_at: Dec 8, 2016 2 pm PSTAdd a friendlier name and description with:
experiments:
button_color:
name: Buttons!
description: >
Different button colors
for the landing page.By default, variants are given the same probability of being selected. Change this with:
experiments:
button_color:
variants:
- red
- blue
weights:
- 85
- 15To help with GDPR compliance, you can switch from cookies to anonymity sets for anonymous visitors. Visitors with the same IP mask and user agent are grouped together.
cookies: falseIf the dashboard gets slow, you can make it faster with:
cache: trueThis will use the Rails cache to speed up winning probability calculations.
If you need more precision, set:
precision: 1You can set multiple goals for an experiment to track conversions at different parts of the funnel. First, run:
rails generate field_test:events
rails db:migrateAnd add to your config:
experiments:
button_color:
goals:
- signed_up
- orderedSpecify a goal during conversion with:
field_test_converted(:button_color, goal: "ordered")The results for all goals will appear on the dashboard.
You may also want to send experiment data as properties to other analytics platforms like Segment, Amplitude, and Ahoy. Get the list of experiments and variants with:
field_test_experimentsYou can configure Field Test to use Ahoy’s visitor token instead of creating its own:
class ApplicationController < ActionController::Base
def field_test_participant
[ahoy.user, ahoy.visitor_token]
end
endauthenticate :user, ->(user) { user.admin? } do
mount FieldTest::Engine, at: "field_test"
endSet the following variables in your environment or an initializer.
ENV["FIELD_TEST_USERNAME"] = "moonrise"
ENV["FIELD_TEST_PASSWORD"] = "kingdom"Assign a specific variant to a user with:
experiment = FieldTest::Experiment.find(:button_color)
experiment.variant(participant, variant: "green")You can also change a user’s variant from the dashboard.
To associate models with field test memberships, use:
class User < ApplicationRecord
has_many :field_test_memberships, class_name: "FieldTest::Membership", as: :participant
endNow you can do:
user.field_test_membershipsA huge thanks to Evan Miller for deriving the Bayesian formulas.
View the changelog
Everyone is encouraged to help improve this project. Here are a few ways you can help:
- Report bugs
- Fix bugs and submit pull requests
- Write, clarify, or fix documentation
- Suggest or add new features
To get started with development:
git clone https://github.com/ankane/field_test.git
cd field_test
bundle install
bundle exec rake compile
bundle exec rake test
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent