angles-n-daemons / scrutinize Goto Github PK

Still needed for a 0.1 release, users need to be able to create experiments with more configuration so they can conduct them.

An experiment page should have the following fields:

Name
Percentage Rollout (caveat for initial release being low)
Evaluation Criterion

Additionally, there should be an edit icon on the experiments which brings them to the experiment edit page.

A go client should be created so that go users (who I imagine will make up a non-negligible amount of the user base) have access to scrutinize

I'd like to add an experiment edit page where users can:

• Modify Metrics
• Modify Rollout

Treatments are manually assigned measurements right now when measurements are given. This is done via a hacky most recent lookup mechanism and takes up space because each measurement needs to be duplicated for each active experiment treatment.

I have since realized I can do reporting on measurements if I filter them by the dates of the experiments and join them onto the treatments user ids to associate a variant. This is a more scalable and reasonable solution in my mind.

Metric table should now have an index on experiment & date.

A Subtype of Experiment, ExperimentRun must be created so that experiments can be stopped, restarted, and rerun. Doing so will follow a more traditional experimentation workflow. Example subtasks for this goal include:

• Creating the ExperimentRun schema
• Modify the ExperimentList to have the following
  • The experiment run iteration
  • A status field for the experiment (scheduled, live, inactive)
  • A killswitch or create experiment run button - depending on status
• Add Create Experiment Run Modal

I'd like to tie Measurements with multiple treatments, so that metrics can be used as Evaluation Criterion for multiple experiments.

Right now if you do experiment.observe(user_id, 'checked_out', 1) you'll only add a link to the treatment associated with the experiment in question. I'd like to record observations for all experiments associated with the user id. The following needs to happen for this, and the changes are breaking.

1. Modify the Measurement schema to no longer have a TreatmentID & Variant
2. Create a new Relation Observation that binds Measurements to Treatments (many to many)
3. Modify the API for creating measurements to have a "Lookback Period" parameter. This parameter will be a duration in seconds specifying how far back from the current moment to look for treatments to bind to the incoming measurement.
4. Modify the clients to use the new API.
5. Modify the reporting tool to query using this new schema.

###Open Question
Should the measurement table be denormalized?

Pro:

• Performance will be increased if the table is normalized (important for analytical functions).

Con:

• If there's no treatment to bind, measurements could be dropped.
• Performance will be dropped, normalizing this will require 2 table joins with a lot of rows

- Old relational model

I'd like to also add some updates to metrics that make them a little more approachable:

• Auto creation of metrics published in code
• Ability to look at metric performance

The performance page will need a bit of a rework. It should first show a list of Run Iterations which can be clicked into their respective reports. The metrics embedded in those reports should then become what the performance page currently is.

• Show instead the report generated
  • Metric Changed T-Test and confidence interval
  • Gray/Green/Red colors to indicate success failure
• If the user clicks into a metric show the graph of the dates (same way it's done now)

Each experiment run should generate a report for an experiment. The changes needed to support this feature are as follows:

Create Report & Report Field schema
2 metric fields (up metrics and down metrics)
When an experiment is turned off - IN A TRANSACTION do a t-test calculation of all the fields.
Do also confidence interval calculations of all fields.

Drop down in performance page to select run.

The current seeding implementation is manual, and difficult to describe. I'd like a more clear and straightforward seeding strategy which lives in the server or the database, not the javascript client.