dannycoates / able Goto Github PK
View Code? Open in Web Editor NEWA/B testing service
Home Page: http://dannycoates.github.io/able/
A/B testing service
Home Page: http://dannycoates.github.io/able/
Ref: mozilla/fxa-content-server#3693
Currently the package.json file hardcodes to [email protected], which [unfortunately] has a non-kosher version of moment.
We should upgrade to the latest version of convict to appease the NSP gods.
Seems like the https_url
is not the right one to use here: https://github.com/dannycoates/able/blob/master/project.js#L25
Project.prototype.pull = function (cb) {
gitUtil.cloneOrFetch(
this.git.https_url,
this.root,
this.git.branch,
cb
)
}
{ user: 'mozilla',
repo: 'fxa-content-experiments',
branch: 'dev',
tarball_url: 'https://api.github.com/repos/mozilla/fxa-content-experiments/tarball/dev',
https_url: 'https://github.com/mozilla/fxa-content-experiments/tree/dev',
travis_url: 'https://travis-ci.org/mozilla/fxa-content-experiments?branch=dev',
api_url: 'https://api.github.com/repos/mozilla/fxa-content-experiments' }
git clone -b dev https://github.com/mozilla/fxa-content-experiments/tree/dev /Users/vladikoff/mozilla/fxa-local-dev/fxa-content-server/experiments
fxa-content-server.route.500.ERROR: Error: Command failed: Cloning into '/Users/vladikoff/mozilla/fxa-local-dev/fxa-content-server/experiments'...
fatal: repository 'https://github.com/mozilla/fxa-content-experiments/tree/dev/' not found
It tries to clone https://github.com/mozilla/fxa-content-experiments/tree/dev
which is not a repository :(
We've had a few minor PRs land recently which bumped convict and uglify-js versions.
Once we get a newer version pushed to npm, we can update express-able package.json to use the latest able and bump publish express-able to npm so we can update our fxa-content-server dependency version.
For A/B experiments to actually work we need a way to analyze the data, but before we can even do that we need a way to report the data.
Right now all that exists is able.report()
which will give you some data for each experiment you're enrolled in; the experiment name and the independentVariables:values that were chosen for each subject (usually only one).
{
"name":"myExperiment",
"choices": {
"1a4b2d54c3e": {
"buttonText": "why not?"
}
}
}
That's useful for knowing how many subjects got each value, but its not enough to do anything. Somehow we need to link choices to events that are relevant to the experiment.
In the most naive way I think it would be nice to define which events my experiment is interested in tracking and then have the choices linked to those events.
Borrowing from Shane's example I'd like to add events
module.exports = {
name: 'signInButtonTextMatters',
hypothesis: 'The sign in button text affects signins',
startDate: '2015-01-01',
subjectAttributes: ['lang'],
independentVariables: ['signInButtonText'],
eligibilityFunction: function (subject) {
return /en-US/.test(subject.lang);
},
groupingFunction: function (subject) {
return {
signInButtonText: this.uniformChoice([
this.defaults.signInButtonText,
'Come on in'
]);
};
},
events: ['signInClicked']
};
So, whenever the signInClicked
event fired, in addition to whatever it normally logs the "report" for that experiment would get logged so that we can correlate the choices with the event.
This means the experiment author will need to know what events are available to track, just as they need to know the independentVariables.
Gluing Able to whatever generates and logs these events will be an Issue for another day, but I'm wondering if a simple list of events is enough or do we need something more powerful?
I believe that github-url-to-object added clone_url
support in github-modules/github-url-to-object@8bd5878 (published as v2.1.0).
We can probably switch from @vladikoff's fork to the core module.
Need to bump to latest version of glob once isaacs/node-glob#268 is fixed.
We want to add some basic unit tests to our experiments but that means the tests/
directory would be hit by the experiment loader. Ideas?
(I wonder if we can make a .tests
directory to deal with that)
less an issue and more a suggestion, but having s3 bucket support could be a good way for the application to support dynamic update of the experiments for a given application
Not sure if we're using CORS, but noticed this while running nsp against a related project:
$ nsp check
(+) 2 vulnerabilities found
┌───────────────┬──────────────────────────────────────────────────────────┐
│ │ Incorrect handling of CORS preflight request headers │
├───────────────┼──────────────────────────────────────────────────────────┤
│ Name │ hapi │
├───────────────┼──────────────────────────────────────────────────────────┤
│ Installed │ 9.3.1 │
├───────────────┼──────────────────────────────────────────────────────────┤
│ Vulnerable │ <11.0.0 │
├───────────────┼──────────────────────────────────────────────────────────┤
│ Patched │ >=11.0.0 │
├───────────────┼──────────────────────────────────────────────────────────┤
│ Path │ able > hapi │
├───────────────┼──────────────────────────────────────────────────────────┤
│ More Info │ https://nodesecurity.io/advisories/45 │
└───────────────┴──────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────┐
│ │ Regular Expression Denial of Service │
├───────────────┼──────────────────────────────────────────────────────────┤
│ Name │ uglify-js │
├───────────────┼──────────────────────────────────────────────────────────┤
│ Installed │ 2.4.24 │
├───────────────┼──────────────────────────────────────────────────────────┤
│ Vulnerable │ All │
├───────────────┼──────────────────────────────────────────────────────────┤
│ Patched │ None │
├───────────────┼──────────────────────────────────────────────────────────┤
│ Path │ able > uglify-js │
├───────────────┼──────────────────────────────────────────────────────────┤
│ More Info │ https://nodesecurity.io/advisories/48 │
└───────────────┴──────────────────────────────────────────────────────────┘
I filed the Hapi issue upstream as hapijs/hapi#2896. Not sure if there is anything super-actionable we can do with it currently, unless we migrate to hapi@11 (which I believe requires Node 4+).
Could default to "local" for now, but as this grows, we'll want to be able to spec other storage options/types
@dannycoates would you be okay with that?
We should add Travis to this repo so run tests
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.