mattt / skylab

Multivariate & A/B Testing for iOS and Mac

This library is no longer being maintained. You can continue to use SkyLab in your projects, but we recommend switching another solution whenever you have the opportunity.

SkyLab

Multivariate & A/B Testing for iOS and Mac

SkyLab is a backend-agnostic framework for multivariate and A/B testing.

Test conditions are persisted across sessions and launches using NSUserDefaults, ensuring that every user will have a consistent experience, no matter which testing bucket they end up in.

SkyLab integrates easily into any existing statistics web service. Depending on your particular needs, this may include posting to an endpoint in test blocks, or perhaps setting an HTTP header for a shared API client.

Requests for integration with any particular backend are heartily encouraged.

This project is part of a series of open source libraries covering the mission-critical aspects of an iOS app's infrastructure. Be sure to check out its sister projects: GroundControl, CargoBay, houston, and Orbiter.

Usage

Check out the included example project to see everything in action.

Simple A/B Test

[SkyLab abTestWithName:@"Title" A:^{
    self.titleLabel.text = NSLocalizedString(@"Hello, World!", nil);
} B:^{
    self.titleLabel.text = NSLocalizedString(@"Greetings, Planet!", nil);
}];

Split Test with Weighted Probabilities

You can pass either an NSDictionary (with values representing the weighted probability of their corresponding key) or an NSArray (with each value having an equal chance of being chosen) into the choices parameter.

[SkyLab splitTestWithName:@"Subtitle" conditions:@{
    @"Red" : @(0.15),
    @"Green" : @(0.10),
    @"Blue" : @(0.50),
    @"Purple" : @(0.25)
} block:^(id choice) {
    self.subtitleLabel.text = NSLocalizedString(@"Please Enjoy This Colorful Message", nil);

    if ([choice isEqualToString:@"Red"]) {
        self.subtitleLabel.textColor = [UIColor redColor];
    } else if ([choice isEqualToString:@"Green"]) {
        self.subtitleLabel.textColor = [UIColor greenColor];
    } else if ([choice isEqualToString:@"Blue"]) {
        self.subtitleLabel.textColor = [UIColor blueColor];
    } else if ([choice isEqualToString:@"Purple"]) {
        self.subtitleLabel.textColor = [UIColor purpleColor];
    }
}];

Multivariate Test

[SkyLab multivariateTestWithName:@"Switches" variables:@{
    @"Left" : @(0.5),
    @"Center" : @(0.5),
    @"Right" : @(0.5)
 } block:^(NSSet *activeVariables) {
     self.leftSwitch.on = [activeVariables containsObject:@"Left"];
     self.centerSwitch.on = [activeVariables containsObject:@"Center"];
     self.rightSwitch.on = [activeVariables containsObject:@"Right"];
}];

License

SkyLab is available under the MIT license. See the LICENSE file for more info.

skylab's Issues

splitTestWithName triggers NSUserDefaultsDidChangeNotification every time it's called

It looks like:

    if ([choices isKindOfClass:[NSArray class]]) {
        if (!choice || ![choices containsObject:choice]) {
            choice = SLRandomValueFromArray(choices);
        }
    } else if ([choices isKindOfClass:[NSDictionary class]]) {
        if (!choice || ![[choices allKeys] containsObject:choice]) {
            choice = SLRandomKeyFromDictionaryWithWeightedValues(choices);
        }
    } else {
        @throw [NSException exceptionWithName:NSInvalidArgumentException reason:NSLocalizedString(@"Parameter `choices` must be either array or dictionary", nil) userInfo:nil];
    }

    [[NSUserDefaults standardUserDefaults] setObject:choice forKey:SLUserDefaultsKeyForTestName(name)];
    [[NSUserDefaults standardUserDefaults] synchronize];

should change to:

    if ([choices isKindOfClass:[NSArray class]]) {
        if (!choice || ![choices containsObject:choice]) {
            choice = SLRandomValueFromArray(choices);
        }
        [[NSUserDefaults standardUserDefaults] setObject:choice forKey:SLUserDefaultsKeyForTestName(name)];
        [[NSUserDefaults standardUserDefaults] synchronize];
    } else if ([choices isKindOfClass:[NSDictionary class]]) {
        if (!choice || ![[choices allKeys] containsObject:choice]) {
            choice = SLRandomKeyFromDictionaryWithWeightedValues(choices);
        }
        [[NSUserDefaults standardUserDefaults] setObject:choice forKey:SLUserDefaultsKeyForTestName(name)];
        [[NSUserDefaults standardUserDefaults] synchronize];
    } else {
        @throw [NSException exceptionWithName:NSInvalidArgumentException reason:NSLocalizedString(@"Parameter `choices` must be either array or dictionary", nil) userInfo:nil];
    }

It looks like the same pattern is used in multivariateTestWithName:variables:block as well.

We ran across this because we were calling splitTestWithName:choices:block: from inside tableView:cellForRowAtIndexPath: and noticed repeated NSUserDefaultsDidChangeNotifications in the logs. Granted, that's not a great way to use it and we've refactored.

Please let me know if you'd rather have a pull request.

Add documentation about backend integration

It would be nice to hear thoughts on how to integration with a backend. Not about how to do network requests, but things like should I just read keys out of NSUserDefaults, what should I do with them to make a meaningful conclusion?

I also didn't see an API for recording outcomes (positive/negative). How do you handle that?

Thanks

Synchronous AB tests with SkyLab

All of SkyLab's methods rely on blocks -- which is a great structure for control-flow between tests. Things look a little strange, though, once you want something synchronously. ie,

- (BOOL)showNewFancyButton 
{
     BOOL showNewFancyButton;
    [SkyLab abTestWithName:@"ShowNewFancyButton" A:^{
        showNewFancyButton = YES;
    } B:^{
        showNewFancyButton = NO;
    }]

    return showNewFancyButton;
}

Looking at the implementation of SkyLab, the above code will work just fine, but we thought it's a little weird to rely on blocks to run synchronously (even though we know the implementation does).

Do you think it'd be cool for SkyLabs to have non-block counterparts to its block-based methods? Or would you just suggest doing synchronous operations like above?

update for swift 3+? ios 10?

any updates coming?

about @[ @"A", @"B" ]

Thank first. I am already using SkyLab and GroundControl.

It makes life easy to init a array or dictionary using @[] or @{}. But it will be a problem for those whose XCode is still under 4.2. I encountered this problem.

In the code you init a array using this:

@[ @"A", @"B" ]

would you please change it to [NSArray arrayWithObjects:@"A", @"B", nil]?

Overflow in expression; result is -2147483648 with type 'int'

This warning is emanated from SkyLab.m line 67:

double r = (random() % (INT_MAX + 1)) * (total / INT_MAX);

I assume INT_MAX + 1 is supposed to be INT_MIN? Or is the overflow intentional?