Make final decisions on the specific "stickiness/gamification" functionality about common-voice

[mikehenrty]

Recommendations:

• Single responsive website, with special collection method for mobile safari (iOS)
  • no apps, too much time and risk
• Gamification: have levels (1-5), with prizes at each achievement level
  • Blurry "secret" prize, becomes clearer the more you contribute
• Don't require "sign up" at first (track users using cookies/localStorage until they are ready), then collect demographic info
• Include visual indicator of Goal for project
• Keep it simple for June, we can build a real app afterwards if we think it's useful

See voice jam etherpad for more info.

[mikehenrty]

Stickiness MVP Proposal

• experience points (voice-coins), perhaps count of how many submissions from your user
• levels gained for reaching X experience points
  1. Level one: invite to join leaderboard (must enter name)
  2. Level two: invite to join contributor list (You can optionally be listed on the contributors page)
  3. Level three: add a sentence for other people to say (you can hear them later)
  4. Level four: project stickers (requires email address, name and home address)
  5. Level five: plushy
  6. Level six: t-shirt
• progress indicator of progress towards next level
  • use blurry picture of prize that becomes more clear the closer you get

[mikehenrty]

@geroter @andrenatal @kdavis-mozilla @mbebenita, thoughts on these for levels?

[geroter]

A few principles:

1. We can afford this as it scales (i.e. we need 200K @ 5min per language) -- no more expensive than equivalent in Mech Turk
2. The incentives link to the overall narrative of the campaign (great insight from Janis)

Reactions to levels:

• our goal is # of hours vs. # of submissions ... so likely the levels should correspond?
• how many seconds is the average submission
• how many seconds/submissions is each level ... validate cost (that we can afford this at scale!)
• is there research out there on levels? are each at a new order of magnitude? or designed around some theory of contribution?
• I think we want a way to communicate with people to encourage repeat contribution (this is key) ... so either there are push notifications in the website/app (need to scope this feature), or we need email address early on

Want to add more details to your proposal @mikehenrty with the above ideas?

[geroter]

@mikehenrty I think we could reasonably get Michael Bernstein to take a look/comment on this, in addition to our UX person/agency.

Also wondering if we might be able to do a quick test of levels/incentives? Thoughts on how we could do that?

[mikehenrty]

Reactions to levels:

• our goal is # of hours vs. # of submissions ... so likely the levels should correspond?

Right, let's base it off length of submission. Let's just hope people don't try and game the system by talking slow (or fast). We can award points based on the amount of milliseconds perhaps.

• how many seconds is the average submission

Right now about 1s. But that could change with #23, so better if we don't rely on that.

• how many seconds/submissions is each level ... validate cost (that we can afford this at scale!)

Yeah, this needs to be worked out, just was doing high level stuff before. I'm not even sure physical prizes are the right approach.

• is there research out there on levels? are each at a new order of magnitude? or designed around some theory of contribution?

I have been thinking about this in terms of reward cycles, which there is some psychological research around (called reinforcement I think). In any case, that is another todo item here: put numbers to the level curve.

• I think we want a way to communicate with people to encourage repeat contribution (this is key) ... so either there are push notifications in the website/app (need to scope this feature), or we need email address early on

email addresses is more realistic than web push, unless we could borrow/rent another coder's time. the iOS app might be easier, but we would have to ask @andrenatal

[mikehenrty]

Also wondering if we might be able to do a quick test of levels/incentives? Thoughts on how we could do that?

One way is to get a bare-bones site up (ie. no branding and no flashy stuff, perhaps even using prototype tool like invision) and ask usertesting.com users their thoughts on levels/prizes.

Another approach could be a descoped MVP of the site, then we use adds to drive traffic to it and measure the drop off rate of different level approaches.

In an ideal world we could do both, but I'm not sure if we will have time for either.

[mikehenrty]

New Stickiness MVP Proposal (still rough)

Let's think about this both in terms of first time experience, and long term rewards. In the beginning, we can have a couple of quick wins that encourage registration (sign up for our contributor list, leaderboard, etc). Adding features in this early stages, like a leaderboard or comment section is an interesting progression idea that is "free." If we are going to reward anything that costs money, use mech-turk numbers to figure out how many hours of data we would need to reward that amount.

Levels

1. Level one: players start here, get bag for collecting clips (use paperclips analogy).
2. Level two: aim to achieve on first session (1-5 clips), invite to join "contributor list", (Display Name, URL, 3mail?)
3. Level three: after 20 clips, you made the leader board!
4. Level four: 10 minutes of data, you can contribute new sentences to the corpora, hear everyone who says it
5. Level five: sticker delivery

• assume mechanical turk pays $0.01 usd for 1 second => .60/minute => $3.60/hour
• If we can deliver stickers for ~$7 usd, we would want around 2 hours to match mech-turk.

7. Level six: t-shirt, using same cost analysis above we would probably want around 7 hours of data
8. Level seven: Angel, donate real money to charities like EFF (random idea)

[kdavis-mozilla]

@mikehenrty I don't know if this is the right place to address this, but how is validation of the data done? In other words, how do we know the user read the correct sentence?

[mikehenrty]

Validation is not currently done in app, so we will still have to figure that part out. That's why any cost analysis we do in comparison to mechanical turk only relates to the collection piece and not validation costs.

[kdavis-mozilla]

@mikehenrty We should seriously consider having the app self-validate, in other words incorporating validation in to use of the app.

[mikehenrty]

@mikehenrty We should seriously consider having the app self-validate, in other words incorporating validation in to use of the app.

Agreed, but my thinking is that for June it is more important to build in the "recording" functionality first, then add verifying functionality if we have time. I'll file a separate bug for that.

We don't have a large developer team, so everything we do needs to be talked about in terms of priorities. Using a marketing lens, my thinking is that donating is more important to the user flow to help explain the Common Voice concept.

[kdavis-mozilla]

@mikehenrty I'm just worried that if this works, we won't have the budget to validate by paying and won't have the community to validate after all hands due to the die down of participants. So, we'll be left with lots of data which may or may not be of use.

[mikehenrty]

@mikehenrty I'm just worried that if this works, we won't have the budget to validate by paying and won't have the community to validate after all hands due to the die down of participants. So, we'll be left with lots of data which may or may not be of use.

Totally understood. There is an opportunity to get validation eyes on our data that we would be missing. Let's discuss the priorities of the verification piece in #25. There are other things we could prioritize this ahead of, and perhaps we could get some extra help from the ET team if needed :)

[mikehenrty]

Also wondering if we might be able to do a quick test of levels/incentives? Thoughts on how we could do that?

To answer this more specifically, what I've done in the past with an rpg game was to try different exponential curves until it felt "right." We could then release it, measure retention rates, tweak the curve, release/re-measure, etc. But I think you are right to push for usertest.com @geroter, I'm just wondering where we are going to find the time.

[mikehenrty]

At this point, I think this is in Geekettez hands. We'll file additional bugs for the engineering tasks and close this one.