Example for online evaluation about ax

[kkashin]

Thanks for the feedback @danicat! You're totally right -- this is something we definitely need to do and are planning to add. The system definitely supports this use case (in fact online / field experimentation is where this whole effort started)! There's some background of how Ax is used for product optimization via online evaluation here.

Just to clarify, you can think of Ax broadly as the management & algorithm layer on top of a system that actually does the randomization or routing of users to A or B (such as PlanOut), so Ax doesn't actually provide the randomization / routing logic itself. We are planning to provide an integration / comprehensive examples of using Ax with PlanOut (https://github.com/facebook/planout). Would that help with providing clearer examples?

There's a bit of background on the design of Ax (formerly AE) for field experimentation in Section 2 in this workshop paper.

If there are other experimentation systems beyond PlanOut that you think it would be valuable to consider integrations with, I'd be really curious to hear.

[lena-kashtelyan]

@vmysla, you were exactly right to expect to be able to obtain a trial from the Service API and log back its results without creating an explicit evaluation function. The evaluation function in the tutorial is just there for the example –– as you might notice, it's never actually passed to AxClient, it's just used to generate the trial results before completing them: ax.complete_trial(trial_index=trial_index, raw_data=evaluate(parameters)). So instead of doing raw_data=evaluate(parameters), you can obtain the trial evaluation data in whichever way is appropriate for your setup. Does that help / make sense?

[ldworkin]

@irvifa This should be doable without any additional work built into Ax, but we would definitely need to provide you with an example! I'll discuss with the team and try to prioritize getting you something in the next two weeks (so by end of January).

[sdsingh]

Hi all, I have a draft of the PlanOut Tutorial available here: planout_tutorial.ipynb.zip.
This will be checked in and released on the site with the next version of Ax (in a week or two).

Please let me know if this answers your questions :)

[ldworkin]

Have you taken a look at this tutorial? https://ax.dev/tutorials/building_blocks.html

This is the one most relevant to you, I think. The parts that you'll want to pay attention to are defining your custom Runner and Metric.

What data needs to be
persisted to the db so that Ax knows what happened to previous users,etc.

You can save data to the database however you want! Then, you'll want to write a custom Metric class that defines how to fetch that data, and converts it into a dataframe that Ax can parse. See an example here: https://ax.dev/tutorials/building_blocks.html#4.-Define-an-optimization-config-with-custom-metrics. In your case, your metric should load from your database and then convert the data to this format (one row for each arm).

Similarly, your custom Runner is what will actually launch the experiment into production. E.g. you could write some values to a database that Flask can read from in order to decide what arms it needs to show.

Let me know if this helps at all!

[danicat]

Hi @kkashin! Yes, I think providing examples with Ax working in combination with PlanOut would be very interesting to see, as in the current state I still find a bit hard to understand where Ax would fit in the big picture (architecture-wise).

I'll have a look at the reference materials you've listed. Thank you!

[vmysla]

@lena-kashtelyan, thank you! Yes, this is very helpful. Will give it a shot

[sdsingh]

Hi @sandys! Short answer: No, Ax and PlanOut would always work together, and Ax will not subsume PlanOut.

You can think of Ax as a tool which manages experiment metadata and designs new experiments. Ax does not actually implement an experimentation framework--something which handles, at a minimum, randomized user assignment. As a result, Ax works together with experimentation frameworks, via the "Runner" abstraction.

PlanOut is an experimentation framework, and it provides a way to implement the experiments designed by Ax. To interact with PlanOut, Ax would use a specific "PlanOut Runner", to create PlanOut scripts which handle the assignment suggested by Ax.

I am currently working on this runner, as well as a tutorial for using it.

[sandys]

Hi Shaun Thanks for replying. That would be very useful - we are looking to build internal api endpoints that design experiments and run them on flask. Ax + planout looks to be the stack that we would like to use.

…

On Mon, 10 Feb, 2020, 22:27 Shaun Singh, ***@***.***> wrote: Hi @sandys <https://github.com/sandys>! Short answer: No, Ax and PlanOut would always work together, and Ax will not subsume PlanOut. You can think of Ax as a tool which manages experiment metadata and designs new experiments. Ax *does not* actually implement an experimentation framework--something which handles, at a minimum, randomized user assignment. As a result, Ax works *together* with experimentation frameworks, via the "Runner" abstraction. PlanOut is an experimentation framework, and it provides a way to implement the experiments designed by Ax. To interact with PlanOut, Ax would use a specific "PlanOut Runner", to create PlanOut scripts which handle the assignment suggested by Ax. I am currently working on this runner, as well as a tutorial for using it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#77?email_source=notifications&email_token=AAASYU3SLBA34KUR2KHHN7DRCGBPVA5CNFSM4HLQ2RZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELJIQMA#issuecomment-584222768>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASYU4R6XDT6UVZ25C3TGTRCGBPVANCNFSM4HLQ2RZA> .

[ldworkin]

Hi @sandys !

@sdsingh , can you maybe re-upload the planout tutorial? It's not clear to me how to find it.

@sandys maybe once you see that, it will at least unblock you?

To clarify, Ax is really agnostic to the backend you use. You can deploy your online trials however you want. The 'Runner' abstraction in Ax is what controls the deployment, and that's entirely customizable. Is there anything in particular you're confused about or that I could help clarify?

[kkashin]

Great - the open source PlanOut integration will likely take a couple of months for us to put together, but in the meantime, we'll get better documentation up in the next 1-2 weeks that explains how Ax can be used for A/B testing (including how it fits in, architecture-wise). The paper also explains this, so let me know if you have any questions as you look at those materials.

[abronner]

Great - the open source PlanOut integration will likely take a couple of months for us to put together, but in the meantime, we'll get better documentation up in the next 1-2 weeks that explains how Ax can be used for A/B testing (including how it fits in, architecture-wise). The paper also explains this, so let me know if you have any questions as you look at those materials.

@kkashin any update / ETA on the open source Ax-Planout integration? looking forward to it .. thanks!

[vmysla]

Does this explain why all examples from ax.dev tutorials do have evaluation function?

I was under the impression that Ax can be used with its Service API to request for hands allocation and report back to the service after user converted.

We were planning to use Ax in our pricing experiments, but as far as I can understand, it looks like I already should have evaluation function that just returns past outcomes for any given hand allocations.

Am I understanding this correctly?

[lena-kashtelyan]

@vmysla, may I ask what you mean when you say that all examples in our tutorials do have evaluation function? For example, through the Service API, you can request trials (which contain arms, aka points, which have parameterizations), then evaluate their parameterizations however you need to, then log the results back to Ax by 'completing' the trials. Does that functionality fit your purpose?

it looks like I already should have evaluation function that just returns past outcomes for any given hand allocations.

Not entirely sure what you meant there –– could you elaborate? A code snippet from one of our tutorials that shows what you are referring to by 'evaluation function that just returns past outcomes' would really help me understand and hopefully give an informative answer : )

[vmysla]

@lena-kashtelyan, thanks for help me to figure this out.
I'm looking at this example of the Service API usage:
https://ax.dev/tutorials/gpei_hartmann_service.html#3.-Define-how-to-evaluate-trials
Step #3 has an evaluation function that has contains a formula:

import numpy as np
def evaluate(parameters):
    x = np.array([parameters.get(f"x{i+1}") for i in range(6)])
    # In our case, standard error is 0, since we are computing a synthetic function.
    return {"hartmann6": (hartmann6(x), 0.0), "l2norm": (np.sqrt((x ** 2).sum()), 0.0)}

another example:

evaluation_function=lambda p: (p["x1"] + 2*p["x2"] - 7)**2 + (2*p["x1"] + p["x2"] - 5)**2,

When I expected that instead of having predefined function, I will call Service API and let it know about the outcome of the given trial once I know if user converted or not (1/0).

[irvifa]

Any exact timeline on this? @kkashin

[kkashin]

@irvifa, @abronner - no updates on the timeline for Ax-PlanOut integration yet unfortunately. The team is focused on making improvements to existing functionality (including better benchmarking suite, improvements to ServiceAPI, and adding cost-aware algorithms). I'll try to have an update for you on timing within a couple of weeks.

[yuriy-muzychuk]

Hi @lena-kashtelyan ! We tried experimenting with the Service API approach for the online case but received some unexpected results. The experiment is very simple: a customer comes to our site, using the get_next_trial() method, we pick one of the 3 values for some parameter ('option1', 'option2', 'option3') and save it together with the trial_id per customer. Then with a 60% chance we convert customers with option1, 30% - with option2 and 10% - with option3. By convert I mean sending the complete_trial() request with trial_id corresponding to the customer and value of 1. We repeat this process for several iterations with 50 customers in each iteration.
We expected the distribution of hands to change from 33% to something that correlates with the conversion rate over time (iteration to iteration). But it keeps the same all the time (33%). Should we somehow manually tell the AxClient to reevaluate the hands at some point?

[lena-kashtelyan]

@yuriy-muzychuk, hi! This seems like a rather separate issue from the one discussed here (examples for online evaluation). Would you mind opening it separately and adding a code snippet showing what exactly you are doing? Thanks!

[irvifa]

@kkashin It's been a while since then. I understand that, however still curious whether it can be done without any additional layer in the Ax (basically you guys only provide an example for this) or there should be another additional layer of abstraction in which we can define the input from randomization tools that we want?

[sandys]

hi guys,
we are extremely interested in the planout vs ax question here. Will Ax subsume much of the functionality in Planout ?
We are really looking to adopt one or the other.

[irvifa]

@ldworkin and @sdsingh Thank you for your answer. I believe it’s already in your roadmap since the last time I talk to Lili about the improvement through the email. Looking forward for the tutorial. 🙂

[sandys]

hi guys,
would it be possible to have an example of an online trial without using planout. Maybe just on top of Flask for simplicity.
I think a lot of people have similar questions - based on the activity in this bug over last 18 months.

[sandys]

Would it be possible to show a non-planout example. I do understand there would be more boilerplate. But I'm wondering if planout is mandatory. Is it not possible to do this with a bit of coding ourselves and Ax?

…

On Mon, 31 Aug, 2020, 22:47 Lili Dworkin, ***@***.***> wrote: Hi @sandys <https://github.com/sandys> ! @sdsingh <https://github.com/sdsingh> , can you maybe re-upload the planout tutorial? It's not clear to me how to find it. @sandys <https://github.com/sandys> maybe once you see that, it will at least unblock you? To clarify, Ax is really agnostic to the backend you use. You can deploy your online trials however you want. The 'Runner' abstraction in Ax is what controls the deployment, and that's entirely customizable. Is there anything in particular you're confused about or that I could help clarify? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#77 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASYU7KJUCYRT7PZ3OHZI3SDPLITANCNFSM4HLQ2RZA> .

[ldworkin]

It definitely is possible! Can you tell us a little more about your setup? How are you planning to randomize users among different options?

As per @sdsingh 's comment here:

You can think of Ax as a tool which manages experiment metadata and designs new experiments. Ax does not actually implement an experimentation framework--something which handles, at a minimum, randomized user assignment. As a result, Ax works together with experimentation frameworks, via the "Runner" abstraction.

If you tell us more about how you're going to handle the experimentation framework part, I can help you understand how Ax would work together with it.

[sandys]

Thanks for asking. It's a simply model api (running on flask) with multiple models. To be honest, we are still trying different ways of randomizing users. But what we are sold on is Ax - as the right framework to do it. But I'm not able to figure out how to do it. What data needs to be persisted to the db so that Ax knows what happened to previous users,etc. So it's really the API usage part (with aspects of db) that's the confusing part vs the conceptual basis of Ax. In fact all the early questions about get_next_trial ,etc are all about the api usage. Would help if you can even create a very trivial online evaluation example ... hopefully without pulling in Planout,etc.

…

On Mon, 31 Aug, 2020, 23:02 Lili Dworkin, ***@***.***> wrote: It definitely is possible! Can you tell us a little more about your setup? How are you planning to randomize users among different options? As per @sdsingh <https://github.com/sdsingh> 's comment here: You can think of Ax as a tool which manages experiment metadata and designs new experiments. Ax does not actually implement an experimentation framework--something which handles, at a minimum, randomized user assignment. As a result, Ax works together with experimentation frameworks, via the "Runner" abstraction. If you tell us more about how you're going to handle the experimentation framework part, I can help you understand how Ax would work together with it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#77 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASYU27GTV3C34LU75CV3TSDPNCPANCNFSM4HLQ2RZA> .

[ldworkin]

These docs might also be helpful:

https://ax.dev/docs/trial-evaluation.html#adding-your-own-runner
https://ax.dev/docs/data.html

[lena-kashtelyan]

With docs @ldworkin added above we'll consider this answered for now, but feel free to reopen!