Unable to run the budget allocator as it gives me warning about coefficients. How can we handle this? Thanks! about robyn

[gufengzhou]

hey thanks for flagging this! There was indeed a bug in the allocator and I've just pushed a fix. Would you please try out the update and see if the error still occurs? Looking at your waterfall plot, some media channels are having no effect. When this happens, channels with 0 effect will not be considered in the budget allocator, because, well, they have 0 effect and there's nothing to optimised.

[gufengzhou]

Good that the fix is working:)

The waterfall plot is not (only) the coefficient plot, but the "decomposition", or to be precise, it's the variable multiplied with the coef. It tells you "var 1 explains x% of your conversations/sales".

The decay plot just shows the selected decay rate of each media channel. When using geometric adstock, it's exactly the decay rate. When using weibull adstock, it's the "average" decay Rate, or the "geometric equivalent for weibull". Hope it helps.

[gufengzhou]

Decay rate is always based on your time grain. If you have daily data, then decay rate of 40% means day 1 has 100 imps, day 2 has 40 imps carried over from day 1, day 3 has 16 imps carried over from day 1, etc.

Regarding using old ModID, well yes, you can use the exact hyperparameter of that old model ID as the set local bound values (to fix all hypPars so to say), you will get one single output from f.robyn, then run the allocator on that (ModID will be 1_1_1 because there's just one model).

Curious to know how you think about the result and over performance of Robyn:)

[Tony-009]

I see! Now it's coming all together in conjunction with information about never grad.
I reckon this is what was required to put my mind at ease. Thanks alot for the time and effort you have put in elaborating on the topics.
We can close this issue 😀
Cheers!

[Tony-009]

Hey Thanks @gufengzhou . I was poking around and I saw that the bug was fixed yesterday and It seems its working now. Appreciate the work and quick response on this! :-)
I had another question to you regarding the aforementioned plot- How would you explain that ? Is it a coefficient plot and its effect on the target variables ? So, can I say that Tv_preemtible had 3.7% impact on the target variable or "conversion? What does "tsd"means?

Would you say for the attached image that specified channels have this much% decay as they stay in the market?
Sorry! I was just confused on various parts so thought of clarifying it here.
Thanks again!

[Tony-009]

Brilliant! This does help alot. So, for decay rate % for geometric adstock is for a the associated time period whether its months/week or year? Also, is there any way to use a previous model result(such as previous ModID) for the budget allocator. I am not sure its possible but wanna make sure.
Thanks!
Cheers!:-)

[Tony-009]

That explains it. I will give it a shot in running the old model as you mentioned. I am glad it has the functionality😀
As I have embedded more and more into robyn and trying to understand the functions created, the idea behind the budget allocator is quite neat. I think its a great MMM tool employing sophisticated never grad and other tools.
I reckon my confusion arises from the results, I am aware of the pareto optimal result" one cannot be better off without making other worst" but how do you figure in terms of NRMSE and DECOMP,SSD? I know RMSE and MAPE and it's impact but how do you look for good NRMSE?In terms of Business logic?
In addition to that, my MAPE is NA? I have tried wrangling, formatting the data, tweaking code but don't know why it's not giving me that.
Furthermore, is there any way to include variables considered exposure without any set data?
Robyn is a great tool and I reckon with our feedback, upgrades it has the potential to be a industry wide norm.
Apologies for long questions. I am actually excited to know more and more about robyn because then I can implement it in all my projects.
Thanks again 😀

[gufengzhou]

DECOMP.RSSD is short for "decomposition root sum of squared distance", a metric we "invented" to account for business logic. The intuition is this: assuming you're spending 90% on TV and 10% on FB. If you get 10% effect for TV and 90% for FB, you'd probably not believe this result, no matter how low the model error (NRMSE) is. If you get 80% TV and 20% FB as effect share, it'll more "realistic". This is where the logic is from: minimising the distance between share of spend and share of effect.

Regarding the question "how do you look for good NRMSE", the job is called multi-objective optimisation and done by Nevergrad. It's how evolutionary algorithm works. We have 2 objective function here: NRMSE and DECOMP.RSSD (when calibrating with experiment, MAPE.LIFT will be added as a third objective). With every new iteration (or generation), the mutation of hyperparameters will always based on the best candidate from the last iteration, so that the learning can be carried over from iteration to iteration. "Best candidate" is defined by the lower/better NRMSE or DECOMP.RSSD. For more details you can check out Nevergrad .

At last, MAPE is NA because you're not calibrating the model. This is expected. Hope it helps:)