Validation procedure of Robyn about robyn

[gufengzhou]

Hi sorry for the late reply, to your question of "budget allocator and initial model not matching", please check this answer from me on another similar issue.

Regarding validation, as explained above, you'll get the same spend share as the initial model by doing this. I just picked a random model using the simulated data :

library(dplyr)
AllocatorCollect1 <- robyn_allocator(
  InputCollect = InputCollect,
  OutputCollect = OutputCollect,
  select_model = select_model,
  date_range = "all", # Default last month as initial period
  channel_constr_low = 0.7,
  channel_constr_up = c(1.2, 1.5, 1.5, 1.5, 1.5),
  scenario = "max_response",
  export = create_files
)
OutputCollect$xDecompAgg %>% filter(solID == select_model & !is.na(spend_share)) %>% select(rn, spend_share, effect_share) %>% arrange(rn)
AllocatorCollect1$dt_optimOut %>% select(channels, initSpendShare, initResponseUnitShare)

As you can see in the result, spend share are the same after setting date_range to all, because the initial model onepager considers all dates, while the allocator contains 4 weeks by default.

And yes, the effect share is different, which is also explained in the linked comment above. For initial model, the effect share is just the % of all weekly avg. effect, or simply the historical share. For allocator, I need to use the weekly avg. spend to simulate the weekly avg. carryover and then simulate the weekly avg. response. It's a simulation process, NOT the historical share anymore.

[gufengzhou]

Just found a bug and pushed a fix on the response function. Now you can validate between initial model, robyn_response & robyn_allocator as followed:

## comparing responses
last_period <- 1
media_sorted <- sort(InputCollect$paid_media_spends)

## get last period response from initial model
val_response_a <- OutputCollect$xDecompVecCollect %>%
  filter(solID == select_model) %>%
  select(ds, media_sorted) %>%
  tail(last_period)

## get last period response from robyn_response
val_response_b <- list()
for (i in seq_along(media_sorted)){
  Response <- robyn_response(
    InputCollect = InputCollect,
    OutputCollect = OutputCollect,
    select_model = select_model,
    metric_name = media_sorted[i],
    date_range = paste0("last_", last_period)
  )
  val_response_b["ds"] <- data.frame(ds = Response$date)
  val_response_b[media_sorted[i]] <- data.frame(response = Response$response_total)

}
val_response_b <- bind_cols(val_response_b)

## get last period response from robyn_allocator
AllocatorCollect1 <- robyn_allocator(
  InputCollect = InputCollect,
  OutputCollect = OutputCollect,
  select_model = select_model,
  date_range = paste0("last_", last_period), # Default last month as initial period
  # total_budget = NULL, # When NULL, default is total spend in date_range
  channel_constr_low = 0.7,
  channel_constr_up = c(1.2, 1.5, 1.5, 1.5, 1.5),
  # channel_constr_multiplier = 3,
  scenario = "max_response",
  export = create_files
)
val_response_c <- AllocatorCollect1$dt_optimOut %>% select(date_min, date_max, initResponseUnit)

val_response_a
val_response_b
val_response_c

When doing last_period <- 1, you can see they all align:

When doing last_period <- 3, initial model & response function align and outputs the historical response for every period, but allocator does runs a simulation behind and thus will use avg. carryover of the last 3 period to determine the result.

[tgtod002]

@gufengzhou , So given the above example looking at the results for facebook_s. to get the expected total response for the alloted period will be:

<style> </style>

	initresponseUnit	Periods	TOTAL
when period 1	106625.5	1	106625.5
when Period last 3	99103.13	3	297309.4

[tgtod002]

I ran the validation using my data...Trying to understand the results.

Val_response_a:

<style> </style>

48	11/27/2022	$ 161,919.87	$      2,361.20
49	12/4/2022	$ 127,123.26	$           58.60
50	12/11/2022	$ 269,868.51	$              0.90
51	12/18/2022	$ 240,025.21	$              0.01
52	12/25/2022	$ 242,438.92	$              0.00

Val_response_b"

<style> </style>

48	11/27/2022	$    161,919.87	$         2,361.20
49	12/4/2022	$    127,123.26	$              58.60
50	12/11/2022	$    269,868.51	$                0.90
51	12/18/2022	$    240,025.21	$                0.01
52	12/25/2022	$    242,438.92	$                0.00

val_response_c:

<style> </style>

	date_min	date_max	initResponseUnit
CSI_SPEND	1/2/2022	12/25/2022	4241.296224
WRAP_SPEND	1/2/2022	12/25/2022	402.289325

why is val_response_c so low. Does it divide by 52? (I ran the MMM for 52 weeks)

Thanks again for all your help on this.

[gufengzhou]

The response level is calculated by a given spend on a given historical carryover/adstock. The given spend for response_c is the avg spend of the last 52 weeks, and the given carryover is the avg carryover of the 52 weeks.

Without looking into your data, the low response_c is because of the relatively high avg carryover and/or relatively low avg spend, compared to the actual daily levels. it's possible.

In your case, I suggest you to experiment with different date_range to find out the appropriate levels. We recommend using rather more recent periods instead of too far backwards to reflect your recent adstocking and saturation behaviour.

[tgtod002]

@gufengzhou, shouldn't the response C, init_response_unit = response(immediate) + carryover response?

My client's creates budget on a 52 week window. Any suggestions to be able to run allocations on this would be welcome?