The Multiview function does not select the best views for the average prediction about redm

[SoftwareLiteracy]

Dear Uriah,

Thank you for your use and analysis of rEDM. I have reproduced your examples, and I think I see what's happening. I see it as a disconnect between the documentation and function behavior. Thank you for pointing this out!

I believe the function is properly reporting the top multiview combinations, the disconnect comes from the value of rho that is reported for each view. More precisely, the value of rho that is used to select the top views may not be the same as that used in the final evaluation of the view, which is reported in the View output table.

This is because the default behavior is to select the top views based on in-sample library and prediction forecast. That is, the default sets pred = lib when evaluating/selecting the top views. I believe this is what is described in the Ye paper. To be pedantic, if the user specifies lib = "1 10" pred = "11 20", then the function evaluates the views with lib = "1 10" pred = "1 10", in-sample prediction, but the final output rho is forecast using the specified pred = "11 20".

You rightly say: why the disconnect? I don't have a good answer. We need to think about this and how to address it. I suspect it is a good thing to have this flexibility, that the documentation/example should be clarified.

In fact, this disconnect (imo) is one reason the trainLib argument was added to Multiview. When True (default), in-sample prediction sets are used in the selection of the top views, as described above. When trainLib = False, then the multiview evaluations are performed according to the specified lib and pred arguments. This allows one to have multiview select views based on out-of-sample predictions, rather than purely in-sample predictions.

To address/affirm this we can use the default trainLib = True and set lib = pred, ala:

> Multiview( dataFrame = block_3sp, lib = "1 99", pred = "1 99", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = TRUE ) $ View
  col_1 col_2 col_3   name_1   name_2   name_3    rho    MAE   RMSE
1     1     2     7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8875 0.3303 0.4302
2     1     2     6 x_t(t-0) x_t(t-1) y_t(t-2) 0.8834 0.3343 0.4385
3     1     2     3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8692 0.3329 0.4635

which yields the same result as trainLib = FALSE if lib = pred`

> Multiview( dataFrame = block_3sp, lib = "1 99", pred = "1 99", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = FALSE ) $ View
  col_1 col_2 col_3   name_1   name_2   name_3    rho    MAE   RMSE
1     1     2     7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8875 0.3303 0.4302
2     1     2     6 x_t(t-0) x_t(t-1) y_t(t-2) 0.8834 0.3343 0.4385
3     1     2     3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8692 0.3329 0.4635

Both of these results select the same top views as the original example, which used default trainLib = TRUE, thereby evaluated the views on lib = pred = 1:99, but reported final rho using different lib = "1 99" and pred = "105 190":

> Multiview( dataFrame = block_3sp, lib = "1 99", pred = "105 190", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = TRUE ) $ View
  col_1 col_2 col_3   name_1   name_2   name_3    rho    MAE   RMSE
1     1     2     7 x_t(t-0) x_t(t-1) z_t(t-0) 0.9208 0.2485 0.3164
2     1     2     6 x_t(t-0) x_t(t-1) y_t(t-2) 0.8677 0.3294 0.4113
3     1     2     3 x_t(t-0) x_t(t-1) x_t(t-2) 0.9319 0.2277 0.2934

As long as lib == pred, the reported values of rho in the output table will reflect the values used in view selection:

 Multiview( dataFrame = block_3sp, lib = "20 100", pred = "20 100", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = TRUE ) $ View
  col_1 col_2 col_3   name_1   name_2   name_3    rho    MAE   RMSE
1     1     2     3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8820 0.3104 0.4226
2     1     2     7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8749 0.3309 0.4306
3     1     3     7 x_t(t-0) x_t(t-2) z_t(t-0) 0.8682 0.3363 0.4411
> 
> 
> Multiview( dataFrame = block_3sp, lib = "20 100", pred = "20 100", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 7, trainLib = TRUE ) $ View
  col_1 col_2 col_3   name_1   name_2   name_3    rho    MAE   RMSE
1     1     2     3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8820 0.3104 0.4226
2     1     2     7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8749 0.3309 0.4306
3     1     3     7 x_t(t-0) x_t(t-2) z_t(t-0) 0.8682 0.3363 0.4411
4     1     2     9 x_t(t-0) x_t(t-1) z_t(t-2) 0.8525 0.3514 0.4645
5     1     2     8 x_t(t-0) x_t(t-1) z_t(t-1) 0.8518 0.3362 0.4657
6     1     2     4 x_t(t-0) x_t(t-1) y_t(t-0) 0.8394 0.3485 0.4896
7     1     4     7 x_t(t-0) y_t(t-0) z_t(t-0) 0.8365 0.3749 0.4900
> 

I agree that this situation is not stated in the documents.

[duriah]

Thank you for your detailed answer!

The matter is now pretty clear. I think the disconnect makes the function a bit less transparent, so it might be a good idea to store the in-sample rho-values of the views somewhere accessible or to mention this behaviour in the documentation. But it is by no means anything urgent.

Thanks!
Uriah