Comments (2)
Dear Uriah,
Thank you for your use and analysis of rEDM. I have reproduced your examples, and I think I see what's happening. I see it as a disconnect between the documentation and function behavior. Thank you for pointing this out!
I believe the function is properly reporting the top multiview combinations, the disconnect comes from the value of rho that is reported for each view. More precisely, the value of rho that is used to select the top views may not be the same as that used in the final evaluation of the view, which is reported in the View output table.
This is because the default behavior is to select the top views based on in-sample library and prediction forecast. That is, the default sets pred = lib
when evaluating/selecting the top views. I believe this is what is described in the Ye paper. To be pedantic, if the user specifies lib = "1 10" pred = "11 20"
, then the function evaluates the views with lib = "1 10" pred = "1 10"
, in-sample prediction, but the final output rho is forecast using the specified pred = "11 20"
.
You rightly say: why the disconnect? I don't have a good answer. We need to think about this and how to address it. I suspect it is a good thing to have this flexibility, that the documentation/example should be clarified.
In fact, this disconnect (imo) is one reason the trainLib
argument was added to Multiview
. When True
(default), in-sample prediction sets are used in the selection of the top views, as described above. When trainLib = False
, then the multiview evaluations are performed according to the specified lib
and pred
arguments. This allows one to have multiview select views based on out-of-sample predictions, rather than purely in-sample predictions.
To address/affirm this we can use the default trainLib = True
and set lib = pred
, ala:
> Multiview( dataFrame = block_3sp, lib = "1 99", pred = "1 99", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = TRUE ) $ View
col_1 col_2 col_3 name_1 name_2 name_3 rho MAE RMSE
1 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8875 0.3303 0.4302
2 1 2 6 x_t(t-0) x_t(t-1) y_t(t-2) 0.8834 0.3343 0.4385
3 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8692 0.3329 0.4635
which yields the same result as trainLib = FALSE if
lib = pred`
> Multiview( dataFrame = block_3sp, lib = "1 99", pred = "1 99", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = FALSE ) $ View
col_1 col_2 col_3 name_1 name_2 name_3 rho MAE RMSE
1 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8875 0.3303 0.4302
2 1 2 6 x_t(t-0) x_t(t-1) y_t(t-2) 0.8834 0.3343 0.4385
3 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8692 0.3329 0.4635
Both of these results select the same top views as the original example, which used default trainLib = TRUE, thereby evaluated the views on lib = pred = 1:99
, but reported final rho using different lib = "1 99"
and pred = "105 190"
:
> Multiview( dataFrame = block_3sp, lib = "1 99", pred = "105 190", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = TRUE ) $ View
col_1 col_2 col_3 name_1 name_2 name_3 rho MAE RMSE
1 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.9208 0.2485 0.3164
2 1 2 6 x_t(t-0) x_t(t-1) y_t(t-2) 0.8677 0.3294 0.4113
3 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.9319 0.2277 0.2934
As long as lib == pred
, the reported values of rho in the output table will reflect the values used in view selection:
Multiview( dataFrame = block_3sp, lib = "20 100", pred = "20 100", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = TRUE ) $ View
col_1 col_2 col_3 name_1 name_2 name_3 rho MAE RMSE
1 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8820 0.3104 0.4226
2 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8749 0.3309 0.4306
3 1 3 7 x_t(t-0) x_t(t-2) z_t(t-0) 0.8682 0.3363 0.4411
>
>
> Multiview( dataFrame = block_3sp, lib = "20 100", pred = "20 100", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 7, trainLib = TRUE ) $ View
col_1 col_2 col_3 name_1 name_2 name_3 rho MAE RMSE
1 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8820 0.3104 0.4226
2 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8749 0.3309 0.4306
3 1 3 7 x_t(t-0) x_t(t-2) z_t(t-0) 0.8682 0.3363 0.4411
4 1 2 9 x_t(t-0) x_t(t-1) z_t(t-2) 0.8525 0.3514 0.4645
5 1 2 8 x_t(t-0) x_t(t-1) z_t(t-1) 0.8518 0.3362 0.4657
6 1 2 4 x_t(t-0) x_t(t-1) y_t(t-0) 0.8394 0.3485 0.4896
7 1 4 7 x_t(t-0) y_t(t-0) z_t(t-0) 0.8365 0.3749 0.4900
>
I agree that this situation is not stated in the documents.
from redm.
Thank you for your detailed answer!
The matter is now pretty clear. I think the disconnect makes the function a bit less transparent, so it might be a good idea to store the in-sample rho-values of the views somewhere accessible or to mention this behaviour in the documentation. But it is by no means anything urgent.
Thanks!
Uriah
from redm.
Related Issues (20)
- Multiview() chooses the maximum lag based on the number of predictor columns used, regardless of E. HOT 2
- Prediction not NA with disjoint/segmented pred HOT 1
- setting knn < E+1 HOT 7
- dgelss failed HOT 5
- make_block producing zeros instead of NaN HOT 4
- Significance of converegcne HOT 3
- When should different surrogate generation functions be used? HOT 1
- error in installation
- Example of allowed libSizes for tau or negative tp HOT 4
- Tp < -1 gives error HOT 3
- Different sampling between legacy and current version? HOT 4
- can not library the "rEDM" HOT 22
- WARNING: ComputeError(): nan predictions found error not computed. HOT 11
- CCM Parallel Processing
- Prediction of single time points HOT 1
- Inverted name for the output generated by block_lnlp function HOT 13
- R 4.2.0 Character Encoding on Windows: Can not load rEDM HOT 5
- version 1.15 dealing with NaN HOT 4
- Questions about 'Nonlinearity' and the significance of ρ under the largest L. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redm.