Comments (11)
Regarding the question of how to multiprocess with CCM, a set of R functions that do this in various ways using foreach %dopar%
and clusterApply
are shown here.
Please note that since CCM
is multithreaded computing both forward and reverse mappings, the number of cores applied to these functions should be less than 1/2 the number of cores available.
Also note that mclapply
should not be used with multithreaded functions/applications such as CCM Multiview EmbedDimension PredictNonlinear PredictInterval
.
from redm.
Thank you for the report.
As we have discussed via email, these are warnings and error-traps since non-sequential data violate a presumption of Takens time-delay embedding theorem (Simplex
, CCM
etc), and, the LAPACK routine dgelss
used to solve the SMap
linear system does not allow NA
.
Accordingly, these are not software bugs, but input data inadequacies.
Some comments on the issue may perhaps be useful:
The first step in EDM (unless one is using a multivariate, non time-delay embedded state-space) is to "recreate" the state-space with time-delay embedding: Takens theorem. It is presumed that data are contiguous in time, as the time-delay operation shifts the time series data vectors successively by τ. I am unsure of the ramifications of violating this presumption, Clairty on this issue is most welcome.
In functions that use Tp != 0
, time to prediction, we are also presuming that the "step" in the state-space between points equates to one time step.
However, if one is not using a time delay embedding but a multivariate data set (embedded = TRUE
), AND, Tp = 0
, then missing data no not violate any specific presumptions, other than the general one that the state space is a complete representation of the system (and presuming the missing data are sparse).
Even so, the Simplex
based routines (CCM
, Multiview
, EmbedDimension
etc) do not prevent NA
, or, use the time vector (dataFrame
first column) for state-space computations, just for tracking observation vectors to report the correct observation time in the predictions. Therefore, NA
are not forbidden, as is easy to demonstrate. A prediction that uses a state-space vector with NA is returned as NA. If statistics are then computed on a prediction vector with NA
: WARNING: ComputeError(): nan predictions found error not computed.
It is up to the user to be aware of the limitations of the data and violations of the underlying presumptions.
I often use statistical filling or interpolation to fill data gaps.
Please note that in CCM
, replacement = TRUE
is not recommended. This introduces a bias since it has the potential to reuse degenerate states in the library used to compute the Simplex result for a specific sample evaluation.
If the number of NA
is small, and only produce a few NA
in the Simplex
output, perhaps you can process CCM
statistics by ignoring NA
in the Predictions. The code simply reports it was unable to compute the statistics since there were NA
in the Predictions, thereby a WARNING
. The Predictions can be returned using includeData = TRUE
in CCM
, in which case the returned object is a list with CCM1_Predictions
, CCM2_Predictions
containing the result of each sample Simplex
prediction. Although, the list is not named to specify what library size was used. However, you can do separate runs with a single libSizes = X
argument for each library size.
Perhaps investigation comparing synthetic time series with/without gaps can provide bounds on the differences.
from redm.
As I tested it, the warning it seems to not have to do with start libSize
, Its is unclear why I am getting this warning and how to solve it, I ran several start points for the libraries sizes and has not worked, I every time the warning shows the ccm
function returns as results 0 all over the library size
from redm.
Thanks for the issue submission.
Please report the following:
library(rEDM)
sessionInfo()
Please give an example of the offending function call.
from redm.
Thanks for the prompt response! I am very busy those days and I couldn't look back on my code, thanks for the help with multiprocessing implementation code too! I promise sometime next week I'll take a look and respond you back.
from redm.
Hi, I am using rEDM package in R but there are major gaps in our data. Clearly, if there are NAs then the rEDM functions (e.g., EmbedDimension, PredictNonLinear, CCM) are not running or showing errors in prediction.
ccm_out_tau <- CCM(dataFrame = df3, E = 6, Tp = 0, columns = "ET", target = "PET", tau = tau,
libSizes = c(10, 962, 5), sample = 100, replacement = F, showPlot = TRUE)
WARNING: ComputeError(): nan predictions found error not computed
theta.rho1 <- PredictNonlinear( dataFrame=df3, E=6,lib="1 962",theta = "",
pred="1 962", columns="ET", target="ET", showPlot = TRUE)
Error in RtoCpp_PredictNonlinear(pathIn, dataFile, dataFrame, pathOut, :
Lapack_SVD(): dgelss failed.
I'm wondering if you can help me solve this issue.
Thank you.
from redm.
hi there, another question regarding the tp
parameter in ccm
function, I made use of rEDM package some years ago and that time the ccm
function accepted tp < 0
as entries, now it does not accept, why? This has to do with the new default of the function that runs the maps in both directions simultaneously? How can I ask for a map say X:Y where the target time for prediction it is negative? This is an attempt to investigate how delayed can be the causal relations.
When I try with tp = -1
I get the following message:
Error in RtoCpp_CCM(pathIn, dataFile, dataFrame, pathOut, predictFile, : CrossMap(): Tp = -1 is inconsistent with maximum libSize = 523
from redm.
This is an error trap expressing that the requested library size exceeds the size of the library that was created.
> library( rEDM )
> sessionInfo()
other attached packages:
[1] rEDM_1.10.2
> df = Lorenz5D
> CCM( dataFrame = df, E = 5, Tp = -1, columns = 'V1', target = 'V5', libSizes = "50 995", sample = 10 )
LibSize V1:V5 V5:V1
1 50 0.6254 0.8019
2 995 0.8468 0.9141
>
> CCM( dataFrame = df, E = 5, Tp = -1, columns = 'V1', target = 'V5', libSizes = "50 1000", sample = 10 )
Error in RtoCpp_CCM(pathIn, dataFile, dataFrame, pathOut, predictFile, :
CrossMap(): Tp = -1 is inconsistent with maximum libSize = 1000
from redm.
How can I know an optimal LibrarySize
to rapidly determine delayed effects when test tp<0
? Here in my test, I know I own a working example for you, I was using the whole series length for the libSizes
, when I substracted 2 from it, it worked for tp<0
, there is any way to estimate this optimal size, to work for tp>0
and tp<0
?
from redm.
This depends on whether Tp is positive or negative, and tau, E, and number of observations.
Something like this should work where N is the number of observation rows
A = abs( tau ) * (E-1)
B = (Tp + 1)
Tp < 0 : libSize_max = N - A + B
Tp > 0 : libSize_max = N - A
from redm.
Thanks for the formulas!
To me the formula that worked was little different:
Tp<0: libSize_max = N - A + B -2
Tp>0: libSize_max = N - A
from redm.
Related Issues (20)
- Multiview() chooses the maximum lag based on the number of predictor columns used, regardless of E. HOT 2
- Prediction not NA with disjoint/segmented pred HOT 1
- setting knn < E+1 HOT 7
- dgelss failed HOT 5
- make_block producing zeros instead of NaN HOT 4
- Significance of converegcne HOT 3
- When should different surrogate generation functions be used? HOT 1
- error in installation
- Example of allowed libSizes for tau or negative tp HOT 4
- Tp < -1 gives error HOT 3
- Different sampling between legacy and current version? HOT 4
- can not library the "rEDM" HOT 22
- CCM Parallel Processing
- Prediction of single time points HOT 1
- Inverted name for the output generated by block_lnlp function HOT 13
- R 4.2.0 Character Encoding on Windows: Can not load rEDM HOT 5
- version 1.15 dealing with NaN HOT 4
- Questions about 'Nonlinearity' and the significance of ρ under the largest L. HOT 1
- block_lnlp() function not found HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redm.