Space about s-jsdm

[florianhartig]

I think it would probably be useful to provide an extra term, because that makes it easier to implement the VP part. The question is how to do the syntax.

Maybe, I would add a term

spatial = XX

and then let XX be filled with functions that we provide, e.g.

spatial = spatialDNN(x, y)
spatial = spatialFormula(~ mem1 + mem2 + ...)
spatial = spatialCAR(x, y)

etc? Btw, I was thinking yesterday that we could use the same algorithm that currently fits the biotic interactions also to fit a spatial CAR structure. In the end, it's just a covariance matrix, right? So, if we don't have a problem to fit 10.000 species, we should be able to implement a 10.000 site CAR term as well.

Just to make things more complicated - if we keep the spatial syntax, maybe we would really make this explicit also for the other components, so that do, vor example

sjSDM(X = NULL, Y = NULL, env = envFormula(), biotic = bioticCov(), spatial = spatialDNN()

and then we could also provide the lasso option directly in the function. You see what I mean?

We should of course think through if this really works and makes sense for the user and programming-wise, but I think it might be easier and more compact to implement.

It's basically the same principle as the varStruct, corStruct ... although it has to be said that this is always terribly confusing to people.

If we do this, we maybe also want to drop X, and let people provide X directly in the env and space components.

[MaximilianPi]

I think it would probably be useful to provide an extra term, because that makes it easier to implement the VP part. The question is how to do the syntax.

Maybe, I would add a term

spatial = XX

and then let XX be filled with functions that we provide, e.g.

Good idea, that way it also does not interfere with the formula syntax

spatial = spatialDNN(x, y)
spatial = spatialFormula(~ mem1 + mem2 + ...)
spatial = spatialCAR(x, y)

etc? Btw, I was thinking yesterday that we could use the same algorithm that currently fits the biotic interactions also to fit a spatial CAR structure. In the end, it's just a covariance matrix, right? So, if we don't have a problem to fit 10.000 species, we should be able to implement a 10.000 site CAR term as well.

Just to make things more complicated - if we keep the spatial syntax, maybe we would really make this explicit also for the other components, so that do, vor example

sjSDM(X = NULL, Y = NULL, env = envFormula(), biotic = bioticCov(), spatial = spatialDNN()

and then we could also provide the lasso option directly in the function. You see what I mean?

We should of course think through if this really works and makes sense for the user and programming-wise, but I think it might be easier and more compact to implement.

It's basically the same principle as the varStruct, corStruct ... although it has to be said that this is always terribly confusing to people.

If we do this, we maybe also want to drop X, and let people provide X directly in the env and space components.

Interesting idea, here a few remarks:

Actually, we do not fit a covariance matrix with the CAR model, right? The weight matrix can be created from a distance matrix or we could even use a sparse neighbor matrix (ICAR). In the CAR, the calculation of the inverse matrix ( (D*(I-aW)^-1 ) is the bottleneck.

However, the memory puts an upper limit on the CAR approach. 10.000 sites results in a 40gb matrix (single precision), which is not feasible for most of the GPUs. On the other hand, how realistic are eDNA communities with ten thousands of sites? If there's not enough memory, people just have to switch to spatial predictors.

Also, I have concerns how useful the CAR actually is. We have to discuss this point.

From the technical side, it should not be to difficult to implement. We can inhere from the model_base class. Also, I think it should be compatible with stochastic gradient descent (but ofc, we have first to simulate this ).

[MaximilianPi]

I'm kind of delaying this intentionally because the API change will break the scripts for the paper... except (maybe) I write a lot of additional code.

Not sure what to do

[florianhartig]

Make a branch?

[MaximilianPi]

Solved by spatial eigenvectors and trend surface models