NumPy array is not writeable, and PyTorch does not support non-writeable tensors about s-jsdm

[MaximilianPi]

Hi François,
this seems like a pytorch specific problem. Can you please provide more information about your setup (cpu or gpu, torch version ($conda info))? If you have installed sjSDM via conda (or install_sjSDM()), you can use the new function install_diagnostic(), which I pushed today (which means you have to fetch the latest version and re-install sjSDM).

Best wishes,
Max

[frmunoz]

Hello Max,

Thank you for your answer!

I have defined a cpu setup.
Here is the output of install_diagnostic() on my machine:

active environment : None
user config file : C:\Users\admmunoz\Documents.condarc
populated config files :
conda version : 4.8.3
conda-build version : 3.18.9
python version : 3.7.4.final.0
virtual packages : __cuda=10.1
base environment : C:\PROGRA3\ANACON1 (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch
package cache : C:\PROGRA3\ANACON1\pkgs
C:\Users\admmunoz\Documents.conda\pkgs
C:\Users\admmunoz\AppData\Local\conda\conda\pkgs
envs directories : C:\PROGRA3\ANACON1\envs
C:\Users\admmunoz\Documents.conda\envs
C:\Users\admmunoz\AppData\Local\conda\conda\envs
platform : win-64
user-agent : conda/4.8.3 requests/2.22.0 CPython/3.7.4 Windows/10 Windows/10.0.18362
administrator : True
netrc file : None
offline mode : False

Best wishes,

François

[MaximilianPi]

Hi François,
unfortunately, at the moment the automatic pytorch installation is still unstable but I am actively working on this.
I committed today a few installation updates, that's why I suggest the following procedure (I hope that a re-installation solves your problem):

Update 1:
Florian is right, I was able to reproduce the error with anaconda 4.8.3... (I thought it is a MacOS related issue because the MacOS anaconda does not even allow you to install torch).

You have to uninstall anaconda 4.8.3 and install 4.8.2: https://repo.anaconda.com/miniconda/

Cheers,
Max

[florianhartig]

I'm surprised that this ran at all, because I thought conda 4.8.3 can't install pytorch - did the install really work, or did it not, in which case the error message should be improved.

[MaximilianPi]

This is a significant issue. I will report this to the pytorch community and we have to update our installation instructions... install_sjSDM() installs now automatically conda 4.8.2 so it affects only people who install anaconda themselves

[MaximilianPi]

Okay, I found the source of this warning. There seems to be no issue with conda 4.8.3 on Linux/Windows but with PyTorch 1.5 and reticulate:

• reticulate translates R matrices to non writeable numpy arrays:
  (reticulate::r_to_py(matrix(0.0))$flags)
• With 1.5 the pytorch team implemented a warning when a non-writeable numpy array is transformed to a tensor (in our case that happens when the env and occ matrices (X, Y) are transformed to torch tensors.
• It does not influence the model's behavior, performance, or results (see PyTorch commit: Warns on read-only Numpy array->tensor conversion (#33615))
• For now, I create copies of the actual data, which is not optimal but at least the data is still on the cpu (under the assumption that the cpu memory is not the bottleneck)

s-jSDM/sjSDM/inst/python/sjSDM_py/model_new.py

Lines 119 to 125 in cc66b35

	# reticulate creates non writeable arrays
	X = X.copy()
	Y = Y.copy()
	if type(SP) is np.ndarray:
	SP = SP.copy()
	if type(RE) is np.ndarray:
	RE = RE.copy()

[frmunoz]

Thank you very much for your answers and update of the package.
I can run now the model, but my dataset is too large for the memory I have on my PC... I will try to find a different way.

FYI, here is the new diagnostic I get with the lastest version,

active environment : None

   user config file : C:\Users\admmunoz\Documents\.condarc

populated config files :
conda version : 4.8.3
conda-build version : 3.18.9
python version : 3.7.4.final.0
virtual packages : __cuda=10.1
base environment : C:\PROGRA3\ANACON1 (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch
package cache : C:\PROGRA3\ANACON1\pkgs
C:\Users\admmunoz\Documents.conda\pkgs
C:\Users\admmunoz\AppData\Local\conda\conda\pkgs
envs directories : C:\PROGRA3\ANACON1\envs
C:\Users\admmunoz\Documents.conda\envs
C:\Users\admmunoz\AppData\Local\conda\conda\envs
platform : win-64
user-agent : conda/4.8.3 requests/2.22.0 CPython/3.7.4 Windows/10 Windows/10.0.18362
administrator : True
netrc file : None
offline mode : False

[MaximilianPi]

There are two ways to reduce the memory strain:

1. step_size (in sjSDM(...step_size=): By default it is set to 10% of your number sites, which is high if you have hundreds/thousands of sites or species. Maybe setting it to 1 - 20 could resolve this.
2. Setting manually the "df" option in biotic=bioticStruct(df = XX). df controls the rank of the sigma matrix. By default it is set to (number of species/2) and by reducing it you lose degree of freedoms for the covariance matrix but you also reduce the memory strain

[MaximilianPi]

I will keep this open... something in the code still causes the warning to appear (it appears after the tests... but I'm unable to track it down, 384 tests and the msg only appears once a session).

For now, the work-around is to use reticulate::r_to_py(matrix)$copy() instead of the R-matrix when calling a python function.