[Windows] save TSNEEmbedding to binary, Directory error about opentsne

[gaardhus]

I've been having the same issue. After downgrading to openTSNE==0.6.0 I was able to both save and load the fitted model.

The example data from the README however works for both 0.6.0, 0.6.1 and 0.6.2 for me. This could possibly have something to do with number of observations and/or the resulting model size?

This is the error message I get, I tried deleting the temp folder before re-running and also tried restarting my computer without any success.

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\shutil.py", line 625, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\tobia\\AppData\\Local\\Temp\\tmpvvo0p0t8\\tmp.ann'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\tempfile.py", line 805, in onerror
    _os.unlink(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\tobia\\AppData\\Local\\Temp\\tmpvvo0p0t8\\tmp.ann'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\tobia\Programming\generate_example_data.py", line 126, in <module>
    pickle.dump(dv_model, f)
  File "C:\Users\tobia\Programming\venv\lib\site-packages\openTSNE\nearest_neighbors.py", line 353, in __getstate__
    b64_index = base64.b64encode(f.read())
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\tempfile.py", line 830, in __exit__
    self.cleanup()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\tempfile.py", line 834, in cleanup
    self._rmtree(self.name)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\tempfile.py", line 816, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\shutil.py", line 757, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\shutil.py", line 627, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\tempfile.py", line 808, in onerror
    cls._rmtree(path)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\tempfile.py", line 816, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\shutil.py", line 757, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\shutil.py", line 608, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\shutil.py", line 605, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\tobia\\AppData\\Local\\Temp\\tmpvvo0p0t8\\tmp.ann'

From the example which runs without any errors:

import pickle
import openTSNE
from openTSNE import TSNE
from sklearn import datasets
print(openTSNE.__version__)

iris = datasets.load_iris()
x, y = iris["data"], iris["target"]

embedding = TSNE().fit(x)

with open('iris.pkl', 'wb') as f:
    pickle.dump(embedding, f)

with open('iris.pkl', 'rb') as f:
    embedding = pickle.load(f)

embedding.transform(x)

[pavlin-policar]

In v0.6.2, I fixed the pickling behaviour so it works as one would expect. Annoy, which is used to find nearest neighbors, is not picklable, but uses its own internal file structure. So in previous versions, the annoy nearest neighbor index was saved to a separate file. So, pickling a TSNEEmbedding object actually produced two files, an annoy file and a pickle file. This was very fragile and there were just a ton of problems if you wanted to move files around.

In v0.6.2, I got rid of this and just included everything in the one pickle file. What happens now is that the annoy file is still saved to disk, but in a temporary directory (this is the reason for the C:\\Users\\tobia\\AppData\\Local\\Temp\\tmpvvo0p0t8\\tmp.ann directory). This file is then serialized into a binary string and saved to the pickle. When unpickling, we then write this binary string into another temporary file, which annoy can then read.

As to your problem: Would it be possible that you don't have permission to write to the Windows tmp directory? I don't have a windows machine, so I can't really test this out. There doesn't seem to be anything wrong with the actual path to the temporary file.

I'll close this for now, but if this problem persists, please reopen this issue.