Pet performance issue about emacs-pet

[wyuenho]

Opening the first python file of a project takes a little longer because it needs to populate the cache, but once the cache is populated, the 2 functions should run substantially faster. Is this not the case for you?

[thisch]

Opening the first python file of a project takes a little longer because it needs to populate the cache, but once the cache is populated, the 2 functions should run substantially faster.

I would have also expected that, but it is not the case.

[janoc]

I see this problem too - it took me a while to understand why opening of my Python files is so incredibly slow.

[wyuenho]

It's expected to take a few seconds when opening the first python file in a project, but opening subsequent python files within the same project should be relatively quick due to the aggressive caching. The slowness mainly comes from the number of subprocesses Emacs has to open in order to parse the config and cache files. It's a problem if opening subsequent python files within the same project is slow, but I'll need more information such as your project layout and benchmark numbers to even confirm it is indeed a bug. There are opportunities to speed up opening even the first file, but it's not worth the effort. Emacs 29 will come with a number of improvements that will enable this.

[janoc]

I see this in Windows 10 where Emacs is generally extremely slugish (had to turn off pet, lsp-mode and a few other things to even get to a point that I don't need to wait 30 seconds to open each file ...), so I guess you can disregard my comment. In Linux/WSL it works fine - there is a slight delay when pet is enabled in the python-mode-hook but nothing dramatic.

[janoc]

OK, I did some more testing - in Linux, under WSL but all files are on the Linux filesystem (not accessing NTFS, that would be very slow).

I have multiple files from the same project open already, my hook is starting pet and looking for several executables when I open another Python file from the same project. This is what the profiler gives me (I have instrumented the entire pet package):

Function                                  Call count     Elapsed       Average elapsed 
my-pet-hook                                  1           20.144855874  20.144855874
pet-executable-find                          10          18.975925474  1.8975925474
pet-virtualenv-root                          11          13.110769078  1.191888098
pet-flycheck-toggle-local-vars               1           9.248647607   9.248647607
pet-find-file-from-project                   73          3.9797493269  0.0545171140
pet-find-file-from-project-root-recursively  33          3.504320876   0.1061915416
pet-use-pyenv-p                              11          1.285627846   0.1168752587
pet-python-version                           11          1.2855474310  0.1168679482
pet-python-version-path                      11          1.2854736979  0.1168612452
pet-use-poetry-p                             11          1.267356647   0.1152142406
pet-pyproject                                11          1.267226448   0.1152024043
pet-pyproject-path                           11          1.267143984   0.1151949076
pet-use-conda-p                              11          1.2144122179  0.1104011107
pet-environment                              11          1.2143436120  0.1103948738
pet-environment-path                         11          1.2142735820  0.1103885074
pet-project-root                             205         0.6063872709  0.0029579866
pet-find-file-from-project-root              73          0.2436195300  0.0033372538
pet-locate-dominating-file                   55          0.228690813   0.0041580147
pet-pipfile-path                             22          0.1521649430  0.0069165883
pet-use-pipenv-p                             11          0.082937444   0.0075397676
pet-pipfile                                  11          0.0822894039  0.0074808549
pet-pre-commit-config                        18          0.0613882240  0.0034104568
pet-pre-commit-config-path                   18          0.0612433259  0.0034024069
pet-use-pre-commit-p                         10          0.0349311759  0.0034931175
pet-pre-commit-config-has-hook-p             8           0.02727297    0.0034091212
pet-flycheck-python-pylint-find-pylintrc     1           0.005619078   0.005619078
pet-flycheck-checker-get-advice              170         0.0006075750  3.573...e-06
pet-flycheck-setup                           1           0.000163408   0.000163408
pet-report-error                             14          5.454...e-05  3.896...e-06
eglot-ensure                                 1           4.0053e-05    4.0053e-05

(my-pet-hook is the top level hook that starts pet, eglot-ensure is there just to check it wasn't language server causing it, ignore those two)

It is clear that something isn't cached because those executables should be the same, regardless of which file from the project I open. I had 3 other Python files from that project open already - and it is this slow whenever I try to open another one. pet-virtualenv-root seems to take ages for some reason.

For info:

• Pipenv project, Pipfile is in not in the project root but in a subdirectory, the source files being opened are in subfolders of that subdirectory with the Pipfile. Virtual environment is created in .venv folder next to the Pipfile.
• We are using git and pre-commit for the hooks (yes, it is that project ...)
• Emacs is self-built from master but I have the same behavior with 28.x and 29 dailies.
• Pet is 7620c18 (HEAD -> main, tag: v1.1.0, origin/main, origin/HEAD) version 1.1.0

[janoc]

Oof, OK I see why the problem happens for me. WSL automatically "helpfully" adds the content of Windows PATH variable to the Linux PATH. So when I didn't create the virtual environment for the current worktree yet, pet started to search on the Windows filesystem for pipenv and other binaries. Which is extremely slow.

Once I have created a local virtual environment, the file opening time dropped from 20 seconds to about 2s. Still a bit sluggish but much more manageable. This is the profile after the the cache has been populated:

my-pet-hook                                  1           2.163337651   2.163337651
pet-executable-find                          10          2.1546226499  0.2154622649
pet-flycheck-toggle-local-vars               1           0.880342973   0.880342973
pet-project-root                             29          0.07939641    0.0027378072
pet-pre-commit-config                        18          0.0552564690  0.0030698038
pet-pre-commit-config-path                   18          0.055118069   0.0030621149
pet-find-file-from-project                   18          0.0549999620  0.0030555534
pet-find-file-from-project-root              18          0.0546787409  0.0030377078
pet-use-pre-commit-p                         10          0.0317966749  0.0031796674
pet-virtualenv-root                          11          0.028649633   0.0026045120
pet-pre-commit-config-has-hook-p             8           0.024069906   0.0030087382
pet-flycheck-python-pylint-find-pylintrc     1           0.005421203   0.005421203
pet-flycheck-checker-get-advice              173         0.0002698909  1.560...e-06
pet-flycheck-setup                           1           0.000240067   0.000240067
eglot-ensure                                 1           2.3442e-05    2.3442e-05
pet-system-bin-dir                           10          9.872...e-06  9.872e-07
pet-report-error                             1           3.47e-06      3.47e-06

I am not sure whether it makes sense to add any sort of fix/workaround for this since this is a Windows/WSL specific nonsense and not really pet's fault but maybe mentioning this in the README would save some hair to some future user. Alternatively, if it detects poetry/pipenv but doesn't find a virtual environment, it could emit a warning message - "Hey, idiot, you forgot to run pipenv install/poetry install!"

However, caching the found executables (and not only the virtualenv root) per project would be likely still good idea that could make pet a good deal snappier. At 200ms per binary it is still 2 seconds delay when opening a file. Such cache would probably make it usable even in Windows once one "pays the price" for populating the cache with the first opened file.

[wyuenho]

I'm going to close this. Please reopen if this is still an issue.