npm postinstall fails in Windows: rd /s /q worker\out\msys => The system cannot find the file specified about mediasoup

[ibc]

@jmillan @nazar-pc are we ok with the temporary solution I suggest in my comment above?

executeCmd('rd /s /q worker\\out\\msys', /* exitOnError */ false);

[ibc]

Asking to a workmate.

[nazar-pc]

So @nazar-pc, I fail to understand that possible reason. It's not possible.

I do not know how/what/where, but it is clear that the files are not there. Or maybe they are extracted into worker/out/msys, but it is not worker/out/msys that we think it is or something (maybe pwd inside of Python script is different, so it extracts files elsewhere, providing an absolute path would fix that particular issue).

[ibc]

Windows ecosystem is pathetic. I've google the above error and there are just tons of posts telling you to reinstall Windows, format HD, etc.

[ibc]

@haiyangwu @nazar-pc maybe you know something about this recent error in Windows?

[nazar-pc]

Looks like it is a cleanup step after getmake.py, maybe make was not fixed to a specific version and has changed now?

[ibc]

A workaround is to avoid error being thrown in the cleanWorkerArtifacts() function as follows:

executeCmd('rd /s /q worker\\out\\msys', /* exitOnError */ false);

So basically we do not clean worker artifacts after building the worker, which IMHO is not that bad.

Looks like it is a cleanup step after getmake.py, maybe make was not fixed to a specific version and has changed now?

Absolutely no idea. getmake.py clearly run this:

get('https://sourceforge.net/projects/mingw/files/MSYS/Base/make/make-3.81-3/make-3.81-3-msys-1.0.13-bin.tar.lzma/download', '847f0cbbf07135801c8e67bf692d29b1821e816ad828753c997fa869a9b89988')

And it does this at the beginning (I have no idea what it is about) which clearly extracts stuff into 'worker/out/msys' folder.

def get(url, digest):
  data = urllib.request.urlopen(url).read()
  assert hashlib.sha256(data).hexdigest() == digest
  tar = tarfile.open(fileobj=io.BytesIO(data))
  tar.extractall('worker/out/msys')
  tar.close()

[nazar-pc]

I would prefer to understand why it happens though. Are you able to spin up a Windows VM to see what is actually going on in file system there?

[ibc]

Definitely I don't have any (virtual) Windows setup to test :(

[ibc]

I have an idea: create a temporal PR that catches that error and then prints the content of the worker/out/msys folder (if it exists, and it should because postinstall installed things in there). Do you know the exact command for that? Is it dir worker\\out\\msys?

[nazar-pc]

I mean you can install any VM management (like VirtualBox Parallels, VmWare), download official Windows ISO and install it in a VM. Then you will have, that is how I debugged some Windows issues, I don't run Windows daily either.

[ibc]

Yes, but I'm super busy with tons of tasks (specially at work) and honestly I cannot deal with virtual machines right now.

[nazar-pc]

I have an idea: create a temporal PR that catches that error and then prints the content of the worker/out/msys folder (if it exists, and it should because postinstall installed things in there). Do you know the exact command for that? Is it dir worker\\out\\msys?

If you want to debug it in CI then makes sense to me, but only as a debugging step. Would be great if someone on Windows debugged this though.

[ArcanisCz]

Hi! I have tried to install mediasoup on my windows machine, and it works for me.

C:\Users\arcan\git\test>npm i mediasoup

added 21 packages in 6m

6 packages are looking for funding
  run `npm fund` for details

• [email protected]
• Microsoft Visual Studio Community 2019 Version 16.11.9
• Windows 11 Home 22H2 22621.2428

msys folder is not present after installation

C:\Users\arcan\git\test\node_modules\mediasoup\worker\out>ls
Release/

Some output from postinstall

The Meson build system
Version: 1.2.1
Source dir: C:\Users\arcan\git\test\node_modules\mediasoup\worker
Build dir: C:\Users\arcan\git\test\node_modules\mediasoup\worker\out\Release\build
Build type: native build
Project name: mediasoup-worker
Project version: undefined
C compiler for the host machine: cl (msvc 19.29.30139 "Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30139 for x64")
C linker for the host machine: link link 14.29.30139.0
C++ compiler for the host machine: cl (msvc 19.29.30139 "Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30139 for x64")
C++ linker for the host machine: link link 14.29.30139.0
Host machine cpu family: x86_64
Host machine cpu: x86_64
Downloading openssl source from https://www.openssl.org/source/openssl-3.0.8.tar.gz
Download size: 15151328
Downloading: ..........
Downloading openssl patch from https://wrapdb.mesonbuild.com/v2/openssl_3.0.8-1/get_patch
Download size: 11463052
...
Deleting C:\Users\arcan\git\test\node_modules\mediasoup\worker\subprojects\libsrtp-2.5.0
Deleting C:\Users\arcan\git\test\node_modules\mediasoup\worker\subprojects\packagecache\openssl-3.0.8.tar.gz
Deleting C:\Users\arcan\git\test\node_modules\mediasoup\worker\subprojects\packagecache\openssl_3.0.8-1_patch.zip
Deleting C:\Users\arcan\git\test\node_modules\mediasoup\worker\subprojects\openssl-3.0.8
make: Leaving directory `/c/Users/arcan/git/test/node_modules/mediasoup/worker'
npm-scripts [INFO] [postinstall] executeCmd(): make clean-pip -C worker
make: Entering directory `/c/Users/arcan/git/test/node_modules/mediasoup/worker'
rm -f -rf /c/Users/arcan/git/test/node_modules/mediasoup/worker/out/pip
make: Leaving directory `/c/Users/arcan/git/test/node_modules/mediasoup/worker'
npm-scripts [INFO] [postinstall] executeCmd(): rd /s /q worker\out\msys
npm info run [email protected] postinstall { code: 0, signal: null }

which seems like the folder msys was there and the command deleted it sucesfully. If I execute command after installation, it says error

C:\Users\arcan\git\test\node_modules\mediasoup>rd /s /q worker\out\msys
The system cannot find the file specified.

which leads me to conclusion that the CI build did not created msys folder. When I create msys folder manually and run the command again, it executes without error and deletes the folder. So the command itself works as expected.

[ArcanisCz]

Only differences I see are

• CI mscv 19.35, my machine 19.29
• CI using mingw32-make, my machine just make
• CI pip 23, my machine pip 21

my make

GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i686-pc-msys

[ibc]

Thanks @ArcanisCz, super useful.

[ArcanisCz]

If the problem is only a failing command in case folder does not exist, maybe smth like https://stackoverflow.com/questions/14502612/delete-a-directory-and-its-files-using-command-line-but-dont-throw-error-if-it can be usefull?

Or the real underlying problem is that the msys folder is not created and should have been there always?

[ArcanisCz]

Do you want me to see contents of the folder before deletion on my system?

[ibc]

If the problem is only a failing command in case folder does not exist, maybe smth like https://stackoverflow.com/questions/14502612/delete-a-directory-and-its-files-using-command-line-but-dont-throw-error-if-it can be usefull?

That's basically the solution I suggested before. See #1179 (comment)

But it's not a nice solution because worker\out\msys folder SHOULD exist at that point. I've explained all the flow in the issue description above (updated right now). It's impossible that such a folder doesn't exist at that time when compilation finishes.

Do you want me to see contents of the folder before deletion on my system?

Yes please. Not sure if you need to checkout mediasoup repo for that.

[nazar-pc]

The only possible reason for worker/out/msys to not exist is for file extraction to fail, which explains why wrong make is used as described above.

[ArcanisCz]

The folder is there from basically start of postinstall script (when meson/python initializes) and its there due to postinstall (was not created when I --ignore-scripts)

[nazar-pc]

That is how it should be. If CI fails then in CI downloading/extraction flow is broken for some reason.

[ArcanisCz]

In case you want to see full output from my machine:

fulllog.txt

Not sure how can I help more now, if you need me to test/try anything please ping me 🙏

[ibc]

The only possible reason for worker/out/msys to not exist is for file extraction to fail, which explains why wrong make is used as described above.

Wow, but... if file extraction fails then it means that getmake.py script failed, and if it failed it means that executeCmd(${String(res.stdout).trim()} worker\scripts\getmake.py) did throw, and hence installMsysMake() did also throw, and hence buildWorker() did also throw, and if so the whole script terminates and cleanWorkerArtifacts() CANNOT be executed so it cannot fail.

So @nazar-pc, I fail to understand that possible reason. It's not possible.

[ibc]

In case you want to see full output from my machine:

fulllog.txt

Your logs say that everything is fine and the whole thing behaves as expected:

npm-scripts [INFO] [postinstall] buildWorker()
npm-scripts [INFO] [postinstall] installMsysMake()
npm-scripts [INFO] [postinstall] executeCmd(): C:\Users\arcan\AppData\Local\Programs\Python\Python38\python.exe worker\scripts\getmake.py
npm-scripts [INFO] [postinstall] executeCmd(): make -C worker

...

npm-scripts [INFO] [postinstall] executeCmd(): rd /s /q worker\out\msys
npm info run [email protected] postinstall { code: 0, signal: null }

Not sure how can I help more now, if you need me to test/try anything please ping me 🙏

Thanks!

[ibc]

Or maybe they are extracted into worker/out/msys, but it is not worker/out/msys that we think it is or something (maybe pwd inside of Python script is different, so it extracts files elsewhere, providing an absolute path would fix that particular issue).

#1187