Installing tensorflow/tfjs-node fails on macOS Sonoma (14.3) on Apple Silicon ("Command failed: node-pre-gyp install --fallback-to-build about tfjs

[gaikwadrahul8]

Hi, @gohai

Thank you for bringing this issue to our attention and I see currently you're using Node v18.19.0 with Python 3.11.7 version so could you please give it try with below mentioned versions from your end and see is it resolving your issue or not ? I would also suggest you to please give it try with different different Python versions from 3.6 to 3.11

1. Python 3.11.6
2. Node v20.10.0
3. npm v10.2.3

If issue still persists please let us know with new error log to investigate this issue further from our end.

Thank you for your understanding and patience.

[gohai]

@gaikwadrahul8 Thank you for looking into this. I installed Node 20, and I am seeing the same issue.

My specific versions are:

1. Python 3.11.7
2. Node v20.12.0
3. npm v10.5.0

New error log below:

error /Users/gohai/Desktop/foo/node_modules/@tensorflow/tfjs-node: Command failed.
Exit code: 1
Command: node scripts/install.js
Arguments:
Directory: /Users/gohai/Desktop/foo/node_modules/@tensorflow/tfjs-node
Output:
CPU-darwin-4.17.0.tar.gz
* Downloading libtensorflow
https://storage.googleapis.com/tf-builds/libtensorflow_r2_7_darwin_arm64_cpu.tar.gz

* Building TensorFlow Node.js bindings
node-pre-gyp install failed with error: Error: Command failed: node-pre-gyp install --fallback-to-build
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using [email protected]
node-pre-gyp info using [email protected] | darwin | arm64
node-pre-gyp info check checked for "/Users/gohai/Desktop/foo/node_modules/@tensorflow/tfjs-node/lib/napi-v8/tfjs_binding.node" (not found)
node-pre-gyp http GET https://storage.googleapis.com/tf-builds/pre-built-binary/napi-v8/4.17.0/CPU-darwin-4.17.0.tar.gz
node-pre-gyp ERR! install response status 404 Not Found on https://storage.googleapis.com/tf-builds/pre-built-binary/napi-v8/4.17.0/CPU-darwin-4.17.0.tar.gz
node-pre-gyp WARN Pre-built binaries not installable for @tensorflow/[email protected] and [email protected] (node-v115 ABI, unknown) (falling back to source compile with node-gyp)
node-pre-gyp WARN Hit error response status 404 Not Found on https://storage.googleapis.com/tf-builds/pre-built-binary/napi-v8/4.17.0/CPU-darwin-4.17.0.tar.gz
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | darwin | arm64
gyp info ok
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | darwin | arm64
gyp info find Python using Python version 3.11.7 found at "/opt/homebrew/opt/[email protected]/bin/python3.11"
gyp http GET https://nodejs.org/download/release/v20.12.0/node-v20.12.0-headers.tar.gz
gyp http 200 https://nodejs.org/download/release/v20.12.0/node-v20.12.0-headers.tar.gz
gyp http GET https://nodejs.org/download/release/v20.12.0/SHASUMS256.txt
gyp http 200 https://nodejs.org/download/release/v20.12.0/SHASUMS256.txt
gyp info spawn /opt/homebrew/opt/[email protected]/bin/python3.11
gyp info spawn args [
gyp info spawn args '/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args 'binding.gyp',
gyp info spawn args '-f',
gyp info spawn args 'make',
gyp info spawn args '-I',
gyp info spawn args '/Users/gohai/Desktop/foo/node_modules/@tensorflow/tfjs-node/build/config.gypi',
gyp info spawn args '-I',
gyp info spawn args '/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
gyp info spawn args '-I',
gyp info spawn args '/Users/gohai/Library/Caches/node-gyp/20.12.0/include/node/common.gypi',
gyp info spawn args '-Dlibrary=shared_library',
gyp info spawn args '-Dvisibility=default',
gyp info spawn args '-Dnode_root_dir=/Users/gohai/Library/Caches/node-gyp/20.12.0',
gyp info spawn args '-Dnode_gyp_dir=/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp',
gyp info spawn args '-Dnode_lib_file=/Users/gohai/Library/Caches/node-gyp/20.12.0/<(target_arch)/node.lib',
gyp info spawn args '-Dmodule_root_dir=/Users/gohai/Desktop/foo/node_modules/@tensorflow/tfjs-node',
gyp info spawn args '-Dnode_engine=v8',
gyp info spawn args '--depth=.',
gyp info spawn args '--no-parallel',
gyp info spawn args '--generator-output',
gyp info spawn args 'build',
gyp info spawn args '-Goutput_dir=.'
gyp info spawn args ]
No receipt for 'com.apple.pkg.CLTools_Executables' found at '/'.

No receipt for 'com.apple.pkg.DeveloperToolsCLILeo' found at '/'.

No receipt for 'com.apple.pkg.DeveloperToolsCLI' found at '/'.

Traceback (most recent call last):
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/xcode_emulation.py", line 1496, in XcodeVersion
    version_list = GetStdoutQuiet(["xcodebuild", "-version"]).splitlines()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/xcode_emulation.py", line 1557, in GetStdoutQuiet
    raise GypError("Error %d running %s" % (job.returncode, cmdlist[0]))
gyp.common.GypError: Error 1 running xcodebuild

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py", line 45, in <module>
    sys.exit(gyp.script_main())
             ^^^^^^^^^^^^^^^^^
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 688, in script_main
    return main(sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 680, in main
    return gyp_main(args)
           ^^^^^^^^^^^^^^
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 665, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 2648, in GenerateOutput
    writer.Write(
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 945, in Write
    self.WriteTarget(
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 1767, in WriteTarget
    self.WriteSortedXcodeEnv(self.output, self.GetSortedXcodePostbuildEnv())
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 2282, in GetSortedXcodePostbuildEnv
    return self.GetSortedXcodeEnv(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 2266, in GetSortedXcodeEnv
    return gyp.xcode_emulation.GetSortedXcodeEnv(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/xcode_emulation.py", line 1882, in GetSortedXcodeEnv
    env = _GetXcodeEnv(
          ^^^^^^^^^^^^^
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/xcode_emulation.py", line 1737, in _GetXcodeEnv
    "XCODE_VERSION_ACTUAL": XcodeVersion()[0],
                            ^^^^^^^^^^^^^^
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/xcode_emulation.py", line 1507, in XcodeVersion
    version = CLTVersion()  # macOS Catalina returns 11.0.0.0.1.1567737322
              ^^^^^^^^^^^^
  File "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/xcode_emulation.py", line 1545, in CLTVersion
    return re.search(regex, output).groupdict()["version"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'groupdict'
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack at ChildProcess.<anonymous> (/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:271:18)
gyp ERR! stack at ChildProcess.emit (node:events:518:28)
gyp ERR! stack at ChildProcess._handle.onexit (node:internal/child_process:294:12)
gyp ERR! System Darwin 23.3.0
gyp ERR! command "/Users/gohai/.nvm/versions/node/v20.12.0/bin/node" "/Users/gohai/.nvm/versions/node/v20.12.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/Users/gohai/Desktop/foo/node_modules/@tensorflow/tfjs-node/lib/napi-v8/tfjs_binding.node" "--module_name=tfjs_binding" "--module_path=/Users/gohai/Desktop/foo/node_modules/@tensorflow/tfjs-node/lib/napi-v8" "--napi_version=9" "--node_abi_napi=napi" "--napi_build_version=8" "--node_napi_label=napi-v8"
gyp ERR! cwd /Users/gohai/Desktop/foo/node_modules/@tensorflow/tfjs-node
gyp ERR! node -v v20.12.0
gyp ERR! node-gyp -v v10.0.1
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute '/Users/gohai/.nvm/versions/node/v20.12.0/bin/node /Users/gohai/.nvm/versions/node

[gohai]

FWIW, the output of manually executing xcodebuild -version is:

xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance

(with error code 1)

[gaikwadrahul8]

Hi, @gohai

The error indicates that xcodebuild is unavailable because the active developer directory points to Command Line Tools, not a full Xcode installation. tfjs-node requires certain components from Xcode itself for building native dependencies.

1. If you haven't already, download and install the full Xcode from the App Store or Apple Developer website so please do it and follow next steps.
2. Set the Developer Directory (if needed):
   Open a terminal and run xcode-select --switch /Applications/Xcode.app/Contents/Developer (replace with the actual path if Xcode is installed elsewhere).
3. Verify Xcodebuild: Run xcodebuild -version. It should now display Xcode version information.

Please refer this official documentation of node-gyp - Node.js native addon build tool for macOS and reattempt installing tfjs-node using npm i @tensorflow/tfjs-node. It should now proceed without the xcodebuild error

Thank you for your cooperation and patience.

[gohai]

@gaikwadrahul8 Thank you for continuing to look into this. The node-gyp documentation you liked indicates that full Xcode is not needed. If I, as described there, run xcode-select --install I see:

xcode-select: note: Command line tools are already installed. Use "Software Update" in System Settings or the softwareupdate command line interface to install updates

Attempting to install tfjs-node using npm fails with the same error.

[gaikwadrahul8]

Hi, @gohai

Thank you for trying above instructions, Open a terminal and run xcode-select --print-path. This should show the path to your Xcode installation like /Applications/Xcode.app/Contents/Developer

While TensorFlow.js 4.17 should work with Node.js v20.10.0 version there might be compatibility issues. Consider trying a different LTS Node.js version using a version manager like nvm so you can switch easily to different Node.js versions to try out and for Python also you can use pyenv to try out Python version less than 3.12.x and greater than 3.6.x .

you can try installing node-gyp globally using npm install -g node-gyp. This might help if the local installation is problematic. Use this approach with caution, as it might affect other projects that rely on specific node-gyp versions.

I would suggest you to in your project directory, delete the node_modules folder and package-lock.json file (if present) or create brand new Node.js project and try to install tjfs-node using npm i @tensorflow/tfjs-node

While less likely, ensure you have a stable internet connection during the installation as tfjs-node might attempt to download pre-built binaries.

Thank you for your cooperation and patience.

[gaikwadrahul8]

Hi, @gohai

I tried to install tfjs-node on my Apple M1 pro laptop with below steps and it's working as expected so please give it try by following commands from my output log and one more thing I have installed full xcode on my Apple M1 pro laptop

I have also trained small model to just cross verify and it's working as expected so please give it try by installing full Xcode on your macOS and see is it working as expected or not ?

If issue still persists please let us know with full error log to investigate this issue further from our end.

(base) gaikwadrahul-macbookpro:Test-8228 gaikwadrahul$ node -v
v20.12.0
(base) gaikwadrahul-macbookpro:Test-8228 gaikwadrahul$ npm -v
10.5.0
(base) gaikwadrahul-macbookpro:Test-8228 gaikwadrahul$ python3.11
Python 3.11.0 (v3.11.0:deaf509e8f, Oct 24 2022, 14:43:23) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
(base) gaikwadrahul-macbookpro:Test-8228 gaikwadrahul$ npm init -y
Wrote to /Users/gaikwadrahul/Desktop/Test-8228/package.json:

{
  "name": "test-8228",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

(base) gaikwadrahul-macbookpro:Test-8228 gaikwadrahul$ touch index.js
(base) gaikwadrahul-macbookpro:Test-8228 gaikwadrahul$ which python3.11
/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
(base) gaikwadrahul-macbookpro:Test-8228 gaikwadrahul$ npm install -g node-gyp --python="/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11”
changed 101 packages in 4s

14 packages are looking for funding
  run `npm fund` for details
(base) gaikwadrahul-macbookpro:Test-8228 gaikwadrahul$ npm install --python="/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11" @tensorflow/tfjs-node

added 121 packages, and audited 122 packages in 1m

9 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

Package.json file output :

{
  "name": "test-8228",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "type": "module",
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@tensorflow/tfjs-node": "^4.17.0"
  }
}

(base) gaikwadrahul-macbookpro:Test-8228 gaikwadrahul$ node index.js
Epoch 1 / 100
eta=0.0 ====================================================================================================> 
278ms 28us/step - MAE=0.326 loss=0.162 val_MAE=0.282 val_loss=0.120 
Epoch 2 / 100
eta=0.0 ====================================================================================================> 
220ms 22us/step - MAE=0.279 loss=0.113 val_MAE=0.260 val_loss=0.0963 
Epoch 3 / 100
eta=0.0 ====================================================================================================> 
201ms 20us/step - MAE=0.263 loss=0.0964 val_MAE=0.284 val_loss=0.116 
Epoch 4 / 100
eta=0.0 ====================================================================================================> 
203ms 20us/step - MAE=0.257 loss=0.0901 val_MAE=0.247 val_loss=0.0861 
Epoch 5 / 100
eta=0.0 ====================================================================================================> 
191ms 19us/step - MAE=0.253 loss=0.0871 val_MAE=0.263 val_loss=0.101 

Thank you for your cooperation and patience.

[gaikwadrahul8]

Hi, @gohai

To confirm, were you able to successfully install tfjs-node on your macOS system by following the provided instructions and workaround? If the installation was successful, please feel free to close this issue.

If you're still encountering problems, please provide us with a new error log after attempting the instructions and workaround. This will help us investigate the issue further on our end.

Thank you for your cooperation and patience in resolving this matter.

[gohai]

Hi @gaikwadrahul8 - thank you for your messages, and efforts in trying to replicate.

I ultimately never got it to work on my machine. I had tried with node v19 earlier, with the same symptoms. I do believe that this is fundamentally because of my computer not having xcode installed.

Unfortunately, our use case is that we had hoped to use tfjs-node to run tests as part of an open source library. We can't mandate, or expect, all our users/junior developers to have xcode installed when they merely want to build the library, or get their feet wet developing. That's feels like too high of a barrier to entry. We thus gave up on this plan. (Feel free to close this issue.)

If there is still some momentum behind developing tfjs further, I believe it would be worth making this more robust so that it can be installed and used on a wide range of host systems (e.g. without xcode, with various node and python versions [the Readme still states that Python 2.x is required]). Thanks.

[gaikwadrahul8]

Hi, @gohai

Thank you for the detailed update and We appreciate you sharing your experience and the specific use case where you wanted to use tjfs-node without installing the full Xcode on your system to run tests as part of an open source library.

We understand your situation regarding the requirement for Xcode and the barrier it creates for some users/junior developers. This is valuable feedback and we'll definitely consider it for future development of tfjs-node.

Improving compatibility across various environments (like different Node.js and Python versions) is definitely a consideration for future development of tfjs-node. We appreciate you bringing this to our attention. I have submitted a PR #8101 to update outdated README.md file about Python version I believe that issue will be taken care by #8101 PR once it got merged.

If you have further questions or suggestions, please don't hesitate to reach out. We're always looking for ways to improve the usability of tfjs-node.

As you mentioned, feel free to consider this issue closed so I'm going ahead and closing this issue for now. If you need any help please feel free to create new issue

Thank you for your cooperation and patience.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No