prefer dynamically built binaries when already installed about clang-tools-pip

[2bndy5]

link is not directory.

Sorry, it's install_dir does not exist.

yep, that fixed the creation of the symlinks to dynamic binaries.

Now we just have to decide how to handle different patch releases of dynamic binaries (v12.0.0 vs v12.0.1) when clang-*-12 is invoked.

[2bndy5]

It is more common to have a --strict option and let the default behavior be greedy. I stole the word "greedy" from regex terminology.

[shenxianpeng]

It's may be just for me, the word strict is easier to understand its behavior here than greedy.

[shenxianpeng]

Sure, please help to release v0.6.0

[shenxianpeng]

It looks good to do so. thanks for your flowchar!

[2bndy5]

I added this feature to the use-native-binaries branch, but the CI fails to create symlinks on the runner

When I tested this locally (with v12.0.1 installed via apt), it all went fine. Notice that I'm only checking that the major version matches. So, -i 12 will symlink to v12.0.0 or v12.0.1.

on the CI runner:

Found a installed version of clang-tidy: 12.0.0 at /usr/bin/clang-tidy-12

on my local machine (in WSL):

Found a installed version of clang-tidy: 12.0.1 at /usr/bin/clang-tidy-12

---

BTW, was there a reason to only run pytest on the main branch? I find it annoying to push something and then manually trigger the workflow every time (for any branch except main).

[shenxianpeng]

I didn't reproduce the CI fails on my environment, its clang-format location is /usr/lib/llvm-12/bin/clang-format, and it failed when linked to it.

It is OK just run pytest on the main branch and create a PR, so that it will not be triggered when git push to other branches.

[2bndy5]

Found a installed version of clang-tidy: 12.0.1 at /usr/bin/clang-tidy-12

its clang-format location is /usr/lib/llvm-12/bin/clang-format

I found that the /usr/bin/clang-tidy-12 was actually a symlink, so I made the prompt show the resolved path to the actual binary which is located in /usr/lib/llvm-12/bin/. Unfortunately, I can't figure out why the CI can't create a symlink to the binary.

As a quick hack, I guess we could make this pkg output the location of the binary to use instead of relying on a symlink in a known location. But the windows runner doesn't seem to be recognizing that v13 is already installed, so my implementation isn't ready for deployment.

[shenxianpeng]

Notice that I'm only checking that the major version matches. So, -i 12 will symlink to v12.0.0 or v12.0.1.

The CI fails may be related here. clang-tools find the exe_name with a major version, it's v12.0.0, so it did not download v12.0.1. but on Windows and macOS there is no v12, so it downloaded v12.0.1

clang-tools-pip/clang_tools/install.py

Line 33 in 1d42409

f"{tool_name}" + (f"-{ver_major}" if install_os != "windows" else "") + suffix

Then the CI runs which clang-format-12.0.1 it returns a non-zero value, so it failed.

[2bndy5]

I haven't said anything about this yet because I was hoping to get our fork of the static binaries repo setup properly.
We should really only be downloading the latest patched binaries and invoking them with only the major version number. So, -i 12 would download v12.0.1 and can be invoked with clang-*-12. It is odd that the Ubuntu runner ships with v12.0.0 instead of v12.0.1, but the LLVM project does seem to have a broad idea of what constitutes breaking changes

[shenxianpeng]

Yes, it's great if we can well maintain the fork binaries repo and fixed the existing issues.

[shenxianpeng]

We should really only be downloading the latest patched binaries and invoking them with only the major version number.

It looks good to me. The binaries repo provides both v12 and v12.0.1, so it also looks makes sense if users could download any version with clang-tools -i command.

[shenxianpeng]

I saw the following message in each action run when trying to create the symbolic link, this is because link directory does not exist when it trying to link to the target.

clang-tools-pip/clang_tools/install.py

Line 196 in 1d42409

link.symlink_to(target)

If we can create the dir before linking, the error will be gone. here is my test repo.

Found a installed version of clang-format: 12.0.0 at /usr/lib/llvm-12/bin/clang-format
Encountered an error when trying to create the symbolic link:
    No such file or directory
Found a installed version of clang-tidy: 12.0.0 at /usr/lib/llvm-12/bin/clang-tidy
Encountered an error when trying to create the symbolic link:
    No such file or directory

Found a installed version of clang-format: 10.0.0 at /usr/lib/llvm-10/bin/clang-format
Encountered an error when trying to create the symbolic link:
    No such file or directory
Found a installed version of clang-tidy: 10.0.0 at /usr/lib/llvm-10/bin/clang-tidy
Encountered an error when trying to create the symbolic link:
    No such file or directory

[2bndy5]

link is not directory. It is an absolute path to the dynamically linked binary which in this case is resolved() before being passed to create_symlink().

I suppose I could try asserting the strict parameter and see if that helps.

[2bndy5]

Found a installed version of clang-format: 12.0.0 at /usr/lib/llvm-12/bin/clang-format

The path seen at the end here is the value used as a target. the link path is created upon calling symlink_to()

[shenxianpeng]

link is not directory.

Sorry, it's install_dir does not exist.

clang-tools-pip/clang_tools/install.py

Line 174 in 1d42409

link = Path(install_dir) / (tool_name + suffix)

[2bndy5]

oh! I hate it when the problem is so simple and I missed it entirely. That should be an easy fix.

[shenxianpeng]

How about checking the minor (specific) version when the user input 12.0.1 otherwise only check the major version when user inputs 12, 11, 10, etc.

[2bndy5]

That seems fine. What if there is no dynamic binary, and we need to download the static binary? Should -i 12 fetch v12.0.1 or should it only get v12.0.0?

[shenxianpeng]

I think if there is no dynamic binary, we need to download the static binary, and -i 12 should only get v12.0.0.

[2bndy5]

Ok. I just asked because most install tools that are given only the major version will install the latest patched release.

[shenxianpeng]

Yes, that's true. Just our case only v12 has two versions, so I guess if people want to install any of them, they can.

[2bndy5]

💡 Maybe we can have a --greedy option to allow users to install the latest minor version (or uninstall all versions of a given major version).

[shenxianpeng]

I am just not sure the option name --greedy is common for most CLI. If so, I have no objection.

[2bndy5]

There are a set of commonly used symbols that are usually interpreted with importance in semantic versioning.

See PEP440 for how python uses certain symbols to describe a specified version. Instead of a new CLI option, we could support prefixed chars to the version number. For example, -i ~12 would install latest patched release v12.0.1, or -i ==12.0.0 would install exactly v12.0.0 only...

[shenxianpeng]

I like the idea to support prefixed chars to the version number instead of a new CLI option.

[shenxianpeng]

I see that new users are still experiencing 'stddef.h' file not found error, see https://github.com/Im-Rises/PhysicalEngine/actions/runs/3177551972
so I would like to merge your changes as soon as possible even though it does not support installing like PEP440, we can add this feature later if it's too late now. what's your thoughts?

[2bndy5]

Found out that we could use chocolatey to install LLVM binaries on Windows runners https://community.chocolatey.org/packages/llvm#versionhistory
The pkg manager chocolatey is already installed on the Windows runners.

[shenxianpeng]

Good find. this should solve the problem "'stddef.h' file not found" using choco to install LLVM binaries on windows, but it may take a little time to implement.
I tend to release what we have already implemented to reduce the impact on existing users.

[2bndy5]

Not sure about the future of this pkg since we could use

• choco for Windows,
• apt for Linux,
• and what could be used for Mac? Maybe homebrew? I don't give Apple any of my money, so I have no way to test homebrew locally.

All of these pkg managers could be used directly in the composite action's steps.

steps:
   - name: Install Binaries (Linux)
     if: runner.os == 'Linux'
     run: sudo apt-get install clang-tidy-${{ inputs.version }} clang-format-${{ inputs.version }}
   - name: Install Binaries (Windows)
     if: runner.os == 'Windows'
     # this may require the full version tag (`12.0.0` instead of `12`)
     run: choco install llvm --version ${{ inputs.version }} -y
   - name: Install Binaries (macOS)
     if: runner.os == 'macOS'
     # TBD
     run: homebrew install ???

[shenxianpeng]

The composite action's steps look good. I also don't have Mac to test, maybe we can use Mac runner on GitHub or Vagrant on local to test.
People don't use Windows and Mac as much as Linux, so we can take our time to deliver it.

[2bndy5]

To be clear, the patch that was just merged works for all supported platforms. Do you want me to publish v0.6.0, so we can avoid further "stddef.h" problems upstream?

[shenxianpeng]

We should be best to test the above composite action steps before we release for cpp-linter-action.

[2bndy5]

agreed. We can release this patch and the action should begin using it immediately (unless you pinned the action to certain version of clang-tools-pip).

[shenxianpeng]

Yes. the action uses this patch immediately. I also thought to pin the action to a certain version of clang-tools-pip, although we have to release more repos but users can get a stable behavior of cpp-linter-action.