cpp-linter / clang-tools-pip

Install clang-format, clang-tidy, clang-query and clang-apply-replacements binaries with clang-tools CLI.

clang-tools CLI

Install clang-format, clang-tidy, clang-query, and clang-apply-replacements binaries with clang-tools CLI.

Important

This package only manages binary executables (& corresponding symbolic links) that are installed using this package's executable script. It does not intend to change or modify any binary executable installed from other sources (like LLVM releases).

Features

• Binaries are statically linked for improved portability.
• Binaries can be specified installed for increased flexibility.
• Binaries are checked with SHA512 checksum. This ensures:
  1. Downloads are not corrupted.
  2. Old binary builds can be updated.

• Installed binaries are symbolically linked for better cross-platform usage. For example (on Windows), the clang-tidy-13.exe binary executable can also be invoked with the symbolic link titled clang-tidy.exe

  Note

  To create symbolic links on Windows, you must enable developer mode from the Windows settings under "Privacy & security" > "For developers" category.

• Customizable install path.

Install clang-tools CLI

Tip

It is recommended to use this package in a virtual environment.

# create the virtual env in the working directory
python -m venv env-name

# to activate on Linux:
source env-name/bin/activate

# to activate on Windows:
./env-name/Scripts/activate

This will ensure that

1. there are no permission problems when installing the tool
2. the installed path (for MacOS and Windows) is within the environment's variable PATH.

Install clang-tools command with pip

pip install clang-tools

Install clang-tools from git repo

pip install git+https://github.com/cpp-linter/clang-tools-pip.git@main

CLI Usage

For a list of supported Command Line Interface options, see the CLI documentation

Examples

Use clang-tools command to install version 13 binaries.

clang-tools --install 13

Or install to a specified directory

clang-tools --install 13 --directory .

Or install a specified tool, such as clang-format and clang-query version 14.

clang-tools --install 14 --tool clang-format clang-query

If the installed directory is in your path, you can run the installed tools.

clang-format-13 --version
clang-format version 13.0.0

clang-tidy-13 --version
LLVM (http://llvm.org/):
  LLVM version 13.0.0
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: skylake

Supported versions

clang-format, clang-tidy, clang-query, clang-apply-replacements

Version	18	17	16	15	14	13	12	11	10	9	8	7
Linux	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️
Windows	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️
macOS	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️

Warning

All clang-tidy v14+ builds for MacOS are still ~1.7 GB in size.

---

Thanks to the project clang-tools-static-binaries. We now used the fork repository that fixed the clang-tidy v14+ Segmentation fault (core dumped). see #56 for details.

clang-tools-pip's Issues

Add more tests to increase code coveage

Support download progress bar

originally posted by 2bndy5 in #3

However, those assets are around 500 MB each, and that might slow down this pkg's executable script (I wish there was a simple way to show a download progress bar).

I recently used the python std lib curses (a c-extension wrapper lib), but that's only natively available on linux. We'd have to either draw the bar manually or use rich (which is a bit bloated for our scenario). This stack overflow answer just shows a textual percentage...

Force overwriting the symlink to the installed binary did not work

When the specific version (v12) of clang-tools was installed, trying to force overwriting the symlink to the installed library did not work.

ubuntu@CN-L:~/clang-tools-pip$ clang-format --version
clang-format version 7.1.0 (tags/RELEASE_710/final)

ubuntu@CN-L:~/clang-tools-pip$ clang-tools -i 12 -f
clang-format-12 already installed
clang-tidy-12 already installed

ubuntu@CN-L:~/clang-tools-pip$ clang-format --version
clang-format version 7.1.0 (tags/RELEASE_710/final)

Root cause: here return the false directly, also need to check overwrite: bool

clang-tools-pip/clang_tools/install.py

Line 65 in a4f8c90

return False

add installed tools' binaries to PATH env var

It might be helpful to add the path to the installed binaries to the OS's PATH env var. Maybe add a CLI flag/switch that performs this?

Change default version to latest

add an `uninstall` CLI option

I've been really wanting this feature for testing purposes, but it will also be useful to users if we automate this ourselves.

• can be specified on CLI via -u or --uninstall
• remove the symlink if it targets the uninstalling binary
• perform uninstall action before doing install action. This will allow users to easily re-install binary if they so desire.
• uninstalling a non-existent binary should fail silently (don't raise an FileNotFoundError)

Build and packaing clang-tools binaries with this project

Better path defaults

This thread is for exploring less intrusive paths default values (paths that don't require admin privileges.

On Linux, the default is ~/.local/bin, but this path doesn't seem to be part of the PATH env var by default. On a fresh Ubuntu install, this path doesn't exist.

Currently, for macosx and windows, the default path is the same as the python executable. This isn't bad if the user is working in a venv, but it requires admin privileges otherwise.

I think we should do a check to see if the selected install path (either default or user-selected) is in the OS env var PATH. If it isn't, then we can warn the user that the install path is not in the PATH env var (similar to what pip does).

If the install path is not in the env var PATH, we might also want to skip the check to see if the tool is already installed.

clang-tools still download binary even native binary exist

clang-tools still download binary even native binary exists, and new created symbolic link does not link to the native binary.

Successfully installed clang-tools-0.6.1
gitpod /workspace/clang-tools-pip (main) $ which clang-format
/usr/bin/clang-format
gitpod /workspace/clang-tools-pip (main) $ clang-format --version
clang-format version 10.0.0-4ubuntu1 
gitpod /workspace/clang-tools-pip (main) $ clang-tools -i 10
Found a installed version of clang-format: 10.0.0 at /usr/lib/llvm-10/bin/clang-format
downloading clang-format (version 10)
    |████████████████████| 100% (of 2647984 bytes)
Installing clang-format-10 to /home/gitpod/.local/bin/
symbolic link created /home/gitpod/.local/bin/clang-format
Found a installed version of clang-tidy: 10.0.0 at /usr/lib/llvm-10/bin/clang-tidy
downloading clang-tidy (version 10)
    |████████████████████| 100% (of 34110384 bytes)
Installing clang-tidy-10 to /home/gitpod/.local/bin/
symbolic link created /home/gitpod/.local/bin/clang-tidy
gitpod /workspace/clang-tools-pip (main) $ 

gitpod /workspace/clang-tools-pip (main) $ ls -l /home/gitpod/.local/bin/clang-format
lrwxrwxrwx 1 gitpod gitpod 39 Oct 13 03:43 /home/gitpod/.local/bin/clang-format -> /home/gitpod/.local/bin/clang-format-10
gitpod /workspace/clang-tools-pip (main) $ ls -l /home/gitpod/.local/bin/clang-tidy
lrwxrwxrwx 1 gitpod gitpod 37 Oct 13 03:43 /home/gitpod/.local/bin/clang-tidy -> /home/gitpod/.local/bin/clang-tidy-10

prefer dynamically built binaries when already installed

Its come to my attention that the statically built binaries may not be respecting the native libc++ distribution installed in the github runners. I think we should go back to relying on the pre-existing binaries that are already installed because these tend not to have problems finding C/C++ std libs.

Here is a flowchart that illustrates my thought process:

Return friendly error message when not found or permission problem

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: renovate.json
Error type: The renovate configuration file contains some invalid settings
Message: Each Custom Manager must contain a non-empty fileMatch array

Add support clang-tools versions to readme

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: renovate.json
Error type: The renovate configuration file contains some invalid settings
Message: Each Custom Manager must contain a non-empty fileMatch array

The targer link is incorrect when clang-tools was not installed to the default path

For example, when the clang-tools were installed with apt-get install command (on Ubuntu)
Actual target path is: /usr/bin/clang-format-10

(myenv) ubuntu@CN-L:~/clang-tools-pip$ which clang-format-10
/usr/bin/clang-format-10
(myenv) ubuntu@CN-L:~/clang-tools-pip$ clang-tools -i 10
clang-format-10 already installed
clang-tidy-10 already installed

But expect target link is: /home/ubuntu/.local/bin/clang-format-10, see the code snippet

clang-tools-pip/clang_tools/install.py

Line 127 in a4f8c90

target = Path(install_dir) / f"{tool_name}-{version}{suffix}"

So program will not find a exist link under /home/ubuntu/.local/bin/, so it will create a link /home/ubuntu/.local/bin/clang-format to /home/ubuntu/.local/bin/clang-format-10 (it's a link not a executble), and it dons't work when run clang-format --version

Install with `clang-tools` command failed on Python 3.6.8

-sh-4.2$ clang-tools -i 12
Traceback (most recent call last):
  File "/home/xshen/.local/bin/clang-tools", line 8, in <module>
    sys.exit(main())
  File "/home/xshen/.local/lib/python3.6/site-packages/clang_tools/main.py", line 67, in main
    args.install, args.directory, args.overwrite, args.no_progress_bar
  File "/home/xshen/.local/lib/python3.6/site-packages/clang_tools/install.py", line 269, in install_clang_tools
    native_bin = is_installed(tool_name, version)
  File "/home/xshen/.local/lib/python3.6/site-packages/clang_tools/install.py", line 41, in is_installed
    [exe_name, "--version"], capture_output=True, check=True
  File "/usr/lib64/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
TypeError: __init__() got an unexpected keyword argument 'capture_output'
-sh-4.2$ python3 --version
Python 3.6.8

Inlcude clang-query and clang-apply-replacements in test

Automatically update cpp-linter/clang-tools-static-binaries tag when a new version is released

Right now I manually update the tag from install.py and test_url.py, see https://github.com/cpp-linter/clang-tools-pip/pull/61/files, so it would be great to update it automatically by tools, but this is a custom update.

I think GitHub dependence should not suit my needs. Updatecli or Renovate should be the best tools for custom update

Support new release of cpp-linter/clang-tools-static-binaries

support cpp-linter/clang-tools-static-binaries latest release master-a2d1eff4

Support clang-tools version 17

Clang version 17 has been supported in the latest release https://github.com/muttleyxd/clang-tools-static-binaries/releases

so the regular updates will be started later, please let me know if there's any objection.

validate latest build of binary using releases' SHA512 checksum

The releases that we download the binary from include a file that contains the SHA512 checksum calculated on the corresponding version of the tool. However, if there is a new release (which updates all builds of all versions), then it would be good to know if the installed binary executable needs to be updated.

I wrote a function to experiment with the releases' checksums that uses python's std hashlib:

from pathlib import Path
import hashlib

def verify_sha512(checksum: str, exe: str) -> bool:
    """Verify the executable binary's SHA512 hash matches the valid checksum.
    
    :param checksum: The path to the downloaded file containing the SHA512 checksum.
    :param exe: The path to the binary executable that is to be verified.

    :returns: `True` if the ``exe`` hash matches the ``checksum`` given,
        otherwise `False`.
    """
    valid_sum = Path(checksum).read_text(encoding="utf-8")
    if " " in valid_sum:
        valid_sum = valid_sum[: valid_sum.find(" ")]
    valid_hash = bytes(
        [
            int(valid_sum[i * 2 : i * 2 + 2], 16)
            for i in range(int(len(valid_sum) / 2), -1, -1)
        ]
    )
    bin_hash = hashlib.sha512(Path(exe).read_bytes()).digest()
    return valid_hash == bin_hash

This can be invoked using:

checksum_url = clang_tools_binary_url(tool_name, version).replace(".exe", "") + ".sha512sum"
checksum_file = download_file(checksum_url, f"{tool_name}-{version}.sha512sum")

# let `path_to_installed_exe` be declared elsewhere (depending on directory) assuming it exists.
is_valid = verify_sha512(checksum_file, path_to_installed_exe)

Switch to cpp-linter/clang-tools-static-binaries to download clang-tools

I think it's time to switch from muttleyxd/clang-tools-static-binaries to cpp-linter/clang-tools-static-binaries @2bndy5

We have known issues about clang-tidy-14, clang-tidy-15, and clang-tidy-16 having Segmentation faults on Ubuntu 22.02, but after we forked clang-tools-static-binaries repo, made some changes (dropped old version and upgrade build system version)rebuilt by ourselves, the Segmentation faults not existing anymore.

Here is my manual test on AlmaLinux release 8.9, our build clang-tidy-14 works.

# download from https://github.com/muttleyxd/clang-tools-static-binaries/releases/download/master-8f72ab3c/clang-tidy-14_linux-amd64 
bash-4.4$ ./clang-tidy-14_linux-amd64 --version
Segmentation fault (core dumped)

# download from https://github.com/cpp-linter/clang-tools-static-binaries/releases/download/master-be694ee7/clang-query-14_linux-amd64
bash-4.4$ ./clang-tidy-14_linux-amd64 --version
LLVM (http://llvm.org/):
  LLVM version 14.0.0
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: skylake-avx512
bash-4.4$

Please see more installation test on Linux, Windows, and MacOS from clang version 7 to 17

I don't exactly know the resean why this issue muttleyxd/clang-tools-static-binaries#18 was gone, the most likely reason is that we upgraded the Linux runner from the original ubuntu-20.04 to ubuntu-22.04.

Support clang-tools version 16

clang-tools version 16 has been released, let me support it?

https://github.com/muttleyxd/clang-tools-static-binaries/releases/tag/master-9ba48406

binary executables should use '.exe' suffix on Windows

I've been trying this out on my Windows machine, and The OS didn't know that the binaries were executable files because they didn't have the .exe file suffix.

Example after installing v12 in a python venv

Executing `clang-format-12` asked me what program to open the file with.

After changing .exe to the clang-format binary file's suffix, I got the expected behavior:

(venv) PS path\to\GitHub\cpp-linter-action> clang-format-12.exe --version
clang-format version 12.0.0

I'm guessing the fix for this would involve modifying the call to

clang-tools-pip/clang_tools/install.py

Line 66 in 4b49644

def move_and_chmod_binary(old_file_name, new_file_name, directory) -> None:

or simply adjusting the new_file_name arg for the Windows platform within the mv_and_chmod() function.

Other Windows considerations

os.chmod() has limited support on Windows. The docs specifically note:

Although Windows supports chmod(), you can only set the file’s read-only flag with it (via the stat.S_IWRITE and stat.S_IREAD constants or a corresponding integer value). All other bits are ignored.

I'm not entirely familiar with Unix permission's octal format, but using 0o755 didn't raise any exceptions.

---

ps - I also would like it if clang-tools -i 12 installed 12.0.1 instead of the older 12.0.0

Add test workflow to test all clang-tools version on all platforms

specify which tool(s) to install

It would be nice to let the user choose which tool(s) to install. This way we could only install clang-tidy in the cpp-linter-action (composite form) if inputs.tidy-checks != '-*. Or we could avoid installing clang-format if the inputs.style != ''.

Since the releases of binary builds also include the clang-query tool, users could specify that as well.

Implementation

Option 1

clang-tools -t clang-format -t clang-tidy -t clang-query

Option 2

clang-tools -t clang-format,clang-tidy,clang-query

I haven't decided which way to support. Maybe there's a way to support either or both options.

I would expect the current default ( clang-tidy + clang-format) to be overridden when -t is specified.

disable usage output when running `clang-tools --uninstall`

clang-tools --uninstall needs to check if it works, at least do not print the usage like it seems not to work.

 $ clang-tools -u 13
Uninstalling version 13 from /home/gitpod/.local/bin/
Nothing to do because `--install` and `--uninstall` was not specified.
usage: clang-tools [-h] [-i VERSION] [-t TOOL [TOOL ...]] [-d DIR] [-f] [-b] [-u VERSION]

options:
  -h, --help            show this help message and exit
  -i VERSION, --install VERSION
                        Install clang-tools about a specific version.
  -t TOOL [TOOL ...], --tool TOOL [TOOL ...]
                        Specify which tool(s) to install.
  -d DIR, --directory DIR
                        The directory where the clang-tools are installed.
  -f, --overwrite       Force overwriting the symlink to the installed binary. This will only overwrite an existing symlink.
  -b, --no-progress-bar
                        Do not display a progress bar for downloads.
  -u VERSION, --uninstall VERSION
                        Uninstall clang-tools with specific version. This is done before any install.

Support clang-tools version 15

Yesterday clang-tools-static-binaries support version 15 https://github.com/muttleyxd/clang-tools-static-binaries/releases

Maybe we should also support this version later. it should be best to have enough tests before we use the new release for cpp-linter-action.

Test workflow was not triggered after PR merged into the main branch

Pls refer to https://github.com/cpp-linter/clang-tools-pip/actions/workflows/python-test.yml

add option to disable progress bar

The CI log shown in the composite action tests show multiple lines for the progress bar because the log output does not support a "return feed" character (\r).

example workflow log

Run clang-tools -i 12
downloading clang-format (version 12)
    |                    |0% (of 2295920 bytes)
    |█                   |5% (of 2295920 bytes)
    |██                  |10% (of 2295920 bytes)
    |███                 |15% (of 2295920 bytes)
    |████                |20% (of 2295920 bytes)
    |█████               |25% (of 2295920 bytes)
    |██████              |30% (of 2295920 bytes)
    |███████             |35% (of 2295920 bytes)
    |████████            |40% (of 2295920 bytes)
    |█████████           |45% (of 2295920 bytes)
    |██████████          |50% (of 2295920 bytes)
    |███████████         |55% (of 2295920 bytes)
    |████████████        |60% (of 2295920 bytes)
    |█████████████       |65% (of 2295920 bytes)
    |██████████████       |70% (of 2295920 bytes)
    |███████████████     |75% (of 2295920 bytes)
    |████████████████    |80% (of 2295920 bytes)
    |█████████████████    |85% (of 2295920 bytes)
    |██████████████████  |90% (of 2295920 bytes)
    |███████████████████  |95% (of 2295920 bytes)
    |████████████████████| 100% (of 2295920 bytes)
Installing clang-format-12 to /home/runner/.local/bin/
symbolic link created /home/runner/.local/bin/clang-format
downloading clang-tidy (version 12)
    |                    |0% (of 38761872 bytes)
    |                    |4% (of 38761872 bytes)
    |█                   |9% (of 38761872 bytes)
    |██                  |14% (of 38761872 bytes)
    |███                 |19% (of 38761872 bytes)
    |████                |24% (of 38761872 bytes)
    |█████               |29% (of 38761872 bytes)
    |██████              |34% (of 38761872 bytes)
    |███████             |39% (of 38761872 bytes)
    |████████            |44% (of 38761872 bytes)
    |█████████           |49% (of 38761872 bytes)
    |██████████          |54% (of 38761872 bytes)
    |███████████         |59% (of 38761872 bytes)
    |████████████        |64% (of 38761872 bytes)
    |█████████████       |69% (of 38761872 bytes)
    |██████████████      |74% (of 38761872 bytes)
    |███████████████     |79% (of 38761872 bytes)
    |████████████████    |84% (of 38761872 bytes)
    |█████████████████   |89% (of 38761872 bytes)
    |██████████████████  |94% (of 38761872 bytes)
    |███████████████████ |99% (of 38761872 bytes)
    |████████████████████| 100% (of 38761872 bytes)
Installing clang-tidy-12 to /home/runner/.local/bin/
symbolic link created /home/runner/.local/bin/clang-tidy

Other tools that use progress bars (like pip) also offer a CLI option to disable the progress bars, so I think this is an adequate addition for this pkg as well.

Desired Implementation

clang-tools --no-progress-bar

or using a short switch:

clang-tools -b

capitalize the first letter of some print

I used https://github.com/cpp-linter/cpp-linter-hooks and found the clang-tools output format output is inconsistent.

I would suggest capitalizing the first letter. e.g change downloading to Downloading

$ pre-commit run --all-files
clang-format.............................................................Failed
- hook id: clang-format
- exit code: 1

downloading clang-format (version 13)
Installing clang-format-13 to /home/xshen/.cache/pre-commit/repol6kggrfo/py_env-python3.11/bin
[Errno 2] No such file or directory: '/home/xshen/.cache/pre-commit/repol6kggrfo/py_env-python3.11/bin/clang-format'

clang-tidy...............................................................Failed
- hook id: clang-tidy
- exit code: 1

downloading clang-tidy (version 13)
Installing clang-tidy-13 to /home/xshen/.cache/pre-commit/repol6kggrfo/py_env-python3.11/bin
[Errno 2] No such file or directory: '/home/xshen/.cache/pre-commit/repol6kggrfo/py_env-python3.11/bin/clang-tidy'