feature `sha1/asm` is enabled on windows about gitoxide

[NobodyXu]

Yeah setting:

# Set the default for dependencies on debug.
[profile.dev.package."*"]
opt-level = "z"

speedup the cloning to 2m.
I guess it's time for binstall to use O2 or even O3 given that gix really need all that optimization.

[NobodyXu]

Some other strangeness is that CI already builds that platform - it will install the max version of gix which activates max-performance as well, and there it works as if the target-specific settings in gix-features do work, sometimes. The only difference I see is that it builds with --debug instead of --release.

Yeah that's really strange, because our CI also build binstall in debug and release and they both failed.

When looking at the Cargo.toml of the sha1 crate, I see that they also try to make that available only on targets that are known to work, so something really isn't working as expected there, sometimes.

Maybe an avenue would be to understand why my CI seems to work on that environment, whereas the one at cargo-binstall doesn't? What's the difference, ideally this can be reproduced here. In the meantime, I think it's worth trying to workaround in cargo-binstall as well by selectively using max-performance-safe instead?

Yeah, I will have another look at it once I'm free, it's really really strange.

[Byron]

Maybe there is a way to narrow down the cause of this by adding gix to a proxy-manifest that is freshly initialized, to see if it still somehow gets sha1-asm activated. If not, one can probably try to make the proxy more similar to the actual cargo-binstall manifest to learn what's causing this.

[NobodyXu]

I was able to reproduce it on my arch64 windows by running JUST_EXTRA_FEATURES=git-max-perf just build on branch speedup/gix:

error: failed to run custom build command for `sha1-asm v0.5.1`

Caused by:
  process didn't exit successfully: `C:\Users\nobodyxu\cargo-binstall\target\debug\build\sha1-asm-75aa7a029eab4153\build-script-build` (exit code: 1)
  --- stdout
  TARGET = Some("aarch64-pc-windows-msvc")
  OPT_LEVEL = Some("0")
  HOST = Some("aarch64-pc-windows-msvc")
  cargo:rerun-if-env-changed=CC_aarch64-pc-windows-msvc
  CC_aarch64-pc-windows-msvc = None
  cargo:rerun-if-env-changed=CC_aarch64_pc_windows_msvc
  CC_aarch64_pc_windows_msvc = None
  cargo:rerun-if-env-changed=HOST_CC
  HOST_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CFLAGS_aarch64-pc-windows-msvc
  CFLAGS_aarch64-pc-windows-msvc = None
  cargo:rerun-if-env-changed=CFLAGS_aarch64_pc_windows_msvc
  CFLAGS_aarch64_pc_windows_msvc = None
  cargo:rerun-if-env-changed=HOST_CFLAGS
  HOST_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("neon")
  DEBUG = Some("true")
  running: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.36.32532\\bin\\HostX86\\arm64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-W4" "-march=armv8-a+crypto" "-c" "-o" "C:\\Users\\nobodyxu\\cargo-binstall\\target\\aarch64-pc-windows-msvc\\debug\\build\\sha1-asm-600eae65c368c0c1\\out\\src/aarch64.o" "src/aarch64.S"
  cargo:warning=cl : Command line warning D9035 : option 'o' has been deprecated and will be removed in a future release
  cargo:warning=cl : Command line warning D9002 : ignoring unknown option '-march=armv8-a+crypto'
  cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/aarch64.S', object file assumed
  cargo:warning=cl : Command line warning D9027 : source file 'src/aarch64.S' ignored
  cl : Command line warning D9021 : no action performed
  exit code: 0
  cargo:rerun-if-env-changed=AR_aarch64-pc-windows-msvc
  AR_aarch64-pc-windows-msvc = None
  cargo:rerun-if-env-changed=AR_aarch64_pc_windows_msvc
  AR_aarch64_pc_windows_msvc = None
  cargo:rerun-if-env-changed=HOST_AR
  HOST_AR = None
  cargo:rerun-if-env-changed=AR
  AR = None
  cargo:rerun-if-env-changed=ARFLAGS_aarch64-pc-windows-msvc
  ARFLAGS_aarch64-pc-windows-msvc = None
  cargo:rerun-if-env-changed=ARFLAGS_aarch64_pc_windows_msvc
  ARFLAGS_aarch64_pc_windows_msvc = None
  cargo:rerun-if-env-changed=HOST_ARFLAGS
  HOST_ARFLAGS = None
  cargo:rerun-if-env-changed=ARFLAGS
  ARFLAGS = None
  running: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.36.32532\\bin\\HostX86\\arm64\\lib.exe" "-out:C:\\Users\\nobodyxu\\cargo-binstall\\target\\aarch64-pc-windows-msvc\\debug\\build\\sha1-asm-600eae65c368c0c1\\out\\libsha1.a" "-nologo" "C:\\Users\\nobodyxu\\cargo-binstall\\target\\aarch64-pc-windows-msvc\\debug\\build\\sha1-asm-600eae65c368c0c1\\out\\src/aarch64.o"
  LINK : fatal error LNK1181: cannot open input file 'C:\Users\nobodyxu\cargo-binstall\target\aarch64-pc-windows-msvc\debug\build\sha1-asm-600eae65c368c0c1\out\src\aarch64.o'
  exit code: 1181

  --- stderr


  error occurred: Command "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.36.32532\\bin\\HostX86\\arm64\\lib.exe" "-out:C:\\Users\\nobodyxu\\cargo-binstall\\target\\aarch64-pc-windows-msvc\\debug\\build\\sha1-asm-600eae65c368c0c1\\out\\libsha1.a" "-nologo" "C:\\Users\\nobodyxu\\cargo-binstall\\target\\aarch64-pc-windows-msvc\\debug\\build\\sha1-asm-600eae65c368c0c1\\out\\src/aarch64.o" with args "lib.exe" did not execute successfully (status code exit code: 1181).


error: Recipe `build` failed on line 204 with exit code 101

(in additional to failure to build ring due to clang not installed).

[NobodyXu]

It's so interesting that these compiler tools are visible in a standard git-bash shell. When I tried it, no PATH was set and no development tool was available. Maybe something is different when installing the whole package like seen in the screenshot.

Yeah it look like an env with patched bash and there're a few catch with it:

• if you run a built-in shell function, says mktemp, it will return a path looks like unix (e.g. /tmp/...)
• if you pass it to other built-ins, like cd, then it will be handled correctly
• if you pass it to other external tools, like cargo-binstall then it malfunctions as it cannot find the path

BTW, the windows CI on GHA demonstrates the same behaviour, so I suppose they might be also using the Git Bash.

[Byron]

Thanks a lot for letting me know! Is this something you would feel comfortable to contribute? It seems like this project's CI is missing that environment or configuration. I will also take a stab at it and at least try to get a reproduction on this project's CI, from there fixing it should get easier.

[NobodyXu]

Yes, I will have a try once I have time.
Reproducing it in CI should be easy.

[NobodyXu]

To be honest, I think this might be a upstream bug.
I googled it and this discussion from 2015 says target_env is internal to compiler, but it's clearly documented in user facing document.

Given that gix-features will not enable sha1/asm unless not(target_env = "msvc") and on x86/arm, it could only be an upstream bug.

[NobodyXu]

I think we can workaround this by doing not(target = "x86_64-windows-msvc"), not(target = "aarch64-windows-msvc") instead of not(target_env = "msvc") given that sha1/asm only supports arm 64 and x86 anyway.

But that will require quite a list of targets since there are many x86 targets (i686, i484)

[Byron]

What I don't understand about this is that as per the gix-features/Cargo.toml, it already excludes the respective environment:

[target.'cfg(all(any(target_arch = "aarch64", target_arch = "x86", target_arch = "x86_64"), not(target_env = "msvc")))'.dependencies]
sha1 = { version = "0.10.0", optional = true, features = ["asm"] }

Clearly, x86_64-pc-windows-msvc should not enable the asm feature, instead it should only pull in the sha1 crate without additional features.

To me this looks like something cargo gets wrong, but maybe I am just not getting how dependencies work with more complex per-target overrides.

My recommendation, in the meantime, is to use max-performance-safe for these platforms, and maybe do that based on target specific dependencies in cargo-binstall instead?

Some other strangeness is that CI already builds that platform - it will install the max version of gix which activates max-performance as well, and there it works as if the target-specific settings in gix-features do work, sometimes. The only difference I see is that it builds with --debug instead of --release.

When looking at the Cargo.toml of the sha1 crate, I see that they also try to make that available only on targets that are known to work, so something really isn't working as expected there, sometimes.

Maybe an avenue would be to understand why my CI seems to work on that environment, whereas the one at cargo-binstall doesn't? What's the difference, ideally this can be reproduced here. In the meantime, I think it's worth trying to workaround in cargo-binstall as well by selectively using max-performance-safe instead?

[NobodyXu]

Clearly, x86_64-pc-windows-msvc should not enable the asm feature, instead it should only pull in the sha1 crate without additional features.

Yeah I think this could be a bug in cargo.

My recommendation, in the meantime, is to use max-performance-safe for these platforms, and maybe do that based on target specific dependencies in cargo-binstall instead?

I agree, though I would probably still turns on flate2/zlib-ng since in cargo-bins/cargo-binstall#1184 we turned on support for custom registry and test it by pulling the crates.io git index.

It took much longer than I'd imagine, on my M1 it took at least 2m and on CI it took 11m.

[Byron]

It took much longer than I'd imagine, on my M1 it took at least 2m and on CI it took 11m.

You mean shallow cloning took 2m and 11m respectively?

It should be much faster, even without zlib-ng I'd expect the time dominated by transfer, and not pack resolution.

❯ gix clone --depth 1 https://github.com/rust-lang/crates.io-index --bare
 08:58:05 read pack done 82.7MB in 45.16s (1.8MB/s)
 08:58:05  indexing done 144.8k objects in 26.77s (5.4k objects/s)
 08:58:05 decompressing done 530.4MB in 26.77s (19.8MB/s)
 08:58:05     Resolving done 144.8k objects in 0.38s (382.3k objects/s)
 08:58:05      Decoding done 2.5GB in 0.38s (6.6GB/s)
 08:58:06 writing index file done 4.1MB in 0.01s (671.2MB/s)
 08:58:06  create index file done 144.8k objects in 27.17s (5.3k objects/s)
HEAD:refs/remotes/origin/HEAD (implicit)
        05fe54b35765137db5650c546a9efc5899cb378b HEAD symref-target:refs/heads/master -> refs/remotes/origin/HEAD [new]
+refs/heads/*:refs/remotes/origin/*
        05fe54b35765137db5650c546a9efc5899cb378b refs/heads/master -> refs/remotes/origin/master [new]

git ( master) [?] via 🐍 took 46s
❯ gix clone https://github.com/rust-lang/crates.io-index --bare crates-io-full
 08:58:52 read pack done 310.7MB in 29.54s (10.5MB/s)
 08:58:52  indexing done 531.1k objects in 29.17s (18.2k objects/s)
 08:58:52 decompressing done 1.8GB in 29.17s (61.9MB/s)
 08:58:54     Resolving done 531.1k objects in 1.59s (333.7k objects/s)
 08:58:54      Decoding done 18.4GB in 1.59s (11.6GB/s)
 08:58:54 writing index file done 14.9MB in 0.02s (638.8MB/s)
 08:58:54  create index file done 531.1k objects in 30.85s (17.2k objects/s)
HEAD:refs/remotes/origin/HEAD (implicit)
        05fe54b35765137db5650c546a9efc5899cb378b HEAD symref-target:refs/heads/master -> refs/remotes/origin/HEAD [new]
+refs/heads/*:refs/remotes/origin/*
        05fe54b35765137db5650c546a9efc5899cb378b refs/heads/master -> refs/remotes/origin/master [new]

Above it shows that --depth 1 is actually slower than a full clone as the remote has more work.

[NobodyXu]

You mean shallow cloning took 2m and 11m respectively?

Yes, I think it could be because the test is run in debug mode.

Even in release, cargo-binstall is compiled using O2, so a better option might be to always compile gix and its dependencies with O3, probably more reliable than max-performance.

[NobodyXu]

Yeah, I will have another look at it once I'm free, it's really really strange.

Building gix directly with:

cargo build --features max-performance,blocking-http-transport-reqwest-rust-tls --target x86_64-pc-windows-msvc

always succeed in dev and release mode, with or without blocking-http-transport-reqwest-rust-tls.

But somehow when building from cargo-binstall, it failed due to sha1-asm.

[NobodyXu]

Running cargo tree -i sha1-asm --target x86_64-pc-windows-msvc in cargo-binstall gives:

sha1-asm v0.5.1
└── sha1 v0.10.5
    └── gix-features v0.31.1
        ├── gix v0.48.0
        │   └── binstalk v0.13.0 (/Users/nobodyxu/Dev/cargo-binstall/crates/binstalk)
        ├── gix-commitgraph v0.17.1
        │   ├── gix v0.48.0 (*)
        │   ├── gix-negotiate v0.4.0
        │   │   └── gix v0.48.0 (*)
        │   ├── gix-revwalk v0.3.0
        │   │   ├── gix-negotiate v0.4.0 (*)
        │   │   ├── gix-revision v0.17.0
        │   │   │   ├── gix v0.48.0 (*)
        │   │   │   └── gix-refspec v0.13.0
        │   │   │       └── gix v0.48.0 (*)
        │   │   └── gix-traverse v0.29.0
        │   │       ├── gix v0.48.0 (*)
        │   │       ├── gix-index v0.20.0
        │   │       │   ├── gix v0.48.0 (*)
        │   │       │   └── gix-worktree v0.21.1
        │   │       │       └── gix v0.48.0 (*)
        │   │       └── gix-pack v0.39.1
        │   │           ├── gix v0.48.0 (*)
        │   │           └── gix-odb v0.49.1
        │   │               └── gix v0.48.0 (*)
        │   └── gix-traverse v0.29.0 (*)
        ├── gix-config v0.25.1
        │   └── gix v0.48.0 (*)
        ├── gix-fs v0.3.0
        │   ├── gix v0.48.0 (*)
        │   ├── gix-ref v0.32.1
        │   │   ├── gix v0.48.0 (*)
        │   │   ├── gix-config v0.25.1 (*)
        │   │   └── gix-discover v0.21.1
        │   │       └── gix v0.48.0 (*)
        │   ├── gix-tempfile v7.0.0
        │   │   ├── gix v0.48.0 (*)
        │   │   ├── gix-lock v7.0.1
        │   │   │   ├── gix v0.48.0 (*)
        │   │   │   ├── gix-index v0.20.0 (*)
        │   │   │   └── gix-ref v0.32.1 (*)
        │   │   ├── gix-pack v0.39.1 (*)
        │   │   └── gix-ref v0.32.1 (*)
        │   └── gix-worktree v0.21.1 (*)
        ├── gix-glob v0.9.1
        │   ├── gix v0.48.0 (*)
        │   ├── gix-attributes v0.14.1
        │   │   ├── gix v0.48.0 (*)
        │   │   └── gix-worktree v0.21.1 (*)
        │   ├── gix-config v0.25.1 (*)
        │   ├── gix-ignore v0.4.1
        │   │   ├── gix v0.48.0 (*)
        │   │   └── gix-worktree v0.21.1 (*)
        │   └── gix-worktree v0.21.1 (*)
        ├── gix-index v0.20.0 (*)
        ├── gix-object v0.32.0
        │   ├── gix v0.48.0 (*)
        │   ├── gix-diff v0.32.0
        │   │   ├── gix v0.48.0 (*)
        │   │   └── gix-pack v0.39.1 (*)
        │   ├── gix-index v0.20.0 (*)
        │   ├── gix-negotiate v0.4.0 (*)
        │   ├── gix-odb v0.49.1 (*)
        │   ├── gix-pack v0.39.1 (*)
        │   ├── gix-ref v0.32.1 (*)
        │   ├── gix-revision v0.17.0 (*)
        │   ├── gix-revwalk v0.3.0 (*)
        │   ├── gix-traverse v0.29.0 (*)
        │   └── gix-worktree v0.21.1 (*)
        ├── gix-odb v0.49.1 (*)
        ├── gix-pack v0.39.1 (*)
        ├── gix-protocol v0.35.0
        │   └── gix v0.48.0 (*)
        ├── gix-ref v0.32.1 (*)
        ├── gix-transport v0.33.1
        │   ├── gix v0.48.0 (*)
        │   └── gix-protocol v0.35.0 (*)
        ├── gix-url v0.20.1
        │   ├── gix v0.48.0 (*)
        │   ├── gix-credentials v0.16.1
        │   │   ├── gix v0.48.0 (*)
        │   │   ├── gix-protocol v0.35.0 (*)
        │   │   └── gix-transport v0.33.1 (*)
        │   └── gix-transport v0.33.1 (*)
        └── gix-worktree v0.21.1 (*)

But doing the same in gix gives:

$ cargo tree --target x86_64-pc-windows-msvc --features max-performance -i sha1-asm
warning: nothing to print.

To find dependencies that require specific target platforms, try to use option `--target all` first, and t
hen narrow your search scope accordingly.

[NobodyXu]

Maybe there is a way to narrow down the cause of this by adding gix to a proxy-manifest that is freshly initialized, to see if it still somehow gets sha1-asm activated. If not, one can probably try to make the proxy more similar to the actual cargo-binstall manifest to learn what's causing this.

I am trying to do this, but it's really hard to setup env on windows.

Specifically, I have problem installing clang, I installed the clang component in vs installer but still doesn't work.

[Byron]

That's interesting, I thought such an environment is already available on the CI of cargo-binstall?

I also failed to setup the microsoft toolchain, and just barely got my own VM to work with the GNU toolchain, if that's somehow related. In any case, I wouldn't be able to help here :/.

[NobodyXu]

That's interesting, I thought such an environment is already available on the CI of cargo-binstall?

Yeah I could just use the Github Action for this.
Thanks, I would probably setup a new github repo for this.

[NobodyXu]

This error I got on my aarch64 windows is very similar to our CI, it failed at link-time after build.rs is run.

Looking at the sha1-asm build.rs, its use of cc::Build::compile is deprecated:

For backwards compatibility, if output starts with “lib” and ends with “.a”, a second “lib” prefix and “.a” suffix do not get added on, but this usage is deprecated; please omit lib and .a in the argument that you pass.

It should pass just a name instead and msvc would deal with that:

The output string argument determines the file name for the compiled library. The Rust compiler will create an assembly named “lib”+output+“.a”. MSVC will create a file named output+“.lib”.

[Byron]

Thanks so much for pushing forward on the resolution for this issue, I appreciate it! Being able to reproduce it locally makes a fix possible! I never managed to get a VM that has MSVC installed (and I tried), it never worked with Rust and I thought the whole process is insane.

Looking at the sha1-asm build.rs, its use of cc::Build::compile is deprecated:

It's certainly worth fixing that as it may help to make it build better in general, but I have some doubts that it's going to make a difference as the problem is that it tries to use these ASM files at all. As far as I know, they don't exist for all platforms and builds should never be attempted, falling into the realm of cargo features. I absolute hope I am misunderstanding this though, it would be great if the attached PR could fix this.

Let me again thank you for your efforts and apologize for just cheering you on from the side-lines. I think once this issue is fixed for cargo-binstall, it will also be fixed for every current and future user of gitoxide which is a big deal for me. This build error has been popping up over and over again and it could not be resolved thus far.

[NobodyXu]

Thanks so much for pushing forward on the resolution for this issue, I appreciate it!

You are welcome!

Being able to reproduce it locally makes a fix possible! I never managed to get a VM that has MSVC installed (and I tried), it never worked with Rust and I thought the whole process is insane.

It's quite hard to setup dev env on windows.
I used visual studio installer to select the following packages:

It's certainly worth fixing that as it may help to make it build better in general, but I have some doubts that it's going to make a difference as the problem is that it tries to use these ASM files at all.

Yeah, though I still think it might help, since they also use the / on windows where it is supposed to use \.

[Byron]

Thanks for your help with this! I remember that I tried to reduce the amount of checkboxes on the right to safe space, but maybe that's the cause of my trouble in the first place.

Could you also tell me how you get a development shell that also includes git? I think that might also have been the reason it never worked out for me - either I have git, or the MS tool suite, but not both (maybe).

Thanks again

[NobodyXu]

Could you also tell me how you get a development shell that also includes git? I think that might also have been the reason it never worked out for me - either I have git, or the MS tool suite, but not both (maybe).

I uses "Git Bash", downloaded from https://git-scm.com/download/win

[NobodyXu]

I also installed strawberry perl for perl and cygwin.

[Byron]

It's so interesting that these compiler tools are visible in a standard git-bash shell. When I tried it, no PATH was set and no development tool was available. Maybe something is different when installing the whole package like seen in the screenshot.