[7.2.0rc2 regression] `testpath.assert_calls` (python package) can't mock subprocess calls for deeply nested files about bazel

[dougthor42]

Two other things I forgot to mention:

• this also happens in rc3
• No additional info in the Google Bazel Users chat yet.

[fmeum]

I can't reproduce this on regular Ubuntu. Could you post the output of running with --announce_rc?

You could also bisect the failure by running Bazelisk with --bisect=7.2.0rc1..7.2.0rc2.

[meteorcloudy]

I also couldn't reproduce on macOS

pcloudy@pcloudy-macbookpro2:~/workspace/tmp/gh-22664
$ bazel --version
2024/06/07 10:33:52 Downloading https://releases.bazel.build/7.2.0/rc2/bazel-7.2.0rc2-darwin-arm64...
Downloading: 63 MB out of 63 MB (100%)
bazel 7.2.0rc2
pcloudy@pcloudy-macbookpro2:~/workspace/tmp/gh-22664
$ bazel test //...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: f6e06514-a6d7-4e1e-8368-716ee44fd328
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=210
INFO: Reading rc options for 'test' from /Users/pcloudy/workspace/tmp/gh-22664/.bazelrc:
  Inherited 'common' options: --enable_bzlmod --nolegacy_external_runfiles
INFO: Reading rc options for 'test' from /Users/pcloudy/.bazelrc:
  Inherited 'build' options: --verbose_failures --announce_rc --disk_cache=/tmp/bazel_disk_cache --repository_cache=/tmp/bazel_repository_cache --experimental_stats_summary
INFO: Reading rc options for 'test' from /Users/pcloudy/workspace/tmp/gh-22664/.bazelrc:
  'test' options: --test_output=errors
INFO: Reading rc options for 'test' from /Users/pcloudy/.bazelrc:
  'test' options: -k --test_output=errors
INFO: Analyzed 2 targets (98 packages loaded, 3159 targets configured).
INFO: Found 1 target and 1 test target...
Target //src/one/two/three:test_foo up-to-date:
  bazel-bin/src/one/two/three/test_foo
INFO: 6 processes: 4 internal, 2 darwin-sandbox.
INFO: Total action wall time 1.10s
INFO: Execution critical path 1.55s (setup 0.00s, action wall time 0.99s)
INFO: Elapsed time 34.98s (preparation 7.84s, execution 27.14s)
INFO: Build completed successfully, 6 total actions
//src/one/two/three:test_foo                                             PASSED in 1.0s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

[meisterT]

I was able to repro and bisect, culprit is d98b02e

[meisterT]

cc @oquenchil

[meisterT]

I can't reproduce this on regular Ubuntu. Could you post the output of running with --announce_rc?

$ bazelisk test --announce_rc //...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=282
INFO: Reading rc options for 'test' from /usr/local/google/home/twerth/t/.bazelrc:
  Inherited 'common' options: --enable_bzlmod --nolegacy_external_runfiles
INFO: Reading rc options for 'test' from /usr/local/google/home/twerth/t/.bazelrc:
  'test' options: --test_output=errors
INFO: Analyzed 2 targets (97 packages loaded, 3244 targets configured).
FAIL: //src/one/two/three:test_foo (see /usr/local/google/home/twerth/.cache/bazel/_bazel_twerth/8160046761d3783ee70008ffde0e375e/execroot/_main/bazel-out/k8-fastbuild/testlogs/src/one/two/three/test_foo/test.log)
...

[fmeum]

@meisterT Could you share the failure with --sandbox_debug?

[fmeum]

I can repro if I make the path of the containing package longer. Are we having long path issues on Unix now? Will investigate.

[fmeum]

testpath emits the path to the Python interpreter into a shebang, which is subject to length limitations set by the kernel at build time. Sandboxed paths happened to be shorter in Bazel 7.0.0 through 7.2.0rc1 due to the bind mounting scheme used by --incompatible_sandbox_hermetic_tmp, but I can reproduce the same failure if I just make the path a bit longer.

I will send a fix to testpath as other Bazel users may run into this, but it's not a bug in Bazel.

[fmeum]

I sent jupyter/testpath#28.

[meteorcloudy]

@fmeum Thanks for investigating!

[dougthor42]

Wow you people are good! Thanks so much! 💚

Also, TIL about bazelisk --bisect. Handy! (though now I see that it's part of the Bug instructions... my bad! 😬 )

[meteorcloudy]

@dougthor42 No worry! I'm updating the bug template to make it more prominent! Thanks for the report again!