Remote execution can't find executable about buck2

[yaoddao]

I met the same question #549 and #550. I think the reason of the problem may be the cooperation between Buck2 and Buildbarn has a problem.

[yaoddao]

I think：

• When using relative paths, eg. 'sh', Buck2 sends 'sh' and path '' to Buildbarn, Buildbarn can not search 'sh' in path ''.
• When using absolute paths, eg. '/usr/bin/sh', Buck2 sends '/usr/bin/sh' and path '' to Buildbarn, Buildbarn directly use '/usr/bin/sh' to run.
• But Bazel does not have this problem, so I suspect Bazel not only sent 'sh' but also additional path information to Buildbarn, when using relative paths 'sh'. This additional information can let Buildbarn to find 'sh' in PATH environment.

[yaoddao]

I think the reason of the problem must be a cooperation problem between Buck2 and Buildbarn.
The cooperation between Buck2 and Buildbuddy does not have the same question.
When buck2 sends Linux command, eg. g++,sh, and so on, Buildbarn only searches the command in path "", rather than in PATH.
But when buck2 sends Linux command, Buildbuddy always searches the command in PATH.

Also, the cooperation between Bazel and Buildbarn also does not have the same question. I think that Bazel not only sends Linux the command but also additional information to Buildbarn. So Buildbarn can search for the command correctly

[stagnation]

To confirm, does buck2 send an empty $PATH variable?
https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto#L591

Here are two reference images for Buildbarn to illustrate the problem:

Standard path, nothing fancy (arbitrary action)

Path is empty:
bazel build --action_env PATH='' ...

[yaoddao]

Hi @stagnation ,
Greatly appreciate your reply. I tested this issue using Bazel. The conclusion is that Bazel has the same problem when using --action_env PATH='', empty PATH.

Reproduce:

• Step 1:

cd ./bb-deployments
./bb-deployments (master)>bazel build --action_env PATH='' --config=remote-ubuntu-22-04 @abseil-hello//:hello_main
WARNING: Output base '/../.cache/bazel/_bazel_rxie/9de92e33c21ddad2a6644b0b9fa6f76a' is on NFS. This may lead to surprising failures and undetermined behavior.
INFO: Invocation ID: 38741df3-0015-4812-a835-11382b08a410
INFO: Build option --action_env has changed, discarding analysis cache.
INFO: Analyzed target @abseil-hello//:hello_main (54 packages loaded, 424 targets configured).
INFO: Found 1 target...
ERROR: /../.cache/bazel/_bazel_rxie/9de92e33c21ddad2a6644b0b9fa6f76a/external/com_google_absl/absl/base/BUILD.bazel:232:11: Compiling absl/base/internal/unscaledcycleclock.cc failed: (Exit 1): gcc failed: error executing command (from target @com_google_absl//absl/base:base) /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 33 arguments skipped)
Action details (uncached result): http://localhost:7984/hardlinking/blobs/sha256/historical_execute_response/4ef62c64b6c9fb5f6cef00f6e0f0b36ddf7df3fff7040c5a18d66745c87b5443-1063/
gcc: fatal error: cannot execute 'as': execvp: No such file or directory
compilation terminated.
Target @abseil-hello//:hello_main failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 2.912s, Critical Path: 0.20s
INFO: 32 processes: 32 internal.
FAILED: Build did NOT complete successfully
../bb-deployments (master)>

• Step 2:
  

• Step 3

./bb-deployments (master)>bazel build --action_env PATH='/usr/bin' --config=remote-ubuntu-22-04 @abseil-hello//:hello_main
WARNING: Output base '/.../.cache/bazel/_bazel_rxie/9de92e33c21ddad2a6644b0b9fa6f76a' is on NFS. This may lead to surprising failures and undetermined behavior.
INFO: Invocation ID: c07731b0-bdf8-4fba-9d99-a375ecb422a9
INFO: Analyzed target @abseil-hello//:hello_main (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target @abseil-hello//:hello_main up-to-date:
  bazel-bin/external/abseil-hello/hello_main
INFO: Elapsed time: 0.456s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

Please kindly help:

• Could you tell me how I configure action_env like in Bazel using Buck2?
• If Buck2 does not have this option, how do I debug it using the command 'buck2 build' in Buck2?
  Thanks!

[yaoddao]

Also, I dumped out PATH when using 'bazel build --config=remote-ubuntu-22-04 @abseil-hello//:hello_main'. Bazel does not use --action_env PATH=''. PATH in Buildbarn is correct:

PS.

./bb-deployments (master)>bazel build --config=remote-ubuntu-22-04 @abseil-hello//:hello_main
WARNING: Output base '/..//.cache/bazel/_bazel_rxie/9de92e33c21ddad2a6644b0b9fa6f76a' is on NFS. This may lead to surprising failures and undetermined behavior.
INFO: Invocation ID: c8f1cd3a-31da-4c31-bd74-c60838a5a461
INFO: Analyzed target @abseil-hello//:hello_main (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /../.cache/bazel/_bazel_rxie/9de92e33c21ddad2a6644b0b9fa6f76a/external/com_google_absl/absl/base/BUILD.bazel:232:11: Compiling absl/base/internal/unscaledcycleclock.cc failed: (Exit 1): gcc failed: error executing command (from target @com_google_absl//absl/base:base) /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 33 arguments skipped)
Action details (uncached result): http://localhost:7984/hardlinking/blobs/sha256/historical_execute_response/3639c883afe1c76da8a3e352e6024f56309c3d6b4d662e1ce0b0ec73206c243a-1064/
external/com_google_absl/absl/base/internal/unscaledcycleclock.cc:15:2: error: #error dump_PATH
   15 | #error dump_PATH
      |  ^~~~~
Target @abseil-hello//:hello_main failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.595s, Critical Path: 0.07s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

[yaoddao]

When the error occurred, I saw the message Buck2 sent to Buildbarn.
Indeed, Buck2 does not send PATH(minimal environment variables. eg. /usr/bin/:/bin, and so on) to Buildbarn, only 'BUCK_SCRATCH_PATH'. So Buildbarn container can not get PATH and every shell command cannot be found in the path "", empty directroy.

PS. Buck2 command:

[stagnation]

Good!

Some more background on Buildbarn, you can indeed set variables for the runner which is often useful. But the PATH resolution is made inside the process, here, rather than through the regular process spawn. So it behaves a little bit different from what you expect. This has the benefit of very strict hermticity, and gives the build client the most flexibility and control of which programs are spawn. The system tools of the runner are not available unless explicitly opted in through the PATH variable in the Command message.

[stagnation]

I checked some more, and could not build the original Buildbarn example with the latest Buildbarn images. That is because the change to the stricter path resolution was made after the initial example: buildbarn/bb-remote-execution@1524fef This details the reasoning and that it complies better to the remote execution API.

The genrule can be made to work by adding a small PATH to the action:
But this probably warrants a design discussion for where to specify the path of a remote worker (I do not know how to express toolchains and platforms best in buck2).

diff --git a/app/buck2_action_impl/src/actions/impls/run/mod.rs b/app/buck2_action_impl/src/actions/impls/run/mod.rs
index ccb2c2b558..2bbb6e405c 100644
--- a/app/buck2_action_impl/src/actions/impls/run/mod.rs
+++ b/app/buck2_action_impl/src/actions/impls/run/mod.rs
@@ -412,6 +412,10 @@ impl RunAction {
             "BUCK_SCRATCH_PATH".to_owned(),
             cli_ctx.resolve_project_path(scratch_path)?.into_string(),
         ));
+        extra_env.push((
+            "PATH".to_owned(),
+            "/usr/bin".to_owned(),
+        ));
         inputs.push(CommandExecutionInput::ScratchPath(scratch));

original buck2 commit from last winter: 134ba21

[yaoddao]

For remote execution of Buildbarn，I think Buck2 should send a default minimum PATH environment(eg. /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin), or like buildbarn/bb-remote-execution#82. Buildbuddy's RE does not require this PATH, nor does the local executor.

[JakobDegen]

Yeah, so the behavior is technically implementation defined here. I wouldn't be opposed to adding something on the buck2 side to attempt to set some sane path, but we should only do that for implementations that actually don't set one on their own, I don't think we really want to override PATH in other cases. As far as I can tell there's no way to ask the backing implementation what it does? If so, that's a bit unfortunate

[EdSchouten]

It is actually configurable on the worker side as well: https://github.com/buildbarn/bb-remote-execution/blob/f9ea0294c9a36683d06aef1840ba39c2eaccfb68/pkg/proto/configuration/bb_worker/bb_worker.proto#L313-L320

[Overhatted]

@EdSchouten I had just discovered that myself. That is almost certainly the way to fix it since it makes more sense for the PATH to be configured on the RE side since it depends on the runner Docker image. Thanks.