Bazel is crashing while uploading BES events to BuildBuddy about buildbuddy

[siggisim]

Hey @amit-mittal - thanks for reporting!

This type of error message most commonly occurs when your BuildBuddy instance restarts for whatever reason. Are you able to see if your BuildBuddy app has any restarts? Could it be running out of memory?

[amit-mittal]

Hey @siggisim - thanks for looking into it!

No, I don't see any crashes or restarts on the BuildBuddy server side. The node on which BuildBuddy is running has 60 GB memory, while only 16 GB is being used.

So far, we are able to repro this issue only while running tests of one of our Go services (if that is relevant).

[siggisim]

Interesting, is there any chance this is a CI run that runs multiple Bazel invocations back-to-back?

By default, Bazel will try to re-use the build event stream connection across invocations - which could be leading to the issues we're seeing here. You can disable this behavior with the bazel flag --keep_backend_build_event_connections_alive=false. I suspect that might solve this issue.

[amit-mittal]

I added --keep_backend_build_event_connections_alive=false in our .bazelrc, but I am seeing below errors while doing bazel build .... one after another, that never used to happen for us. The below errors go away if I remove this new setting, so I don't think the overhead of creating the new connection as part of every run would work for us.

WARNING: The background upload of the Build Event Protocol for the previous invocation failed with the following exception: 'com.google.devtools.build.lib.util.AbruptExitException: The Build Event Protocol upload failed: All retry attempts failed. UNAVAILABLE: UNAVAILABLE: Channel shutdown invoked UNAVAILABLE: UNAVAILABLE: Channel shutdown invoked'. Ignoring the failure and starting a new invocation..

WARNING: The background upload of the Build Event Protocol for the previous invocation failed to complete in 5.003 seconds. Cancelling and starting a new invocation...

I don't think it should matter, but the run is happening on the developer machine (MacOS).

Regarding the multiple invocations, we are NOT running bazel commands in parallel, but as part of the usual developer workflow, we do run bazel commands one after another. That is one of the reasons, that we have --bes_upload_mode=nowait_for_upload_complete set, so the developers are not blocked while the events are being uploaded.

[siggisim]

There are (unfortunately) lots of Bazel bugs with the --bes_upload_mode= flag:

• bazelbuild/bazel#11392
• bazelbuild/bazel#11408

There is some work being done to improve the BES artifact uploader:

• bazelbuild/bazel#11473
• bazelbuild/bazel#13655

Do you see the same error without that --bes_upload_mode= flag?

Do you see the same error if you add the flags --remote_timeout=3600 and --bes_timeout=3600s (wondering if a timeout is being hit and not handled gracefully)?

[amit-mittal]

That's true! 😞

I don't think we would be able to change the --bes_upload_mode to a blocking call, but we can try it out. As we upload the events in async mode, we'll also try increasing the timeout and share the findings.

We will also prioritize upgrading bazel to v4.2.1, to pick up the fixes in the BES uploader, if there were any. Thanks for helping to investigate the issue!

[BalestraPatrick]

We also see this crash only on our Linux CI (macOS CI is fine). We're on Bazel 4.2.0 and we don't even set --bes_upload_mode, but we still see the exact same crash. Setting --keep_backend_build_event_connections_alive=false did not make a difference for us.

[siggisim]

This is likely fixed by bazelbuild/bazel#13959 which hasn't made it into any Bazel releases yet.

Are either of you able to share your grpc log for one of these invocations captured with Bazel's --experimental_remote_grpc_log=? You can send it to [email protected]

[siggisim]

Going to close this issue now that bazelbuild/bazel@e855a26 seems to have made it into Bazel 5.0 release candidates bazelbuild/bazel#14013

Please re-open this issue if you're able to reproduce with this with Bazel 5.0 (in which case either a grpc log or a BuildBuddy Cloud invocation would be super helpful).