Cache misses compared to other cache implementations about buildbuddy

[siggisim]

Hey @joeljeske - thanks for reporting this, and sorry for your experience!

Is it possible that you were hitting the size limit of your configured cache, and artifacts were getting evicted? Our default cache size is 10GB, while bazel-remote's is 100GB (in the example config).

Would love to help get to the bottom of this - is there a an easy repro case you can point to?

In the meantime, you can use BuildBuddy UI with any grpc cache - including bazel-remote.

[joeljeske]

I had bumped to 100gb in my setup. Also, it was exactly the same number of cache hits (on the same actions) between runs, which doesn't seem like a cache eviction issue. Cache eviction would likely be more random seeming.

I don't have an easy public repo available :(

And yes I did see the use of the UI with any cache and that is what I'm experimenting with. Thanks!

[siggisim]

Hey @joeljeske - I've been trying to reproduce this but with the 10+ repos I've tested - the number of cache hits with BuildBuddy and bazel-remote have been identical.

Do you have any info on the type of actions that consistently aren't getting cached? Is it protobuf? Do you have an example from execution log where you're seeing a remote_cache_hit: false?

Something like this would be incredibly helpful, as I could try to reproduce the behavior with a similar action:

---------------------------------------------------------
command_args: "/bin/bash"
command_args: "-c"
command_args: "source external/bazel_tools/tools/genrule/genrule-setup.sh; cp third_party/zlib/BUILD.tools bazel-out/darwin-fastbuild/bin/third_party/zlib/BUILD.bazel"
environment_variables {
  name: "PATH"
  value: "/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"
}
platform {
}
inputs {
  path: "external/bazel_tools/tools/genrule/genrule-setup.sh"
  digest {
    hash: "bed43d0bb00449e82a04fbb6535f490fa5d7ed140b1c3ad2d463d5c295e9e836"
    size_bytes: 858
    hash_function_name: "SHA-256"
  }
}
inputs {
  path: "third_party/zlib/BUILD.tools"
  digest {
    hash: "bc1d8834f25ecd6711c6404a3a81fb5025995bbcc5829407ef481fc86c629d51"
    size_bytes: 320
    hash_function_name: "SHA-256"
  }
}
listed_outputs: "bazel-out/darwin-fastbuild/bin/third_party/zlib/BUILD.bazel"
remotable: true
cacheable: true
progress_message: "Executing genrule //third_party/zlib:embedded_build_file"
mnemonic: "Genrule"
actual_outputs {
  path: "bazel-out/darwin-fastbuild/bin/third_party/zlib/BUILD.bazel"
  digest {
    hash: "bc1d8834f25ecd6711c6404a3a81fb5025995bbcc5829407ef481fc86c629d51"
    size_bytes: 320
    hash_function_name: "SHA-256"
  }
}
runner: "remote cache hit"
remote_cache_hit: true
---------------------------------------------------------

[joeljeske]

Here is a gist of an example cache miss. https://gist.github.com/joeljeske/52b6a88d25f3658cbcf5277850e41adf

I am able to:

• build a target and mark as upload to cache
• clean
• build same target and still have sandboxed local processes

Moreover, I can perform those steps multiple times and get the exact same execution logs. Then I can swap the remote cache url to bazel-remote (grpc or http), and get cache hits when repeating those steps above. (I have disabled --disk cache for these trials)

[siggisim]

Hey @joeljeske - thanks for helping us get to the bottom of this!

We were able to create a rule that pretty closely mapped to the execution log you shared:

rollup_bundle(
    name = "react__bundle",
    config_file = "rollup.config.js",
    entry_point = "@npm//:node_modules/react/index.js",
    format = "system",
    output_dir = True,
    deps = [
        "@npm//react",
        "@npm//rollup-plugin-commonjs",
        "@npm//rollup-plugin-node-resolve",
        "@npm//rollup-plugin-replace",
    ],
)

This uncovered an issue with the way Action Cache results were being validated that affected actions with directory outputs.

We've pushed a fix for this in release v1.2.1.

Really appreciate your help in debugging this issue!