Making sha256 or sha1 sum for http_archive optional about buck2

[thoughtpolice]

FWIW, I'm not a Go programmer, but by default I don't think there's a way around this. You have to download the files and calculate the proper hash manually as part of the generation step. Many other tools work this way in some form or another, not just in Buck2-land but elsewhere.

But, to work around this today, instead of using ctx.actions.download_file you instead could try using ctx.actions.run("curl -L -o ...") to download the file through a command, then untar it, then check the file hash. I think this won't work with Remote Execution (your RBE runners probably have their network cut off), but it's close to what you want.

---

I think that to support this Buck would need a feature that the Nix package manager calls "fixed output derivations" or "FODs". These are build rules where you don't give the hash of the downloaded file itself, but instead give a hash of the expected output files that some arbitrary commands produce.

For example, if you download https://example.com/foo-0.1.tar.gz, you can give the hash of this file directly. But this hash is fragile, because the server admin may recompress the file with different compression settings. But the content of the tarball would remain the same in any case. Therefore, you want to download a file, then run a command to extract it, then check the hash of the output files. This feature also means you can e.g. completely change the download URL to something arbitrary and it will still work and get cache hits/early cut-off, assuming that the contents are in fact the same.

I don't know what FODs would look like in Buck2 at all.

[dvtkrlbs]

Another way could be not using a GOPROXY and using the default VCS resolving algorithm go uses. This is much more complicated than using the user supplied or default GOPROXY but would probably circumvent unnecessary download.

[JakobDegen]

@thoughtpolice you could imagine doing this in a dynamic_outputs like way. Imagine essentially the exact same API that dynamic_outputs has today, except that on the call to dynamic_outputs you also have to specify the hash all of the output artifact that you're later going to bind within the dynamic_outputs. In exchange for that, the analysis_context in the dynamic function could give you access to a version of ctx.actions.download_file that doesn't require specifying the hash. That's really only part of what I understand FODs to provide, you'd probably also want such a construct to change caching behavior in other ways that is less obvious to really see the benefits.

I agree with Austin though, your best option right now is to have a locally run action that invokes curl.