RPM inconsistency issue in KubeVirt caused by bazeldnf about kubevirt

[fanzhangio]

@rmohr Can you please take a look ?

[rmohr]

Suspect cause:
This is suspect to be a defect of bazeldnf. The bazeldnf selects package from upstream centos8/9 or local docker container (in our case). For some reason, bazeldnf messed up.
Guess this might be related to https://github.com/rmohr/bazeldnf/blob/67cb93b4fd7dd217cafac3625dd09313dde9dad2/pkg/bazel/bazel.go#L125C1-L135C36, which seems trying to fix it but does not work for this case.
We are using KubeVirt release-1.0.1

I don't think it is related to this. This path is only executed when resolving dependencies from a remote repository to write the expected checksums to the WORKSPACE file (would be called during jack/rpm-deps.sh to update dependencies in the WORKSPACE file but it does not actually download any RPMs, nor does it itself calculate a checksum).

rpm( // this is line 2360
name = "libmpc-0__1.1.0-9.1.el8.x86_64",
sha256 = "4fbf751702394211896af9f253ad2f38c80f603e007a7c9ed2b9c8e5d97f2f95",
urls = ["http://rpms:80/AppStream/Packages/libmpc-1.1.0-9.1.el8.x86_64.rpm"],
)

So here it seems to have the correct digest. The rpm rule is just a thin wrapper around what http_archive does. If there would be a bug passing down the sha256 to the bazel downloader, I think we would know about this already, but maybe that would be something ...

Error in download: java.io.IOException: Error downloading [http://rpms:80/AppStream/Packages/libmpc-1.1.0-9.1.el8.x86_64.rpm] to /root/.cache/bazel/_bazel_root/6f347497f91c9a385dcd9294645b76e0/external/libmpc-0__1.1.0-9.1.el8.x86_64/rpm/downloaded: Checksum was 4fbf751702394211896af9f253ad2f38c80f603e007a7c9ed2b9c8e5d97f2f95 but wanted 93c2232d1885ec6265159f4669aeb13335a80e74d3ae0832f624678d87ea3638

This seems to look like we would have in the WORKSPACE file the checksum 93c2232d1885ec6265159f4669aeb13335a80e74d3ae0832f624678d87ea3638 specified, but according to your code snippet above you have 4fbf751702394211896af9f253ad2f38c80f603e007a7c9ed2b9c8e5d97f2f95 there. This looks to me like a very suspicius thing when you say that the checksum of the file actually is 4fbf751702394211896af9f253ad2f38c80f603e007a7c9ed2b9c8e5d97f2f95 in the container.

It looks to me like some important bit of info is missing or not entirely correct. Because what I would expect to see in your case would be a message the other way round, that it wanted 4fbf751702394211896af9f253ad2f38c80f603e007a7c9ed2b9c8e5d97f2f95 but got 93c2232d1885ec6265159f4669aeb13335a80e74d3ae0832f624678d87ea3638.

Just to highlight it again:

Bazeldnf should calculate the checksum based on the downloaded RPM in the builder, in this case it is /root/.cache/bazel/_bazel_root/6f347497f91c9a385dcd9294645b76e0/external/libmpc-0__1.1.0-9.1.el8.x86_64/rpm/downloaded. The checksum should be consistent.

Bazeldnf does not do that. It just writes a checksum it gets from the repo information into the WORKSPACE file. If you have there 4fbf751702394211896af9f253ad2f38c80f603e007a7c9ed2b9c8e5d97f2f95, then we just call BAZELs download things and are no longer involved during build (https://github.com/rmohr/bazeldnf/blob/main/internal/rpm.bzl#L25).

[fanzhangio]

Thanks @rmohr for your response. This is a very weird error we don't understand. It is reproducing on local dev env and CI. We expect the WORKSPACE is the only place where customized RPM is updated, e,g, checksum and rul, as what we already do in above example. The incorrect checksum this error was suggesting (93c2232d1885ec6265159f4669aeb13335a80e74d3ae0832f624678d87ea3638) seems is from an upstream rpm stream. We've explicitly cleaned the cache by Makefile and hack/dockerized, wondering if there are something still missing.

Can you please point me any suggestions or workaround that can fix this blocker ?

[rmohr]

Can you please point me any suggestions or workaround that can fix this blocker ?

Can you grep the WORKSPACE file and all cache directories for the 93c2232d1885ec6265159f4669aeb13335a80e74d3ae0832f624678d87ea3638 checksum and maybe backtrace where this value is even coming from?

[fanzhangio]

I see the files rsynced into builder container is not identical to the kubevirt on the host..

The synced WORKSPACE inside the container is 93c2232d1885ec6265159f4669aeb13335a80e74d3ae0832f624678d87ea3638:

[root@f80633154e04 kubevirt]# grep -B3 libmpc-1.1.0-9.1.el8.x86_64.rpm /root/go/src/kubevirt.io/kubevirt/WORKSPACE 
rpm(
    name = "libmpc-0__1.1.0-9.1.el8.x86_64",
    sha256 = "93c2232d1885ec6265159f4669aeb13335a80e74d3ae0832f624678d87ea3638",
    urls = ["http://rpms:80/AppStream/Packages/libmpc-1.1.0-9.1.el8.x86_64.rpm"],

This is not the same of what is defined in KubeVirt project on host, which is :

➜  kubevirt git:(release-1.0.1) ✗ grep -B3 libmpc-1.1.0-9.1.el8.x86_64.rpm WORKSPACE 
rpm(
    name = "libmpc-0__1.1.0-9.1.el8.x86_64",
    sha256 = "4fbf751702394211896af9f253ad2f38c80f603e007a7c9ed2b9c8e5d97f2f95",
    urls = ["http://rpms:80/AppStream/Packages/libmpc-1.1.0-9.1.el8.x86_64.rpm"],

I just directly downloaded rpm from RPM container. The checksum is 4fbf751702394211896af9f253ad2f38c80f603e007a7c9ed2b9c8e5d97f2f95

[root@f80633154e04 tmp]# wget http://rpms:80/AppStream/Packages/libmpc-1.1.0-9.1.el8.x86_64.rpm
[root@f80633154e04 tmp]# sha256sum libmpc-1.1.0-9.1.el8.x86_64.rpm
4fbf751702394211896af9f253ad2f38c80f603e007a7c9ed2b9c8e5d97f2f95

digging..

update:

I figured out the cause. Thanks !