Newer createrepo_c doesn't generate comps readable EL7 about createrepo_c

[kwizart]

Side note, I had to use --general-compress-type=gz (but I plan to use =xz) because I expect zstd will not work on EL7.
This is patched kojid, but I expect a sane default should be picked as koji is unlikely to have a per build target compression format...

[dralley]

Using a backported createrepo_c 1.0.2-2

But why? Is there any reason to use such a new version of createrepo_c with a distro that will be out of support in 6 months?

[kwizart]

I'm sorry, I think you miss-understood, I'm not using createrepo_c {on} el7, I'm using to generate repodata {for} el7!

[sergiomb2]

libcomps was updated recently (one month ago) , can be related ?

[kwizart]

libcomps was updated recently (one month ago) , can be related ?

no, only updating newer creatererepo_c changes this behavior with no uncompressed comps.xml

Unless we want to deal with koji adapting createrepo_c tasks for each build target, I would suggest the following as createrepo_c is concerned.

• Revert to uncompressed comps.xml for el7 compatibility unless tuned-off
• Revert to bzip2 compressed defaults (or xz that also seems to work with latest RHEL7.x).
• Once EL7 goes EOL by June 2024, revisit theses defaults. (and tune them back if using --compatibility mode).
• Then there is a need to understand why createrepo_c 1.0.0 consumes more memory than the previous versions.

[kontura]

Indeed this is not related to libcomps. These are changes in createrepo_c, some more details are here.

If we introduced the compatibility mode which restores the old defaults would that be sufficient for koji?
It could be used in all build targets.

[dralley]

My current workaround is to manually uncompress the comps file and modify the repodata.xml as appropriate.

Is that workaround not sufficient to last 6 months? Are you editing repodata.xml manually or are you able to use modifyrepo?

Revert to uncompressed comps.xml for el7 compatibility unless tuned-off

I'm fine with adding to the --compatibility flag if its absolutely required.

Revert to bzip2 compressed defaults (or xz that also seems to work with latest RHEL7.x).

On this topic you are mistaken, yum and dnf only looks for the group and group_gz tags. With the previous behavior of createrepo_c, only setting the compression type to gz would create a useful compressed comps file, because the clients were not looking for group_xz (etc). So for any other compression type, yum would download the uncompressed group file. Therefore it's equivalent in effect to disabling compression altogether for comps metadata.

See https://bugzilla.redhat.com/show_bug.cgi?id=1904360 and https://bugzilla.redhat.com/show_bug.cgi?id=2056318

DNF unlike Yum supports the group metadata being compressed (just like any other metadata file, no special behavior, no need to look for alternate tags). But yum is obviously not able to handle group being compressed.

[kwizart]

Is that workaround not sufficient to last 6 months? Are you editing repodata.xml manually or are you able to use modifyrepo?

Looks like a tremendous hack, that would be also needed by the fedora infra, so totally un-appropriate as a workaround.

The kojid usage of createrepo_c(/mergerepos_c) assumes a recent builder to generate content that can be read by all buildroot. (so RHEL7 -> RHEL N and non-EOL Fedora) If this assumption is broken, there is a need to accommodate the tool to build sane defaults for kojid usage.
In kojid context, we cannot assume createrepo_c from the buildroot to be used to generate the appropriate content for the target since we have a chickend and egg problem (createrepo_c comes from the builder userspace).

On this topic you are mistaken, yum and dnf only looks for the group and group_gz tags.

Right, but zck cannot be uncompressed by RHEL7 's yum as I expect.
Then there is the comps issue there, that you have perfectly described.

See also the upstream ticket for koji : https://pagure.io/koji/issue/3945

[kwizart]

Also worth to mention my report to the fedora infra https://pagure.io/fedora-infrastructure/issue/11637
And a(slightly) related ticket for the effects of the option on rawhide users https://pagure.io/releng/issue/11664

[dralley]

Ok, sounds like we want to overload --compatibility then.

Should --compatibility imply --general-compress-type=gz --compress-type=gz also?

[ppisar]

--compatibility is documented as "compatibility with classical createrepo". So whatever createrepo did, createrepo_c --compatibility should do too.

I also cannot see in the manual how to keep an uncompressed format. One may want to have or not to have both compressed and uncompressed formats in the output. Or one may want to have multiple compressions. Current manual reads:

    --compress-type COMPRESSION_TYPE
       Which compression type to use. Supported compressions are: bzip2, gzip, zck, zstd, xz.

   --general-compress-type COMPRESSION_TYPE
       Which compression type to use (even for primary, filelists and other xml).

Now --general-compress-type gz --general-compress-type bz2 produces bzip2 only.

Also the manual is wrong because neither bzip2 or gzip as quoted there are recognized. And zck seems rejected all the time. What's the relation to --zck option?

I personally don't like compressing by default and changing the default compression whenever a new compression type is invented. It breaks compatibility as reported in this issue.

Would you like this interface?:

    --compress-type COMPRESSION_TYPE
       Which compression type to use. Supported compressions are: none, bz2, gz, zck, zstd, xz.
       "none" means no compression. You can use this option multiple times to output in multiple compress types.
       If this option is not used, "--compress-type none" is implied".

[dralley]

Also the manual is wrong because neither bzip2 or gzip as quoted there are recognized. And zck seems rejected all the time. What's the relation to --zck option?

Yes I had also noticed this recently. I'll submit a PR to fix that.

Would you like this interface?:

Are you saying now or in the future? Because currently the combination of --compress-type and --general-compress-type is extremely confusing. #363

[ppisar]

I wrote my comment mainly to deter people from thinking on --compatibility as a moving target which changes compression formats over time as what Fedora considers compatible. It's not fair to other RPM distributions. And to warn from adding new and new options without thinking on a long-term design.

I did not know about #363. My comment would rather fit there. I will copy it there. I also find --compress-type and --general-compress-type confusing.

I'd like to keep this ticket for fixing --compatibility semantics to include gz compression.
However, I'm not against to a large overhaul per #363. Even now. The future can be now.

[dralley]

I wrote my comment mainly to deter people from thinking on --compatibility as a moving target which changes compression formats over time as what Fedora considers compatible. It's not fair to other RPM distributions. And to warn from adding new and new options without thinking on a long-term design.

My understanding is that SUSE has actually supported zstd compressed metadata since SUSE 15, which released before RHEL 8 did. I don't know about Maegia or other distros though.