Universalize (as much as possible) the binary repositories for RPM-base distros about virtualmin-install

[iliajie]

Yes, of course! Testing it right now. I don't know about all systems but I only care about CentOS 7-9 and Fedora. I think this is all we have to support?

[swelljoe]

So, to make this happen, @iliajie we need to confirm that all the packages in this universal binary repo work on every distro you want to support, and that the groups (LAMP and LEMP stacks) are correct for every distro you want to support.

[swelljoe]

If we have missing packages currently listed in the groups on some distros, we need to decide if we want to remove that dependency or provide that dependency in our repos (if the OS provides a newer one, it should be fine...since we're histrorically mostly pulling from Fedora when we need a package not in RHEL or EPEL, though I don't know if that's any longer a concern).

ClamAV is going to be a pain in the ass, just because it is always a pain in the ass. I don't know how or why yet, but I know that it will be, somehow.

[iliajie]

we need to confirm that all the packages in this universal binary repo work on every distro you want to support

It doesn't for jailkit - fails with dependency error pyhton_paltform, if I remember correctly off-hand.

We need to handle not setting up EPEL during install

That is already done right, tested and working (at least with latest PR)

This will break everybody who's already done a Virtualmin 7 install. But, it's a beta, so we can ask them to update and tell them how. Or I can make links from the old repos to the new.

Having symlinks would be fine for some time.

but the PR to do that also replaces release packages with a bunch of shell scripting, which is bad).

Already reverted back on the PR to using release package.

Will it be OK across all supported distros to use the same one?

Fedora has different set of package. CentOS Stream too. Very slight but different. I think we can just add few extra more package to default stack for all RHELs, like wanted cronie by Fedora.

If we have missing packages currently listed in the groups on some distros, we need to decide if we want to remove that dependency or provide that dependency in our repos (if the OS provides a newer one, it should be fine...since we're histrorically mostly pulling from Fedora when we need a package not in RHEL or EPEL, though I don't know if that's any longer a concern).

I will review and add a single file to yum-groups that must cover everything for all distros. (like today)

ClamAV is going to be a pain in the ass, just because it is always a pain in the ass. I don't know how or why yet, but I know that it will be, somehow.

Also will check, but presumably won't be a big problem.

---

Tell me how do you want to call the repo (URL) so I can add it to the PR to download correct release .rpm file.

[swelljoe]

One other concern Ilia raised in conversation is that Virtualmin's internal upgrade to Pro feature modifies the repo files directly rather than installing a new release package. And, even if we made it work the right way, it'd still end up making an .rpmnew file instead of replacing with a Pro version of the repo file(s). This is also an issue on running ./install.sh --setup unless the repo package is removed first.

Ideally the Virtualmin internal upgrade would remove the GPL release package and install the Pro one, rather than modifying the file directly.

Another improvement would be if auth details could be stored outside of the repo file, but I don't believe that's possible for HTTP basic auth. I'm looking into it. Maybe somebody else has a clue...

[swelljoe]

Repo path can just be rpm, perhaps?

[swelljoe]

Oh, but we need architecture in there, too, since we want to support ARM at some point soonish. Umm... I dunno.

Maybe: https://software.virtualmin.com/vm/7/rpm/$arch?

[swelljoe]

Oops, I forgot the Virtualmin version, which is needed to determine which signing keys are used.

Also, I'll make this work without auth, so it won't need a separate repo for Pro and GPL.

[iliajie]

Ideally the Virtualmin internal upgrade would remove the GPL release package and install the Pro one, rather than modifying the file directly.

This is for later please.

Maybe: https://software.virtualmin.com/vm/7/rpm/$arch?

Oh, great. I wanted to suggest VM version number. Looks fine!

[iliajie]

Also, I'll make this work without auth, so it won't need a separate repo for Pro and GPL.

Sure, great!

[iliajie]

So the full link would be:

https://software.virtualmin.com/vm/7/rpm/virtualmin-release-latest.noarch.rpm

Correct?

I'm working on merging yum-groups now into a single file. Looks very promising for all CentOS 7-9 + Fedora - packages are there.

[swelljoe]

Fedora has different set of package. CentOS Stream too. Very slight but different. I think we can just add few extra more package to default stack for all RHELs, like wanted cronie by Fedora.

Which packages? Why are we wanting cronie?

[iliajie]

Yeah, cronie but it's available on all distros CentOS 7-9 - so, all cool!

[swelljoe]

Why are we wanting cronie?

[iliajie]

Cron module?

[swelljoe]

Oh. Is that not in the core OS?

[swelljoe]

I just assumed cron was something we could always rely on, on any Linux system.

[swelljoe]

And, for the package name/path, I don't know exactly, yet. I'm considering splitting it into two, one for universal packages (which needs auth for Pro repos, but doesn't need anything for GPL) and one for binaries (which never need auth for either Pro or GPL).

Not sure what to name them. They're both gonna be "universal" in some sense.

[swelljoe]

Maybe virtualmin-universal-release and virtualmin-arch-release? Does that seem sensible, to you?

And, they could be stored in their own path at, like /vm/7/rpm/release so they don't have to have $basearch in the path when downloading them and don't need to be maintained separately.

Would we ever need to update these packages? I guess if there was a key invalidation and we needed to roll new keys...maybe I should make this a repo, too, so users can update their release packages.

[iliajie]

We can never rely on anything. Some distros don't have tar and rsync. what kind of linux system is this without those.

I will handle this.

So, question -

<packagereq type="default">perl-lib</packagereq>

Will try to install but won't fail, correct?

---

Added to PR using a new releasever free .rpm.

[swelljoe]

lol, I went to create a repo for this, and found I already have a virtualmin-release repo. I'd forgotten about it. It's not up to date. But, I'm refactoring everything anyway... https://github.com/virtualmin/virtualmin-release

[swelljoe]

OK, I think I actually want to do:

virtualmin-universal-release - for the pure-Perl Virtualmin modules and Webmin/Usermin, etc.

virtualmin-release - for the binaries

[iliajie]

so, to files?

[swelljoe]

Oh, wait! I have a better idea.

virtualmin-universal-gpl-release and virtualmin-universal-pro-release

This solves the problem of Virtualmin updating the repo files directly! and Very clearly states in the package name which repos are used!

[iliajie]

Okay. Let me fix the PR then.

[swelljoe]

Wait, stop making PRs until this is settled!

We need a minor change to the Virtualmin upgrade process.

@jcameron can you make it so the Virtualmin Upgrade to Pro feature handles upgrades via packages rather than via updating the yum repo files directly, if the virtualmin-gpl-release package is installed? You can then just install virtualmin-pro-release which will replace the repo file with a new one. Still need to update the /etc/virtualmin-license file to include the serial and key.

[iliajie]

Well, this won't solve change license process?

[iliajie]

Wait, stop making PRs until this is settled!

Those are commits to the PR. :)

[swelljoe]

Well, this won't solve change license process?

But, it will, though? Two packages that conflict with each other very gracefully solves the problem of rpmnew files being generated, and handles the upgrade process without anyone every touching a repo file with a script. (The small script to set the auth details is bundled in the package, as it's always been, but Virtualmin no longer needs to care.)

[iliajie]

I meant virtualmin change-license command.

[swelljoe]

Ah! OK, in the change-license case, I think it's fine to hit the repo file directly.

That's never an upgrade/downgrade to/from pro/gpl, right?

[swelljoe]

But, it'll need an update to recognize the new paths. Unless I make the repo file name stay the same for the Pro/GPL sensitive version. e.g. virtualmin.repo contains the universal Perl packages, which includes auth and has Pro or GPL version of package, and then...I dunno what I call the other one.

[iliajie]

No, change-license cannot do upgrades.

How is the package for Pro users is going to have the correct key/serial on the package? Upon download time or it will be fixed afterwards on the system?

[swelljoe]

Oh, would also be possible to remove the release package and install it again...

[swelljoe]

Will be fixed at package install time (this is suboptimal, but the alternatives are worse). But, we no longer have to worry about rpmnew when switching from pro to gpl or vice versa if the package names are different.

[iliajie]

But, it'll need an update to recognize the new paths. Unless I make the repo file name stay the same for the Pro/GPL sensitive version. e.g. virtualmin.repo contains the universal Perl packages, which includes auth and has Pro or GPL version of package, and then...I dunno what I call the other one.

Maybe as both repos are needed for all installs we could place them into a single package?

[swelljoe]

Maybe as both repos are needed for all installs we could place them into a single package?

OK, I guess we can do that. But, I still think I want to split out the repos into their own files, so one of them never gets updated. It'll be exactly what's in the release (no license info). Maybe not necessary though if the binary one includes the $basearch variable so it can be the same two release packages.

Yeah, I think that works. So, we have either virtualmin-pro-release or virtualmin-gpl-release which both produce a file called /etc/yum.repos.d/virtualmin.repo containing both the perl repo and the binary repo.

[swelljoe]

Need to test what happens when installing GPL release over Pro release, to be sure RPM will allow it to replace a modified file. I think it'll make it rpmsave, which is fine.

[swelljoe]

OK, if no complaints about that, I'm going to branch and make two new release packages for this, and start making the new repos.

[swelljoe]

@jcameron, if you're following along, I updated the package name above (for the part that applies to you and the Virtualmin upgrade to Pro feature).

[iliajie]

Yeah, I think that works. So, we have either virtualmin-pro-release or virtualmin-gpl-release which both produce a file called /etc/yum.repos.d/virtualmin.repo containing both the perl repo and the binary repo.

That's fine!

Need to test what happens when installing GPL release over Pro release, to be sure RPM will allow it to replace a modified file. I think it'll make it rpmsave, which is fine.

I think it won't replace it, particularly if that virtualmin.repo was modified. May be just don't make it a %conf file and just forcefully remove each install/re-install?

OK, if no complaints about that, I'm going to branch and make two new release packages for this, and start making the new repos.

Alright, I'm building a new virtualmin-comps.xml to work across all systems.

[swelljoe]

It's a different package. It's not upgrade/downgrade.

[swelljoe]

Are you sure about clamav being the same package name(s) across all supported distros/versions?

[iliajie]

Also weeded out clamav being installed on minimal..

[swelljoe]

Why are we depending on chrony? https://github.com/virtualmin/virtualmin-yum-groups/blob/401dbf6a8a4d81eae24f4337b213b421e1cca43b/virtualmin-centos-8-comps.xml#L47

We shouldn't be making decisions about how people set time?

[iliajie]

This is not a dependency. It will try to install. It's an awesome and new way to sync time.

Like:

systemctl restart chronyd

[iliajie]

We shouldn't be making decisions about how people set time?

As far as I remember right now, there is no other way for CentOS 8+ to do it.

[iliajie]

.. and I made Webmin support it too.

[swelljoe]

systemd-timesyncd is everywhere now. But, I guess chrony is the default on RH/Fedora. Seems like if somebody is already using timesyncd, this would replace it. But, I don't guess anyone has complained in the two years since you made that change.

[iliajie]

Yeah, chronyd is fine for RHELs. Although, I like systemd-timesyncd but widely it only works with Debian systems.

[iliajie]

New version of install.sh script will use both to force sync user time, to avoid issues with installing proper ca-certificates package.

[iliajie]

All done with comps - virtualmin/virtualmin-yum-groups@41913c6

[iliajie]

Must work on Alma, Rocky, CentOS 7-9, Fedora and RHEL. Even probably Oracle 8 too.

[swelljoe]

OK, I guess we'll find out in a couple of hours when I'm finished pushing the new repos.

[iliajie]

Alright. So we stick with:

virtualmin-universal-gpl-release and virtualmin-universal-pro-release, correct?

[swelljoe]

No. virtualmin-gpl-release and virtualmin-pro-release

See above.

[swelljoe]

Since we're only doing one repo file, they can have one release package.

[iliajie]

Yeah, yeah. I actually meant that but copy paste from strings.

[iliajie]

Okay, all done - ca59e86

[iliajie]

Forgot login and host. Fixed - 6c981c1

[swelljoe]

Here's the PR for the release packages. I'll build/test before merging.

virtualmin/virtualmin-release#1

[iliajie]

Alright. Please test using install.sh from my PR.

[swelljoe]

Done.