Virtualmin 7 checklist about virtualmin-install

[jcameron]

Dropping all except PHP-FPM - we might want to at least give users the ability to use CGI mode for systems with many low-traffic websites, to avoid the overhead of running a bunch of FPM servers.

Dropping AWstats and Webalizer - would be easy, but I wonder how many people still use these?

Dropping mod_php - this would allow deleting a bunch of code, as there's a lot of special case handling needed to support running PHP scripts as a different user.

[swelljoe]

Development happens in the open. I will announce when anything is expected to work. I didn't name it an RC, which I understand can be misleading. I didn't think to change it. But, even if the script works (which it probably would, if the packages/repos all existed), the majority of the work is outside of the script. It literally cannot work, and should not be expected to work, until the repositories are complete and working.

I'm sorry it's taking so long. I have a full-time job that isn't Virtualmin (because I literally lived in poverty while working on Virtualmin full-time). I'm doing the best I can.

[jcameron]

Yeah, Virtualmin should hide the mod_php mode if the needed Apache module is missing, and the CGI mode if suexec doesn't work.

[swelljoe]

https://forum.virtualmin.com/t/virtualmin-7-installer-beta/115421

[swelljoe]

Closing this, as we mostly made it.

[swelljoe]

Yes, DAV is not installed or configured anymore. I'm gonna deprecate the module and remove it from the repos, as well, so no one installs it thinking it'll do the useful thing they assume it does based on the name.

[iliajie]

I dropped MySQL for Ubuntu 18.04 and Ubuntu 20.04 replacing it with MariaDB on stack packages.

Joe, I haven't updated the version number intentionally, as when the package will be updated on existing installs it will try to replace MySQL with MariaDB and this isn't something what we want. Is there a way to build a new Ubuntu packages without touching already installed ones? Or we just don't update the versions and rebuild the packages, so they would be picked up with new installs as we need, while current installs will stay intact?

[iliajie]

Drop old distros

Done.

Beta version 7.0.0 of the install script won't advertise Ubuntu 16.04, 18.04 and Debian 9. However, it will still be possible to install Ubuntu 18.04, while trying to install Ubuntu 16.04 or Debian 9 will return an error.

.. while current installs will stay intact?

Yes, current installs will stay intact (as we discussed on PMs -- Update for future readers).

[marclaporte]

One step closer: virtualmin/virtualmin-gpl@0ec9ade

[chris001]

Just tried to run the 7.0-RC4 install script from this repo on Ubuntu 22.04 LTS, and it failed with this error:

Downloading repository metadata                                         [ERROR] Failed with error: 100
[  ✘  ]

[ERROR] Something went wrong. Exiting.
[ERROR] The last few log entries were:
Hit:4 http://us.archive.ubuntu.com/ubuntu jammy-security InRelease
Ign:5 https://software.virtualmin.com/vm/7/gpl/apt virtualmin-jammy InRelease
Err:6 https://software.virtualmin.com/vm/7/gpl/apt virtualmin-jammy Release
  404  Not Found [IP: 163.172.162.254 443]
Get:7 https://software.virtualmin.com/vm/7/gpl/apt virtualmin-universal InRelease [10.7 kB]
Err:7 https://software.virtualmin.com/vm/7/gpl/apt virtualmin-universal InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D9F9010760D62A6B
Reading package lists...
E: The repository 'https://software.virtualmin.com/vm/7/gpl/apt virtualmin-jammy Release' does not have a Release file.
W: GPG error: https://software.virtualmin.com/vm/7/gpl/apt virtualmin-universal InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D9F9010760D62A6B
E: The repository 'https://software.virtualmin.com/vm/7/gpl/apt virtualmin-universal InRelease' is not signed.
Made it here...why?
Downloading repository metadata: [2022-04-29 22:55:37 UTC] [ERROR] Failed with error: 100
[2022-04-29 22:55:37 UTC] [ERROR] Something went wrong. Exiting.
[2022-04-29 22:55:37 UTC] [ERROR] The last few log entries were:

Isn't RC4 supposed to run without error on Ubuntu 22.04?

Welcome to the Virtualmin GPL installer, version 7.0.0-RC4

  This script must be run on a freshly installed supported OS. It does not
  perform updates or upgrades (use your system package manager) or license
  changes (use the "virtualmin change-license" command).

  The systems currently supported by install script are:

    Red Hat Enterprise Linux derivatives
      - Alma Linux and Rocky 8 on x86_64
      - CentOS 7 and 8 on x86_64
      - RHEL Linux 7 and 8 on x86_64

    Debian Linux derivatives
      - Ubuntu 20.04 LTS and 22.04 LTS on i386 and amd64
      - Debian 10 and 11 on i386 and amd64

[chris001]

Sorry! I'd expected the 7.0.0-RC4 script to work on debian/ubuntu because it says it works on 22.04 LTS in the Release note on github, and also on the welcome banner screen! Let us know if you need anything to get the 7.0.0-RC4 script install to work on Ubuntu 22.04 LTS! Or a deb package that has LEMP/nginx! Or a list of commands to disable and remove Apache2 from vm7 and add the LEMP/nginx plugin to a running LAMP/apache2 vm7 installed from the vm7 deb!

[chris001]

Thanks @swelljoe 🙏 🙇 💪 🥇 please tell us if you need anything from us to get the script working on debian based distros as I'm sure many here would be happy 😀 to test 🧪 and provide you with feedback telemetry in the form of comments with the install logs 🌴 , even code up an automatic install script tester script with github's Travis-CI and every OS in the Travis VM, that'd sure make life easier!

[swelljoe]

Yeah, I feel most strongly about mod_php going away forever, as it is very solidly the consensus opinion in the PHP community that it is a dangerous anti-pattern, to be avoided at nearly any cost. I don't think PHP7 even ships with a mod_php version anymore. But, we'll still have PHP 5 on at least one or two of our supported distros in Virtualmin 7 (unless I also drop CentOS 7 and Ubuntu 18.04). But, maybe we just block installation of it, I dunno.

I'll run a poll to see if anybody is using AWstats and/or Webalizer. Won't really answer the question because the forum is like 1% of our users, but might give us some insight.

I'm not sure about CGI. We don't have suexec anymore on CentOS 8 (unless I start shipping our own Apache package again), so it's already gone by necessity there. Ubuntu and Debian can still have it, but I'm uncertain of the benefit. PHP-FPM will be a lot of shared libs, I think, so it's probably not that intensive, and CGI means you're spinning up more Apache processes instead, when it gets loaded, so it any environment with a lot going on, CGI will be more resource intensive I'm pretty sure. And, nginx also does not have CGI or suexec, so we're dealing with an edge case on only two configurations out of...like ten (i.e. if we grid out which combination of distro+web server, only Ubuntu+Debian running Apache have CGI, regardless of what we do, every other combination CGI is impossible already). I lean toward saying let's just hide it.

Historically, I have liked to be able to say "Switch execution modes and see if it behaves differently" as a troubleshooting tactic but that's just because we had flakiness from trying to support so many combinations across so many different distros and it pretty frequently went wrong in some combinations. My hope is that if we simplify, we will stop having so much confusion, so many users breaking their systems trying to get things to work, etc. And, PHP-FPM is what the PHP devs recommend.

But, if you'd rather take it slow, maybe we can just hide CGI and fCGI by default, maybe behind an Advanced clicker or something. I can keep including and configuring suexec-custom on Debian/Ubuntu for that purpose, so it can be turned on from the GUI without needing any package installation or anything. But, we already don't have these options on CentOS 8, so...normalizing on the optimal/best practices path seems better than having differences.

[swelljoe]

Then again, if those options automatically hide themselves when suexec and mod_fcgid is not installed, maybe I should not install them by default, and just document how to turn them on (by installing those packages).

[swelljoe]

Added removing DAV Login module from the repos to the checklist.

[iliajie]

Added removing DAV Login module from the repos to the checklist.

I think I already removed it?

[swelljoe]

I realized while polling users that maybe we still don't support multiple PHP versions with PHP-FPM? I guess we have to keep supporting fcgid+suexec until we have support for multiple versions with FPM (it's not a limitation of FPM, it's just Virtualmin that doesn't support more than one, if we need the installer to do something to prep for multi-version FPM, I can do that, though I think it's just enabling the service for the desired version(s) and setting up the virtual host to proxy to the right one).

[marclaporte]

Joe wrote: "Simplify, simplify, simplify. What can we get rid of in the GUI?"

The features exist because someone thought they should be there. When there is an attempt to remove something, there is a very good chance someone will say "please don't remove that, it's super important to me". A few things can be deprecated by the evolution of technology, but even if 10-15% of the features were no longer needed, Virtualmin/Webmin would still be a huge number of options/settings. I've been working on Tiki Wiki CMS Groupware since 2002 and it's the same challenge. We have over 2000 preferences (settings) in the admin panels! Openfire lives this as well, albeit less acutely. For all these projects, it's the richness of 2 decades of collaboration covering all kinds of use cases and edge cases!

I will quote Clay Shirky: "It's Not Information Overload. It's Filter Failure"
https://www.youtube.com/watch?v=LabqeJEOQyI

Here are some things that can help:

Filters

How to hide the preferences that are rarely changed? So beginners start with perhaps 15% of the options. This helps. And as they become more experienced, they can turn on the "advanced" mode. In basic mode, if an advanced preference is set to a non-default value, it is still shown. A challenge is that some preferences depend on others, so we have code to deal with that (if a sub-preference is at a non-default value, its parent is shown even if it is at the default value)

Context

Try to put all related settings near each other. Even if that means adding something twice (if it relates to more than one topic). This is great: virtualmin/virtualmin-gpl@5dc0c56

WYSIWYCA

A few places, I noticed some elements that are in the menu, but then, not active. Other times, it offers us to install (and that is fine).

Example:

In Tiki, we have dubbed this WYSIWYCA (What You See is What You Can Access)
https://doc.tiki.org/WYSIWYCA
This helps especially with non-admins that have fewer permissions.

Search

Thankfully, there is already a search engine. IMO, the Webmin search is well positioned. But I have trouble getting used to the Virtualmin search in the middle of the page. I would envisage one combined search engine, above the Webmin/Virtualmin switcher

Thank you!

[whereisaaron]

@jcameron wrote:

Dropping all except PHP-FPM - we might want to at least give users the ability to use CGI mode for systems with many low-traffic websites, to avoid the overhead of running a bunch of FPM servers.

You can just run one FPM process if you like. Or just pool of 1-4 same as Apache default child process. I think you can configure an almost as small FPM set-up for small installs.

@swelljoe wrote:

I realized while polling users that maybe we still don't support multiple PHP versions with PHP-FPM?

I have multiple PHP versions (ondrej/php PPA) working with FPM on Ubuntu 18.04 with Virtualmin. Seems to work ok. Just had to install all the right modules. Different pool files in /etc/php/<version>/fpm/pool.d/*.conf for different virtualmin sites. Seems to work, but not sure how well doing things like change PHP versions works. All changes are a bit dicey with FPM on 6.x so I avoid fiddling 😄

[MattWilcox]

I am down with that entire list! Default FPM? Testing it always works? Latest stuff? Yes, please!

Maybe people relying on legacy features should just continue a "legacy" Virtualmin? There has to be a cut-off at some point, for sanity's sake.

[iliajie]

I am down with that entire list! Default FPM?

Default FPM - no problem and great! However, removing/hiding CGI/FCGI completely is not an option, at least from my point of view. I assume Joe wanted this for consistency with RHEL. It's understandable. However, CGI/FCGI modes are available and supported on Debian/Ubuntu, and what's more in certain cases can be preferred over FPM.

Besides, setting per-directory PHP version, which works only with CGI/FCGI modes might be a needed feature in some cases, and is very well implemented and works very nicely (thanks to Jamie).

[iliajie]

PHP-FPM only. Hide all other execution modes, by default, and make sure PHP-FPM always works, every time.

Yes, PHP-FPM will work, however I strongly suggest leaving FCGI and CGI modes. We may need to distribute a patched version of suexec and replace it with config check or provide the whole httpd package from our repos for all CentOS/RHEL/Rock/Alma 8 users to have FCGI working.

Newest available PHP version, by default, unless Script Installer requires something older?

Yes, this is what we do already. We just install PHP provided by OS repos. This is the best practice, I think. If users need other versions they are welcome to install whatever they want manually.

However, at the moment it's not the case for CentOS 7, which has PHP 5.4 provided by OS repos -- what do we do with CentOS 7 by the way -- perhaps hide it (not remove) from installer too? We could stop using SCL repo for CentOS 7 and not install two PHP versions 5.4 and 7.2 as we do now but only install one PHP 7.3 or PHP 7.4 from REMI non-safe repos, which will suppress default installation of PHP 5.4 completely. Although, I'd rather just left things as they are but hidden CentOS 7 from the installer.

Make mod_php invisible on new installs, and impossible to turn on without editing config files manually. It should never exist in a new Virtualmin system and it should not be easy to install/enable it. Maybe make Virtualmin stack packages conflict with mod_php so people don't accidentally install it

This is already the case, we don't make mod_php available by default on new installs. Forcing a user not to install it, is not something we should do, I think. For instance existing Z-Push install script requires it. There might be developers that could need this for testing purposes. There might be something else. I think, we're good here already, no work needed.

More customizable? Probably just make --bundle with --include/--exclude a little smarter.

I have actually spent some time in the past fixing non working --exclude option. Making it smarter could be a major rework, I don't think we have nor a need or resources for doing it right now.

---

@jcameron @swelljoe I have added one more checklist option that we must pay attention to, I'm sure. Have a look. What you think?

[marclaporte]

About "Update docs" and "Update hero and promo materials on website": Do you see these as blockers to the release?

If so, may I suggest a two-phased approach?
1- Release v7 beta without much fanfare, and let the active community (in the forums) help make it more robust, focusing on the code.
2- After 3 to 6 months of testing/refining, release 7.0, where we mostly focus on marketing / promotion / docs.

Thanks!

P.S. I see Beta 1 here:
63a4a2d
But not here:
https://github.com/virtualmin/virtualmin-install/releases

[iliajie]

I think hero for Virtualmin.com was updated already?

Promo in the process, planning to work on it as part of index page renovation.

We must find the way to re-import docs properly. This is also on my to-do.

I think testing shouldn't take that much time. Jamie, what do you think of creating devel packages for testers? I don't think everyone feels comfortable using Git.

[iliajie]

Balance what Virtualmin related packages are installed using upgrade.cgi and what we install with stack/groups. For both installs (GPL and Pro), in my opinion, we don't need to install anything much of extra. I would suggest only installing -virtualmin-awstats and -virtualmin-htpasswd (latter for LAMP only), without installing -virtualmin-git, -virtualmin-init, -virtualmin-registrar, -virtualmin-slavedns, -virtualmin-sqlite which are rarely used. This will also be a good thing for making UI less loaded.

@swelljoe Joe, I would also suggest not to install virtualmin-mobile-theme package at all, which globally disables Authentic Theme for mobile devices.

[iliajie]

I have fixed installing extra packages upon upgrade to Pro, which now will only install Virtualmin Pro package (as expected).

Also, default clean install will only get virtualmin-awstats and virtualmin-htpasswd packages installed, without installing anything less popular.

[chris001]

Similar key errors trying to install 7.0-RC4 on Ubuntu 20.04 LTS.
Seems the 7.0-RC4 script is trying, and failing, to use the Virtualmin 6 key (D9F9010760D62A6B) for the Virtualmin 7 install ?!

Downloading repository metadata                                         [ERROR] Failed with error: 100
[  ✘  ]

[ERROR] Something went wrong. Exiting.
[ERROR] The last few log entries were:
Hit:4 http://us.archive.ubuntu.com/ubuntu focal-security InRelease
Get:5 https://software.virtualmin.com/vm/7/gpl/apt virtualmin-focal InRelease [10.7 kB]
Err:5 https://software.virtualmin.com/vm/7/gpl/apt virtualmin-focal InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D9F9010760D62A6B
Get:6 https://software.virtualmin.com/vm/7/gpl/apt virtualmin-universal InRelease [10.7 kB]
Err:6 https://software.virtualmin.com/vm/7/gpl/apt virtualmin-universal InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY **D9F9010760D62A6B**
Reading package lists...
W: GPG error: https://software.virtualmin.com/vm/7/gpl/apt virtualmin-focal InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D9F9010760D62A6B
E: The repository 'https://software.virtualmin.com/vm/7/gpl/apt virtualmin-focal InRelease' is not signed.
W: GPG error: https://software.virtualmin.com/vm/7/gpl/apt virtualmin-universal InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D9F9010760D62A6B
E: The repository 'https://software.virtualmin.com/vm/7/gpl/apt virtualmin-universal InRelease' is not signed.
Downloading repository metadata: [2022-04-30 03:57:35 UTC] [ERROR] Failed with error: 100
[2022-04-30 03:57:35 UTC] [ERROR] Something went wrong. Exiting.
[2022-04-30 03:57:35 UTC] [ERROR] The last few log entries were:
chris@u2004to2204:~$ sudo apt-key list
/etc/apt/trusted.gpg
--------------------
pub   rsa4096 2017-05-01 [SC] [expires: 2024-04-29]
      E36F 0664 7D8E BD2B E364  2BCE D9F9 0107 60D6 2A6B
uid           [ unknown] Virtualmin, Inc. (Package signing key for Virtualmin 6) <[email protected]>
sub   rsa4096 2017-05-01 [E] [expires: 2024-04-29]

pub   rsa4096 2022-01-08 [SC] [expires: 2029-01-06]
      586C 427B 9590 C1C4 35A5  FE39 3E57 0892 B9A0 B8B7
uid           [ unknown] Virtualmin, Inc. (Package signing key for Virtualmin 7) <[email protected]>
sub   rsa4096 2022-01-08 [E] [expires: 2029-01-06]

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      790B C727 7767 219C 42C8  6F93 3B4F E6AC C0B2 1F32
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2012) <[email protected]>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      8439 38DF 228D 22F7 B374  2BC0 D94A A3F0 EFE2 1092
uid           [ unknown] Ubuntu CD Image Automatic Signing Key (2012) <[email protected]>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub   rsa4096 2018-09-17 [SC]
      F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2018) <[email protected]>

chris@u2004to2204:~$

[chris001]

@iliajie @swelljoe Thanks for your work getting out this awesome 7.0 update to the installer! The note on release 7.0.0-RC4 says "Fix Ubuntu 22.04 and Debian 11 installation", however we're getting fatal errors running it on fresh Ubuntu 20.04 LTS and 22.04 LTS servers, as you can see above, the script is halting on the vm 7 repo "because the public key is not available", and on the universal repo, "InRelease is not signed". Expect the script to handle all details to succeed, however, is there some required step we're not doing right?! What do we need to do to get this to install 7.0 without fatal script error halting it?! Because this script is the only supported way we can install the higher performance LEMP bundle with nginx on 22.04 LTS which has the much faster PHP 8. Thanks.

[swelljoe]

The repos for 22.04 and Debian 11 do not exist yet.

[chris001]

Thanks! However we also get fatal key errors above while trying to run installer 7.0.0-RC4 on 20.04 Focal also, yet the focal repos seem to exist?

[swelljoe]

They're old copies of VM6 repos we used during development. The only Virtualmin 7 install that is expected to work (and that has been announced to work) is RHEL and derivatives.

[chris001]

More customizable? Probably just make --bundle with --include/--exclude a little smarter.

Agreed. Make it more foolproof! Result should be fewer support requests?

1. The installer should Install all required packages. An install on 22.04 LTS several packages were missing.
2. Smart function to add lines to the repo sources if not already added.
3. Smart function to add repo keys if not already added.
4. Modifications to postfix/dovecot config should be upgrade safe.
5. Running the script again should be non destructive to the server in case there was an error that halted it in the middle and you have to run it again. Option to reset the installed config to original factory settings.
6. Options to add and remove LEMP and LAMP to a running virtualmin, for those who installed one web server but want the other one, commonly happens.

[swelljoe]

An install on 22.04 LTS several packages were missing.

I've said repeatedly that 22.04 is not finished. It is not expected to work. I don't know how to be more clear about that. Of course packages are missing, I haven't published the damned things yet.

[MattWilcox]

Welcome to the Virtualmin GPL installer, version 7.0.0-RC4

This script must be run on a freshly installed supported OS. It does not
perform updates or upgrades (use your system package manager) or license
changes (use the "virtualmin change-license" command).

The systems currently supported by install script are:

Red Hat Enterprise Linux derivatives
  - Alma Linux and Rocky 8 on x86_64
  - CentOS 7 and 8 on x86_64
  - RHEL Linux 7 and 8 on x86_64

Debian Linux derivatives
  - Ubuntu 20.04 LTS and 22.04 LTS on i386 and amd64
  - Debian 10 and 11 on i386 and amd64

Then this messaging is a mistake causing issues - it says explicitly that the software is intended to work on Ubuntu 22.04 LTS.

[iliajie]

Then this messaging is a mistake causing issues - it says explicitly that the software is intended to work on Ubuntu 22.04 LTS.

No it's not, as your are using development version of install.sh script, and development assumes that something may not work, and it doesn't work for Debian 11 and Ubuntu 22.04 at moment.

It is coming soon.

[MattWilcox]

Devs seem to be assuming that anyone seeing that message knows the caveats about it, when it seems somewhat clear that's not the case.

I'm just trying to stop headaches from unclear messaging and show why people may be assuming a thing which devs believe shouldn't be assumed.

I'd also add that many people see an RC - a release candidate... and believe its supposed to be feature complete. It is, after all, a release candidate. So anything that's in the software "should work here now" - if it doesn't, and that's expected, is it really an RC?

Regardless, much appreciate the work being done and personally am patiently waiting :)

[MattWilcox]

I appreciate it, and the work you do - so thanks :)

[MattWilcox]

Cheers for all the hard work, while I can't beta test this it looks like a really great series of updates.