[BUG] An existing VM start asks for a pull-secret (when not provided) which leads to a possible interruption of the startup flow about crc

[gbraad]

Unable to repro with the pull-secret-file option:

PS> crc start
PS> crc stop     # had to resort to -f as the cluster did not properly stop (different issue then described. not pull-secret related?)
PS> crc start
WARN A new version (2.25.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.25.0/crc-windows-installer.zip
INFO Checking minimum RAM requirements
INFO Checking if running in a shell with administrator rights
INFO Checking Windows release
INFO Checking Windows edition
INFO Checking if Hyper-V is installed and operational
INFO Checking if Hyper-V service is enabled
INFO Checking if crc-users group exists
INFO Checking if current user is in crc-users and Hyper-V admins group
INFO Checking if vsock is correctly configured
INFO Checking if the daemon task is installed
INFO Checking if the daemon task is running
INFO Checking admin helper service is running
INFO Loading bundle: crc_microshift_hyperv_4.13.0_amd64...
CRC requires a pull secret to download content from Red Hat.
You can copy it from the Pull Secret section of https://console.redhat.com/openshift/create/local.
? Please enter the pull secret
Failed to ask for pull secret: interrupt
PS> crc config set pull-secret-file C:\Users\gbraad\Downloads\pull-secret (1).txt
Successfully configured pull-secret-file to C:\Users\gbraad\Downloads\pull-secret
PS> crc start
WARN A new version (2.25.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.25.0/crc-windows-installer.zip
INFO Checking minimum RAM requirements
INFO Checking if running in a shell with administrator rights
INFO Checking Windows release
INFO Checking Windows edition
INFO Checking if Hyper-V is installed and operational
INFO Checking if Hyper-V service is enabled
INFO Checking if crc-users group exists
INFO Checking if current user is in crc-users and Hyper-V admins group
INFO Checking if vsock is correctly configured
INFO Checking if the daemon task is installed
INFO Checking if the daemon task is running
INFO Checking admin helper service is running
INFO Loading bundle: crc_microshift_hyperv_4.13.0_amd64...
INFO Creating CRC VM for MicroShift 4.13.0...
INFO Generating new SSH key pair...
INFO Starting CRC VM for microshift 4.13.0...
INFO CRC instance is running with IP 127.0.0.1
INFO CRC VM is running
INFO Updating authorized keys...
INFO Check internal and public DNS query...
INFO Check DNS query from host...
INFO Starting Microshift service... [takes around 1min]
INFO Waiting for kube-apiserver availability... [takes around 2min]
INFO Adding microshift context to kubeconfig...
Started the MicroShift cluster.

Use the 'oc' command line interface:
  PS> & crc oc-env | Invoke-Expression
  PS> oc COMMAND
PS> crc status
CRC VM:          Running
MicroShift:      Running (v4.13.0)
RAM Usage:       919.5MB of 3.902GB
Disk Usage:      4.822GB of 16.1GB (Inside the CRC VM)
Cache Usage:     20.72GB
Cache Directory: C:\Users\gbraad\.crc\cache

[gbraad]

In the initial phase:

$ crc start -p pull-secret
$ crc stop

was the cluster in that case correctly operational? Without the crc status command I can not be sure about this, as it might well be an issue with the pull-secret-file itself.

[gbraad]

Need more information as I can not repro with these steps:

I did notice the following

PS> crc stop
INFO Stopping the instance, this may take a few minutes...
Stopped the instance
PS> crc status
CRC VM:          Running
MicroShift:      Stopped (v4.13.0)
RAM Usage:       449.3MB of 3.902GB
Disk Usage:      4.859GB of 16.1GB (Inside the CRC VM)
Cache Usage:     20.72GB
Cache Directory: C:\Users\gbraad\.crc\cache
PS> crc start -p "C:\Users\gbraad\Downloads\pull-secret (1).txt"
WARN A new version (2.25.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.25.0/crc-windows-installer.zip
INFO Loading bundle: crc_microshift_hyperv_4.13.0_amd64...
INFO A CRC VM for MicroShift 4.13.0 is already running
Started the MicroShift cluster.

Use the 'oc' command line interface:
  PS> & crc oc-env | Invoke-Expression
  PS> oc COMMAND
PS> crc status
CRC VM:          Running
MicroShift:      Stopped (v4.13.0)
RAM Usage:       449.3MB of 3.902GB
Disk Usage:      4.859GB of 16.1GB (Inside the CRC VM)
Cache Usage:     20.72GB
Cache Directory: C:\Users\gbraad\.crc\cache

So that means the cluster didn't really stop, did this happen somehow... or did the stop fail before?

This ties in with:

# This operation completes in 5 seconds as successfully 
crc start -p pull-secret

When the cluster was really stopped:

PS> crc stop -f
INFO Stopping the instance, this may take a few minutes...
Stopped the instance
PS> crc status
CRC VM:          Stopped
MicroShift:      Stopped (v4.13.0)
RAM Usage:       0B of 0B
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     20.72GB
Cache Directory: C:\Users\gbraad\.crc\cache
PS> crc start -p "C:\Users\gbraad\Downloads\pull-secret (1).txt"
WARN A new version (2.25.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.25.0/crc-windows-installer.zip
INFO Checking minimum RAM requirements
INFO Checking if running in a shell with administrator rights
INFO Checking Windows release
INFO Checking Windows edition
INFO Checking if Hyper-V is installed and operational
INFO Checking if Hyper-V service is enabled
INFO Checking if crc-users group exists
INFO Checking if current user is in crc-users and Hyper-V admins group
INFO Checking if vsock is correctly configured
INFO Checking if the daemon task is installed
INFO Checking if the daemon task is running
INFO Checking admin helper service is running
INFO Loading bundle: crc_microshift_hyperv_4.13.0_amd64...
INFO Starting CRC VM for microshift 4.13.0...
INFO CRC instance is running with IP 127.0.0.1
INFO CRC VM is running
INFO Check internal and public DNS query...
INFO Check DNS query from host...
INFO Starting Microshift service... [takes around 1min]
INFO Waiting for kube-apiserver availability... [takes around 2min]
INFO Adding microshift context to kubeconfig...
Started the MicroShift cluster.

Use the 'oc' command line interface:
  PS> & crc oc-env | Invoke-Expression
  PS> oc COMMAND
PS> crc status
CRC VM:          Running
MicroShift:      Running (v4.13.0)
RAM Usage:       928.2MB of 3.902GB
Disk Usage:      4.927GB of 16.1GB (Inside the CRC VM)
Cache Usage:     20.72GB
Cache Directory: C:\Users\gbraad\.crc\cache

works as expected as the scenario described. So need more information about the pull-secret-file used or if the status in between the runs was an incomplete stop?

[anjannath]

I am able to reproduce on linux

 ➜  crc git:(main) crc status
Machine does not exist. Use 'crc start' to create it

⠀ ➜  crc git:(main) crc start -p ~/Downloads/pull-secret
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking if running on a supported CPU architecture
INFO Checking if crc executable symlink exists
INFO Checking minimum RAM requirements
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd socket units
INFO Checking if systemd-networkd is running
INFO Checking if NetworkManager is installed
INFO Checking if NetworkManager service is running
INFO Checking if dnsmasq configurations file exist for NetworkManager
INFO Checking if the systemd-resolved service is running
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists
INFO Checking if libvirt 'crc' network is available
INFO Checking if libvirt 'crc' network is active
INFO Loading bundle: crc_microshift_libvirt_4.13.6_amd64...
INFO Creating CRC VM for MicroShift 4.13.6...
INFO Generating new SSH key pair...
INFO Starting CRC VM for microshift 4.13.6...
INFO CRC instance is running with IP 192.168.130.11
INFO CRC VM is running
INFO Updating authorized keys...
INFO Configuring shared directories
INFO Check internal and public DNS query...
INFO Check DNS query from host...
INFO Starting Microshift service... [takes around 1min]
INFO Waiting for kube-apiserver availability... [takes around 2min]
INFO Adding microshift context to kubeconfig...
Started the MicroShift cluster.

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc COMMAND

⠀ ➜  crc git:(main) crc stop
INFO Stopping the instance, this may take a few minutes...
Stopped the instance

⠀ ➜  crc git:(main) crc status
CRC VM:          Stopped
MicroShift:      Stopped (v4.13.6)
RAM Usage:       0B of 0B
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     30.03GB
Cache Directory: /home/anjan/.crc/cache

⠀ ➜  crc git:(main) crc start
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking if running on a supported CPU architecture
INFO Checking if crc executable symlink exists
INFO Checking minimum RAM requirements
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd socket units
INFO Checking if systemd-networkd is running
INFO Checking if NetworkManager is installed
INFO Checking if NetworkManager service is running
INFO Checking if dnsmasq configurations file exist for NetworkManager
INFO Checking if the systemd-resolved service is running
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists
INFO Checking if libvirt 'crc' network is available
INFO Checking if libvirt 'crc' network is active
INFO Loading bundle: crc_microshift_libvirt_4.13.6_amd64...
INFO Starting CRC VM for microshift 4.13.6...
INFO CRC instance is running with IP 192.168.130.11
INFO CRC VM is running
INFO Configuring shared directories
INFO Check internal and public DNS query...
INFO Check DNS query from host...
INFO Starting Microshift service... [takes around 1min]
CRC requires a pull secret to download content from Red Hat.
You can copy it from the Pull Secret section of https://console.redhat.com/openshift/create/local.
? Please enter the pull secret
interrupt

⠀ ➜  crc git:(main) crc start -p ~/Downloads/pull-secret
INFO Loading bundle: crc_microshift_libvirt_4.13.6_amd64...
INFO A CRC VM for MicroShift 4.13.6 is already running
Started the MicroShift cluster.

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc COMMAND

⠀ ➜  crc git:(main) crc status
CRC VM:          Running
MicroShift:      Unreachable (v4.13.6)
RAM Usage:       438.1MB of 3.84GB
Disk Usage:      4.798GB of 16.1GB (Inside the CRC VM)
Cache Usage:     30.03GB
Cache Directory: /home/anjan/.crc/cache

[adrianriobo]

PS> crc config set pull-secret-file C:\Users\gbraad\Downloads\pull-secret (1).txt

This step is not part of the flow, does it has any impact on the keyring? @gbraad , also the failure was shown to @anjannath and @praveenkumar as part of a meeting

So need more information about the pull-secret-file

The pull secret used is the one from the crc-qe user, the same used on every e2e execution, so I can confirm no issue in relation with the pull secret

or if the status in between the runs was an incomplete stop?

I did not check that, but now I see from @anjannath execution that status is reported as Stopped

[gbraad]

On Windows I am not able to reproduce this as described. I followed the exact instructions, and tried to otherwise repro it (as with the set config). But I can not get this unles the VM was only in a tristate: the VM was still running.

I believe it is unclear WHEN you break the pull-secret. Did you actuall provide the pull-secret amd let the start happen... or you interrupted it at the point you are supposed to provide he pull-secret?

When no pull-secret is provided by cancel:

? Please enter the pull secret
interrupt

the VM does not start in my case.

---

Otherwise, if the start did happen it would be interrupting a startup flow, and that will always result in an issue. That is only working in the start in terminal 1, stop in terminal 2 for which we have a cancel operation. This is also what the daemon uses. So I am confused as to what you are observing here. I miss some information to why it does not repro on my end on Windows.

[adrianriobo]

I interrupt when it is suppose I should provide it, and then execute the crc start -p ... which succeeds in 5 secs

[gbraad]

In short, you do not provide any pull-secret.

? Please enter the pull secret
interrupt

in that case, can you provide a crc status before the start with the pull-secret? I am curious if something got started somehow.

[gbraad]

@anjannath you tested on Linux, @praveenkumar on macOS and @adrianriobo on Linux?

[adrianriobo]

I tested it on Linux and Mac, so I assume on Windows was failing too...now I am running it on Windows ...

btw can we change back the issue name?

[gbraad]

@adrianriobo The run #3801 (comment) only was able to get into a 5 sec startup time for the crc start, as in that case the VM is started (running), but the cluster is not functional.

btw can we change back the issue name?

I am therefore not convinced we have found the issue. I am unsure this is related to the pull-secret being interrupted or not.

[gbraad]

Thanks for the call. It seems to be related to 'existing VM' that has a side-effect. It already starts in the background for the Microshift preset, and in the meantime also asks for the pull-secret.

pull-secret being interrupted

It is any interruption, but in this case we ask unnecessarily for the pull-secret. The VM is already starting ...

So when the crc start is given without -p, it will start the VM. This can be observed with another temrinal and running crc status. This means that the pull-secret question allows a break in the startup-flow. In short, it was waiting for the pull-secret to resume the startup (as in the regular OCP flow we do not ask for this; the VM already recorded the info).

[gbraad]

We will perform a startHost and follow this up with bundle checks like:

crc/pkg/crc/machine/start.go

Lines 551 to 552 in 557ec94

	if vm.bundle.IsMicroshift() {
	// **************************

and part of the sart we ask for a pull-secret:

crc/pkg/crc/machine/start.go

Lines 295 to 302 in 557ec94

	if !exists {
	telemetry.SetStartType(ctx, telemetry.CreationStartType)
	// Ask early for pull secret if it hasn't been requested yet
	_, err = startConfig.PullSecret.Value()
	if err != nil {
	return nil, errors.Wrap(err, "Failed to ask for pull secret")
	}

somehow we have determined that on the Microshift preset this is also necessary on the exist-side? Not sure where this is enforced

[cfergeau]

somehow we have determined that on the Microshift preset this is also necessary on the exist-side?

It's a bug with the microshift preset, I filed #3619 a while ago.

[gbraad]

This is the direct result of this: so in a way a duplicate.

[cfergeau]

on the other hand, this also describes a more general issue, which is not pull-secret related:

Abort the operation with [Ctrl + c]
Start the cluster with the pull secret
This operation completes in 5 seconds as successfully

If the startup is interrupted with ctrl+c, or because of an error, ... subsequent crc start won't detect/recover from it.

[gbraad]

Right, this is the issue I also talked about, however the move having this execute in the daemon might resolve this. As in that case a stop or interrupt will trigger the cancel operation:

crc/pkg/crc/machine/sync.go

Lines 31 to 32 in 557ec94

	startCancel context.CancelFunc

which allows for a 15 second timeframe to cancel the operation:

crc/pkg/crc/machine/sync.go

Lines 15 to 16 in 557ec94

	const startCancelTimeout = 15 * time.Second

The other issue is that stop on Windows is not reliable anymore.

subsequent crc start won't detect/recover from it.

Determining the state just by a 'Running' of the VM state is insufficient.

crc/pkg/crc/machine/start.go

Lines 709 to 710 in 557ec94

	if vmState != state.Running {
	return false, nil

We need to also include the actual cluster state. When the cluster is stopped or unreachable we can not just rely on a start/resume. In that case a restart seems more appropriate.

[praveenkumar]

In Openshift it doesn't fail because we first check if pull secret exist but here in microshift we are not doing it so hence this bug.

diff --git a/pkg/crc/machine/start.go b/pkg/crc/machine/start.go
index febd16..8ea5c4 100644
--- a/pkg/crc/machine/start.go
+++ b/pkg/crc/machine/start.go
@@ -1050,12 +1050,16 @@ func addPodmanSystemConnections(c *types.ConnectionDetails) error {
 
 func startMicroshift(ctx context.Context, sshRunner *crcssh.Runner, ocConfig oc.Config, pullSec cluster.PullSecretLoader) error {
        logging.Infof("Starting Microshift service... [takes around 1min]")
-       content, err := pullSec.Value()
-       if err != nil {
-               return err
-       }
-       if err := sshRunner.CopyDataPrivileged([]byte(content), "/etc/crio/openshift-pull-secret", 0600); err != nil {
-               return err
+
+       // If pull secret already present in the VM, then no need to copy it again.
+       if _, _, err := sshRunner.RunPrivileged("Checking if pull secret already present in the VM", "test", "-f", "/etc/crio/openshift-pull-secret"); err != nil {
+               content, err := pullSec.Value()
+               if err != nil {
+                       return err
+               }
+               if err := sshRunner.CopyDataPrivileged([]byte(content), "/etc/crio/openshift-pull-secret", 0600); err != nil {
+                       return err
+               }
        }
        if _, _, err := sshRunner.RunPrivileged("Starting microshift service", "systemctl", "start", "microshift"); err != nil {
                return err

[gbraad]

This is the 'ensure' mechanism, right?

[praveenkumar]

This is the 'ensure' mechanism, right?

yes, for pull secret which I think more pressing than kubeconfig file steps.

[gbraad]

In that case, this is a duplicate of #3619 and can be closed, unless there are objections. The stop issue on Windows is a seperate problem.

[praveenkumar]

/close

Duplicate of #3619

[openshift-ci]

@praveenkumar: Closing this issue.

In response to this:

/close

Duplicate of #3619

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.