[flaky-test] Certificate Rotation about rancher

[snasovich]

Reopening to track unpinning per #45577 (comment). Changing milestone so it doesn't appear it's blocking 2.9.0 release.

[snasovich]

@Oats87 , yes - as this one is for the flaky tests that were addressed by pinning the version so using this one to track version unpinning. #45577 (comment)

[andreas-kupries]

While I am not seeing the flakiness of the cert rotation I see a more general breakage reported around cluster provisioning, it seems.

This is for a PR sitting on release/v2.9: #45269

In case it matters, a local k3s-based Rancher starts up just fine for that PR.

My latest drone logs are at

• https://drone-pr.rancher.io/rancher/rancher/39161
• https://drone-pr.rancher.io/rancher/rancher/39153

The message is generally the same across various failures:

... failed on: prov cluster is not ready: timeout waiting condition: context deadline exceeded

Question: Is each test creating its own cluster ? And removing it later ?
Because I see that there are passing tests too.

I see only 2 tests fail in in each of the 5 provisioning stages, mostly different across the stages.

Failing: Test_Provisioning_Custom_OneNodeWithDelete, Test_Provisioning_MP_SingleNodeAllRolesWithDelete, Test_Provisioning_Custom_ThreeNode, Test_Operation_SetA_Custom_CertificateRotation, Test_Operation_SetA_MP_CertificateRotation, Test_Operation_SetB_Custom_EtcdSnapshotOperationsOnNewCombinedNode, Test_Operation_SetB_MP_EtcdSnapshotOperationsWithThreeEtcdNodesOnNewNode

In the build-pr failures I see the same failed on ... message, after the unit tests were run and passed.

[andreas-kupries]

Created a PR without material code changes (comment fix).
Seeing failed builds there too, see https://drone-pr.rancher.io/rancher/rancher/39168
However none of the context deadline exceeded from my branch :(
Now wondering if the addition of the status field, and its handling slowed something down enough to trigger these timeouts.

[bigkevmcd]

Yeah, I'm still getting Test_Operation_SetB_MP_EtcdSnapshotOperationsWithThreeEtcdNodesOnNewNode failing.

For the nature of the change in this PR #45572, this should just not be affected.

[Oats87]

Doing investigation into this, I'm seeing that there were issues with operations taking significantly longer after v1.27.11+rke2r1 was released. It was almost a 200 second difference in my benchmark setup.

As such, as a temporary workaround, we can pin the RKE2 version to v1.27.10+rke2r1 for now, which should hopefully unblock CI.

[snasovich]

Adding to a milestone and some additional labels to ensure we circle back on this and address version pinning.

[slickwarren]

no QA required - closing this issue

[Oats87]

@snasovich do we want to have two issues that track one PR...?

[snasovich]

Moving to v2.9-Next2 as the prerequisite issue (#46034 (comment)) was moved to that milestone.