Flaky: e2e TestGameServerCreationAfterDeletingOneExtensionsPod about agones

[ashutosji]

I am able to reproduce this issue locally.

Steps:

1. Installed dev version of Agones in cluster
2. Scaled agones-extensions deployment replicas to 4 using this command: kubectl scale deployment agones-extensions --replicas=4 -n agones-system

deployment.apps/agones-extensions scaled

3. Then, I ran go test -count=1 on https://github.com/googleforgames/agones/tree/main/test/e2e/extensions and got below error:

WARN[2024-01-30 08:37:30.546] Error creating inClusterConfig, trying to build config from flagsunable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined  error="unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined" source=framework
INFO[2024-01-30 08:37:30.548] Starting e2e test(s)                          cloudProduct=generic featureGates="CountsAndLists=false&DisableResyncOnSDKServer=false&Example=false&FleetAllocationOverflow=true&GKEAutopilotExtendedDurationPods=false&PlayerAllocationFilter=false&PlayerTracking=false" gameServerImage="us-docker.pkg.dev/agones-images/examples/simple-game-server:0.25" namespace= perfOutputDir= pullSecret= stressTestLevel=0 version=
INFO[2024-01-30 08:37:30.548] Cleaning up now.                             
INFO[2024-01-30 08:37:31.597] Finished cleanup.                            
INFO[2024-01-30 08:42:31.773] Length of pod list is 4                       test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Name of extensions pod 0: agones-extensions-7d5bbd6bb8-4xrc6  test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Host IP 10.138.0.51                           test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Pod IPs [{10.72.11.73}]                       test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Name of extensions pod 1: agones-extensions-7d5bbd6bb8-dkj5p  test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Host IP 10.138.0.49                           test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Pod IPs [{10.72.9.6}]                         test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Name of extensions pod 2: agones-extensions-7d5bbd6bb8-p7b2j  test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Host IP 10.138.0.44                           test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Pod IPs [{10.72.4.8}]                         test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Name of extensions pod 3: agones-extensions-7d5bbd6bb8-v2gnd  test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Host IP 10.138.0.43                           test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Pod IPs [{10.72.3.22}]                        test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 08:42:31.773] Removing one of the Extensions Pods: agones-extensions-7d5bbd6bb8-dkj5p  test=TestGameServerCreationAfterDeletingOneExtensionsPod
panic: test timed out after 10m0s
running tests:
        TestGameServerCreationAfterDeletingOneExtensionsPod (10m0s)

goroutine 705 [running]:
testing.(*M).startAlarm.func1()
        /usr/lib/google-golang/src/testing/testing.go:2380 +0x385
created by time.goFunc
        /usr/lib/google-golang/src/time/sleep.go:177 +0x2d

goroutine 1 [chan receive, 10 minutes]

Similarly, When I scale replica to 1 then i got below error which is expected behavior as per test:

WARN[2024-01-30 07:11:50.114] Error creating inClusterConfig, trying to build config from flagsunable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined  error="unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined" source=framework
INFO[2024-01-30 07:11:50.116] Starting e2e test(s)                          cloudProduct=generic featureGates="CountsAndLists=false&DisableResyncOnSDKServer=false&Example=false&FleetAllocationOverflow=true&GKEAutopilotExtendedDurationPods=false&PlayerAllocationFilter=false&PlayerTracking=false" gameServerImage="us-docker.pkg.dev/agones-images/examples/simple-game-server:0.25" namespace= perfOutputDir= pullSecret= stressTestLevel=0 version=
INFO[2024-01-30 07:11:50.116] Cleaning up now.                             
INFO[2024-01-30 07:11:51.197] Finished cleanup.                            
INFO[2024-01-30 07:16:51.362] Length of pod list is 1                       test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 07:16:51.362] Name of extensions pod 0: agones-extensions-7d5bbd6bb8-p7b2j  test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 07:16:51.362] Host IP 10.138.0.44                           test=TestGameServerCreationAfterDeletingOneExtensionsPod
INFO[2024-01-30 07:16:51.362] Pod IPs [{10.72.4.8}]                         test=TestGameServerCreationAfterDeletingOneExtensionsPod
--- FAIL: TestGameServerCreationAfterDeletingOneExtensionsPod (300.16s)
    high_availability_test.go:36: 
                Error Trace:    /usr/local/google/home/ashutoshnsingh/Documents/googleforgames/agones/test/e2e/extensions/high_availability_test.go:36
                Error:          Received unexpected error:
                                context deadline exceeded
                Test:           TestGameServerCreationAfterDeletingOneExtensionsPod
    high_availability_test.go:46: 
                Error Trace:    /usr/local/google/home/ashutoshnsingh/Documents/googleforgames/agones/test/e2e/extensions/high_availability_test.go:46
                Error:          "1" is not greater than "1"
                Test:           TestGameServerCreationAfterDeletingOneExtensionsPod
                Messages:       Cluster has no Extensions pod or has only 1 extensions pod
panic: runtime error: index out of range [1] with length 1 [recovered]
        panic: runtime error: index out of range [1] with length 1

goroutine 56 [running]:
testing.tRunner.func1.2({0x1a9de40, 0xc00091fe48})
        /usr/lib/google-golang/src/testing/testing.go:1631 +0x24a
testing.tRunner.func1()
        /usr/lib/google-golang/src/testing/testing.go:1634 +0x377
panic({0x1a9de40?, 0xc00091fe48?})

[ashutosji]

I did bit debugging, the wait.PollUntilContextTimeout() function is getinng false and nil value on each poll and resulting it's terminate polling after timeout duration here: https://github.com/googleforgames/agones/blob/main/test/e2e/extensions/high_availability_test.go#L108
And this is happening because of this if condition: https://github.com/googleforgames/agones/blob/main/test/e2e/extensions/high_availability_test.go#L114-L116