Cilium connectivity test leaves state behind in the cluster on unclean exit about cilium-cli

[ti-mo]

The work in #215 adds context cancellation to most parts of the test runner, and will already remove policies applied during the currently-active test when ctrl+C is hit.

I would like to extend this by adding cilium-test namespace cleanup behaviour and making it opt-out with a --preserve or --skip-cleanup flag of some sort to allow for quick iteration during development. How does that sound?

[joestringer]

I'm somewhat inclined to just close this issue out cc @tklauser @michi-covalent. While I think that we may be able to improve the UX in this area, I'm not sure it's such a pain point for users or developers that someone is interested to dig into this issue and propose improvements.

[fujitatomoya]

cilium/cilium#30428 is proposed doc fix, @joestringer @blackliner it would be appreciated if you can take a look.

[joestringer]

Related, I ran the cilium connectivity test in a GKE cluster and it successfully completed, then it left behind cilium-test pods and namespace. When I tried to delete it, kubectl hung for 20 minutes or more. Anyone seen this before?

$ k delete ns cilium-test
namespace "cilium-test" deleted

(Yes, workaround is to ^C and you're good to go, but why is it hanging?)

[tgraf]

Simplest answer here is probably just firing off a best-effort request to the apiserver to delete the cilium-test namespace upon exit (including when triggered via a signal).

Delete is not being done on purpose right now to allow re-running tests quickly. We can make that conditional on a flag though.

[fujitatomoya]

Related, I ran the cilium connectivity test in a GKE cluster and it successfully completed, then it left behind cilium-test pods and namespace. When I tried to delete it, kubectl hung for 20 minutes or more. Anyone seen this before?

Just FYI, this does not happen to me with the following versions.

Client Version: v1.26.7
Kustomize Version: v4.5.7
Server Version: v1.26.12

[joestringer]

Simplest answer here is probably just firing off a best-effort request to the apiserver to delete the cilium-test namespace upon exit (including when triggered via a signal).

[tgraf]

I think it is hanging because of a finalizer not working correctly. It probably happens for all namespaces.

[ti-mo]

Also, it looks like 78bdb23 added support for cleaning up the cilium-test namespace which feels a bit incoherent as cilium install doesn't deploy a cilium-test namespace. This should all be managed by the test harness instead.

[joestringer]

I would like to extend this by adding cilium-test namespace cleanup behaviour and making it opt-out with a --preserve or --skip-cleanup flag of some sort to allow for quick iteration during development. How does that sound?

I like this idea. The CLI should work reasonably for the user first, then if developers want something special we can have a dedicated flag or developer mode for common use cases there. In the Cilium Ginkgo CI we have the "holdEnvironment" flag, I think I saw @jrajahalme propose a similar flag for this CLI recently too so you can run a test and stop in the middle if something fails. Might make sense to integrate such "don't clean up" logic into that too.

Also, it looks like 78bdb23 added support for cleaning up the cilium-test namespace which feels a bit incoherent as cilium install doesn't deploy a cilium-test namespace. This should all be managed by the test harness instead.

➕

[disparo13]

Do you know if there's any progress on that?

[joestringer]

I don't believe that any of the behaviour has changed since the original report. By now, I think it's mostly expected by users of the connectivity test that the namespace and Pods remain after the test.

[blackliner]

What is the official recommendation to manually cleanup? https://docs.cilium.io/en/stable/contributing/testing/e2e/ should maybe document this?