kudobuilder / kuttl Goto Github PK
View Code? Open in Web Editor NEWKUbernetes Test TooL (kuttl)
Home Page: https://kuttl.dev
License: Apache License 2.0
KUbernetes Test TooL (kuttl)
Home Page: https://kuttl.dev
License: Apache License 2.0
The newly added technique to start a background process is working in testing but leaves the process running after exit. It isn't clear yet why... theories:
for ref: https://medium.com/@felixge/killing-a-child-process-and-all-of-its-children-in-go-54079af94773
It would be worth added a retry
to commands... this would retry to the command upon failure. Controls might include:
Perhaps it could be useful to define command aliases for a testsuite for long pathed commands that are repeated.
example:
commands:
- command: ../../../bin/dispatch -n dispatch-tasks serviceaccount create multi-namespace
RunCommands
in kubernetes.go
returns background processes which we ignore for kubectl
commands and steps
. However there should be an error or warning and that command should not leak. (which it currently does)
currently it is possible to specify a CRD dir in TestSuite after which there is a CRD wait.
It is also documented that a CRD could be the result of a TestStep with a pattern to follow to "wait" via an Assert.
It would be good to support a way to waiting other that TestSuite. It would be good to support CRD via a command exec from TestSuite.
If possible we should:
k kudo init --crd-only
could be used to define the CRD (which would wait)Since KUTTL has it's own domain, we should update the CRDs to be kuttl.dev/v1 (or v1beta1)
for a bad kuttl-test.yaml for with kuttl v0.1.0 the kuttl cli panics... please don't
to duplicate... change apiVersion: kudo.dev/v1beta1
to apiVersion: kuttl.dev/v1beta1
we should probably also switch to kuttl.dev
apiVersion: kudo.dev/v1alpha1
kind: TestStep
delete:
- apiVersion: v1
kind: ConfigMap
name: my-static-test
namespace: rq
kubectl:
- create configmap my-static-test --from-file=bundle=/tmp/bundle.zip --namespace=rq
The kuttl cli overrides the testsuite file... with the exception of the test env. (this is a require to fix that).
If the kuttl-test.yaml
has a configuration for startControlPlane: true
and the cli is started with --start-kind
.... the cli override should start kind.
Perhaps we should have an override that allows to point at a cluster (currently it is the default that is selected if nothing is provided).
When kuttl runs commands specified in a TestStep or TestSuite, they can run forever as we use exec.Command
and not exec.CommandContext
.
https://github.com/kudobuilder/kuttl/blob/master/pkg/test/utils/kubernetes.go#L929
According to KEP-05:
The cmd package should provide conveniences for running arbitrary commands (i.e. shelling out) from Go code in integration tests.
Current functionality:
package cmd
func Exec(
command string,
arguments []string,
environmentVariables map[string]string,
) (
exitStatus int,
stdout *bytes.Buffer,
stderr *bytes.Buffer,
error error,
)
KUDO needs to use this new dependency
kudo test
What happened:
While working on a test that includes a webhook, I noticed that timeout
parameter is not used when the test step fails in the apply-phase:
harness.go:66: going to run test suite with timeout of 3000000 seconds for each step
--- FAIL: kudo/harness (0.02s)
--- FAIL: kudo/harness/terminal-failed-job (7.67s)
logger.go:37: 14:51:05 | terminal-failed-job | Ignoring failjob-operator as it does not match file name regexp: ^(\d+)-([^.]+)(.yaml)?$
logger.go:37: 14:51:05 | terminal-failed-job | Ignoring job-timeout-operator as it does not match file name regexp: ^(\d+)-([^.]+)(.yaml)?$
logger.go:37: 14:51:05 | terminal-failed-job | Creating namespace: kudo-test-heroic-dodo
logger.go:37: 14:51:05 | terminal-failed-job/0-install | starting test step 0-install
logger.go:37: 14:51:05 | terminal-failed-job/0-install | running command: {kubectl kudo install --instance failjob-operator-instance ./failjob-operator %!s(bool=true) %!s(bool=false) %!s(bool=false)}
logger.go:37: 14:51:12 | terminal-failed-job/0-install | Error: failed to install instance failjob-operator-instance: installing Instance: Internal error occurred: failed calling webhook "instance-admission.kudo.dev": Post https://kudo-controller-manager-service.kudo-system.svc:443/admit-kudo-dev-v1beta1-instance?timeout=30s: dial tcp 10.103.201.124:443: connect: connection refused
logger.go:37: 14:51:12 | terminal-failed-job/0-install | operator.kudo.dev/v1beta1/failjob-operator created
operatorversion.kudo.dev/v1beta1/failjob-operator-0.1.0 created
case.go:156: exit status 255
case.go:159: failed in step 0-install
The code suggests that the timeout
is only respected during the assertion phase of the step. I often use the timeout to "pause" the execution of the failing step and look into the cluster (using kubectl
). A number of k8s admission controllers might reject applied resources so it might be beneficial to extend timeout to the whole step and not just the assertion phase.
What you expected to happen:
The timeout
should be applied to the whole step.
How to reproduce it (as minimally and precisely as possible):
Using any step command that fails (e.g. false
)
Anything else we need to know?:
Environment:
kubectl version
):kubectl kuttl version
):uname -a
):What happened:
We currently use a very naive loop for implementing the test timeout:
The timeout parameter currently means the number of times the check is run and not the number of seconds. Instead a Context should be used to enforce the timeout.
Add ability to have before each and after each conditions defined in TestSuite to be applied / asserted before and after each test
A great strategy for "ensuring" a controller or resources is to have a step 0 assert of that instance running
see kudobuilder/kudo#1473
test/e2e/cli-install-uninstall/00-assert.yaml
apiVersion: v1
kind: Pod
metadata:
labels:
app: kudo-manager
control-plane: controller-manager
namespace: kudo-system
status:
phase: Running
Assuming you may want this as a precondition on all tests... it would be great to assert this at the end of TestSuite setup prior to tests.
There are two issues with the built-in kind cluster functionality (that are likely related):
kind delete cluster
was not ran and --skip-cluster-delete
was set), kuttl will still attempt to use it and hang forever.These may be regressions related to the kind version bump from 0.5.1 to 0.7.0 as upstream kind changed a lot of the API and how kind clusters are managed.
Perhaps pull kudo docs and modify to be generic
The CLI should function without a kuttl-test.yaml
as long as all required details are provided
for instance k kuttl test --start-control-plane pkg/test/test_data/
should run without a file
Asserts today are easy for YAML or K8s objects... however there are users wanting to do a regex match against pod logs. Is there a way to make this easier?
or at a min, allow a command used for assertion that can be repeat for each test.
perhaps there is something more here... pull and logging a pod log to test might be useful.
According to KEP-05:
Harness
in harness.go
).We should add a new setting to the commands struct called script
that accepts a bash script as a string that we run. When running the script, we can set the $NAMESPACE
and $KUBECONFIG
variables. This will allow users to more easily write scripts inline when necessary.
apiVersion: kudo.dev/v1alpha1
kind: TestStep
commands:
- script: until ! kubectl get pod --namespace $NAMESPACE my-pod; do sleep 1; done
According to KEP-05:
Additionally, Operator-specific helper methods will be provided in the library that can be used by library users.
We will also endeavor to stay close to the API used by the kudo-cassandra-operator testing library to ensure that migration of existing tests is easy.
What happened:
Currently, the test harness panics if it cannot find etcd and the kube-api-server and start-control-plane
is set. We should handle this more nicely. Either we download them for the user or bail out with a nice error.
The k8s package should provide conveniences for interacting with Kubernetes objects.
This should likely include WaitFor functions
package k8s
func Init(kubectlOptions *kubectl.KubectlOptions) error
func CreateNamespace(namespaceName string) error
func DeleteNamespace(namespaceName string) error
Need a way to define an assertion that is true for every test in the testsuite
we will need to be clear if this is every teststep or every test... perhaps configurable.
User should be able to define a yaml assertion that is tested for each test and not need to repeat that for every test configuration. This is part of the spirit of the composability KEP.
currently kubebuilder with etcd
is required and needs to be documented as such
According to KEP-05:
There should be a shared testing library that provides all necessary facilities for complex KUDO-based operators to have extensive and thorough integration tests.
Take a look at the kudo-cassandra-operator testing facilities to get a better understanding to how this differs from the current KUDO test harness, and provides functionality that is still not clear how to implement in the KUDO test harness, including a single verbose and useful test run output, useful for development and CI.
There's a private protokep for this in Google Drive.
What would you like to be added:
In the test harness, you can match objects without specifying a name, for example:
apiVersion: v1
kind: Pod
metadata:
labels:
app: nginx
status:
state: READY
It should be possible to specify how many objects match.
Why is this needed:
This can be used to verify that a certain number of pods are ready or a certain number of plan executions exist.
The best part about data-driven tests is we can easily perform fuzz testing and create generative tests around parameters. @jbarrick-mesosphere wdyt?
this is a feature request, for the kuttl API to provide a means to obtain the Pod log when a test step executes a Pod, the API would let you get back the Pod log via the API call to execute a Test Step.
Our use case includes running custom container images within a Test Step, those images will need to produce log output in a particular format (e.g. v1alpha2 in our case), then we can aggregate or do other processing on the test log output, our test logs from the pods will show various kinds of feedback useful for the end-user (e.g. suggestions or error descriptions).
According to KEP-05:
The kubectl package should provide conveniences for interacting with a Kubernetes cluster as one would do via kubectl from the shell.
package kubectl
func Exec(
namespaceName string,
containerName string,
command string,
argvuments []string,
environmentVariables map[string]string,
) error
Should we have a kuttl plugin? seems useful.
According to KEP-05:
What happened:
There is a race condition when creating namespaces in Kubernetes that if you create a Pod in the namespace before the ServiceAccount exists, it will fail.
We should resolve this for the tests by waiting for the default
ServiceAccount to exist in created namespaces before continuing on to the test steps to prevent errors like:
case.go:102: pods "influxdb-querier" is forbidden: error looking up service account kudo-test-lenient-lab/default: serviceaccount "default" not found
What would you like to be added:
Add a gracePeriod or timeout to the error teststep type.
Why is this needed:
I have a controller that migrates one type of object to another. I want to be able to specify that after some time, my original object no longer exists. Currently I cannot do that as it may take some amount of time for the controller to pick up the original object, create the derived object, and remove the original. Currently, the error teststep would fail immediately as the original object will likely still exist.
[ ] go run cmd/kubectl-kuttl/main.go test pkg/test/test_data/
[] CI setup to run tests and ^^ tests
[] update makefile for tests and integration-test
[] circle ci
[] goreleaser release
[] .git support for templates and PR defaults
For presentation mentioned on #38
possibly this script: https://github.com/kubernetes/kubernetes/blob/master/test/cmd/apps.sh
StartKUDO bool json:"startKUDO"` should be removed
When the BG process is started KUTTL moves forward with other commands (and tests).. however the commands could need the BG process to be ready to success. We do want to the process in the background however we need to know when it is live.
An example use case:
commands:
- command: ./bin/kubectl-kudo init --crd-only - command: ./bin/kubectl-kudo init --crd-only
- command: ./bin/manager
background: true
- command: ./bin/kubectl-kudo install --skip-instance ./repository/cassandra/3.11/operator/
The kudo install
command will fail if manager isn't ready. We need manager to be "ready" before the install
command.
We need a contribs doc...
in it we need to make sure devs are aware that we are dependent on kubebuilder... along with other dependents and requirements to get started.
seeing projects is commonly looking for a configmap for a teststep
example
apiVersion: kudo.dev/v1alpha1
kind: TestStep
commands:
- command: 'kubectl create configmap example-cm --from-file=../../jobs/tensorflow.py -n tensorflow-testing'
looks like a need for create, from file, from values with a namespace
Currently, the test harness polls for object state when tests are running which offers opportunity for races (and is likely more inefficient). For example, to test delete steps in KUDO properly, we need to be able verify that a Job is created and is completed before it gets deleted, without racing KUDO, if we poll, there is a chance we can miss the Job before KUDO deletes it.
Based on KEP-05
A generic Kubernetes client (currently this is contained inside of pkg/test/harness.go
)
KUTTL currently always creates namespaces for each test step, but some users may want to run their tests in a specific namespace or target an existing deployment. This would also be useful for running kudo test
in less privileged environments where it is unable to create namespaces.
It is possible today to target static namespaces by specifying them on any commands or manifests, but it would be useful to add a --namespace
flag to kudo test that controls the namespace used.
For example, if --namespace
is set on kudo test
, instead of creating namespaces for each test step and using the created namespace as the default for each resource, kudo test
will set the default namespace to the --namespace
flag and use that instead.
This code was pulled from kudo... where there was a runKUDO
function that injected kudo with the kudo
flag.
We need a way to declare a controller to add... or a lifecycle hook
kuttl.io / kuttl.dev either.. both... it would be nice to have this in place prior to April 1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.