Add command-line integration tests with bats about kubectx

[ahmetb]

I think this now can be closed. Thanks a lot @feloy and @doodlesbykumbi.

[feloy]

I would be happy to try and work at this

[feloy]

@doodlesbykumbi probably the kubectl command only, outputting a list of namespaces ("kubectl get namespaces" seems to be the only command used)

[doodlesbykumbi]

@feloy @ahmetb here's my attempt #117

[ahmetb]

@feloy sounds great!

I've developed CLI tests with bats some time ago at https://github.com/Azure/run-command-extension-linux/tree/master/integration-test/test, so my knowledge is limited.

Please feel free to fork & set up travis on your own fork to iterate, happy to answer any questions!

[ahmetb]

Also if you have a test plan (i.e. test cases you're planning to cover), please summarize here to discuss.

[doodlesbykumbi]

Feel free to edit this

Test framework

• Based on bats
• Runs in Travis CI

Test plan

It looks like @feloy has covered some of these but based on the help text of kubectx:

Public interface TCs

• no kubectl detected (non-zero exitcode)
• list the contexts
• list the contexts when no kubeconfig exists
• switch to context
• switch to the previous context
• switch to the previous context, but there's no previous context
• switch to non-existing context (non-zero exitcode)
• rename context
• rename current-context
• delete context
• delete current-context
• delete non-existing context
• help

kubens:

• TODO ADD MORE HERE
• fails to retrieve list of namespaces (non-zero exitcode)

Unit TCs

• Enumerate all the functions in ./kubectx

[ahmetb]

I edited a bit. I think you entirely omitted the kubens, so if you can fill out that would be great!

Here's the kinds of things I added (mostly negative/edge cases):

• no kubectl detected (non-zero exitcode)
• list the contexts when no kubeconfig exists
• switch to the previous context, but there's no previous context
• delete non-existing context
• switch to non-existing context (non-zero exitcode)
• fails to retrieve list of namespaces (non-zero exitcode) (for kubens)

I hope this gives a good idea about what I'm mainly looking for.

[feloy]

Hi @ahmetb @doodlesbykumbi
This test plan sounds good to me.

About unit-testing the internal functions of kubectx and kubens, a problem is that the test script won't be able to source the kubectx/kubens scripts (to get the functions definitions) without executing the main part. A solution would be to cut these scripts to have a library one and a main one, but I'm not sure that would be a good experience for the user: having, as today, a single file for each command is very convenient

[feloy]

@doodlesbykumbi on which tests do you prefer to work?

If you agree, you could work on the positive tests cases you listed in the plan, and I work on the negative/edge cases ones that Ahmet added.

[ahmetb]

Let's set up the test infra first:

1. can you create a PR with a test/ dir, with one or two simple tests?

• "kubectx -h" doesn't fail
• "kubens -h" doesn't fail

2. I'll create a PR adding travis CI with bats.

a problem is that the test script won't be able to source the kubectx/kubens scripts

I'm not interested in unit-testing methods but rather the CLI tool itself. (I'll change the issue title.) So we should just invoke ../kubectx in the test.

[feloy]

@ahmetb ok, i'll create aPR with 2 simple tests for kubectx / kubens

[ahmetb]

BTW I looked at bats, last time I used it was around 2016. Around that time it was semi-maintained. Looks like it's entirely abandoned for now (last commit in 2016 and last release in 2014).

If you know more interesting alternatives, I'd love to consider those before going all in on bats.

[ahmetb]

For studying how to write tests with bats, I have the following lengthy files still used after many years:

• https://github.com/Azure/custom-script-extension-linux/blob/master/integration-test/test/handler-commands.bats
• https://github.com/Azure/custom-script-extension-linux/blob/master/integration-test/test/basic.bats

One big recommendation is to do echo "$output" after each cmd before testing anything –otherwise it'll just say test failed with no other indicator (like output) of failure. (If test doesn't fail, echo is suppressed.)

[ahmetb]

As of #113 we now have 17 tests for kubectx, which is a good place to begin. I intend to go through the code some time and see if we can add more around the "edge cases".

It's a good time to begin working on kubens tests. It can be started by copy-pasting from the existing tests and modifying them.

[doodlesbykumbi]

Happy to have a go at kubens :) Should have a PR in a couple hours

[doodlesbykumbi]

Well, here's a wall. We'd need to mock the Kubernetes API to carry out any non-trivial tests :(

[ahmetb]

Well, here's a wall. We'd need to mock the Kubernetes API to carry out any non-trivial tests :(

Saw that coming!

I think we will need to override the $KUBECTL variable from the environment, then we can write a small bash script named mock-kubectl for the parts we need.

This part will need refactoring to use the $KUBECTL if it exists in the environment.

kubectx/kubectx

Lines 176 to 183 in df557e4

	if hash kubectl 2>/dev/null; then
	KUBECTL=kubectl
	elif hash kubectl.exe 2>/dev/null; then
	KUBECTL=kubectl.exe
	else
	echo >&2 "kubectl is not installed"
	exit 1
	fi