helm / chart-testing

CLI tool for linting and testing Helm charts

Helm

Helm is a tool for managing Charts. Charts are packages of pre-configured Kubernetes resources.

Use Helm to:

• Find and use popular software packaged as Helm Charts to run in Kubernetes
• Share your own applications as Helm Charts
• Create reproducible builds of your Kubernetes applications
• Intelligently manage your Kubernetes manifest files
• Manage releases of Helm packages

Helm in a Handbasket

Helm is a tool that streamlines installing and managing Kubernetes applications. Think of it like apt/yum/homebrew for Kubernetes.

• Helm renders your templates and communicates with the Kubernetes API
• Helm runs on your laptop, CI/CD, or wherever you want it to run.
• Charts are Helm packages that contain at least two things:
  • A description of the package (Chart.yaml)
  • One or more templates, which contain Kubernetes manifest files
• Charts can be stored on disk, or fetched from remote chart repositories (like Debian or RedHat packages)

Install

Binary downloads of the Helm client can be found on the Releases page.

Unpack the helm binary and add it to your PATH and you are good to go!

If you want to use a package manager:

• Homebrew users can use brew install helm.
• Chocolatey users can use choco install kubernetes-helm.
• Scoop users can use scoop install helm.
• Snapcraft users can use snap install helm --classic

To rapidly get Helm up and running, start with the Quick Start Guide.

See the installation guide for more options, including installing pre-releases.

Docs

Get started with the Quick Start guide or plunge into the complete documentation

Roadmap

The Helm roadmap uses GitHub milestones to track the progress of the project.

Community, discussion, contribution, and support

You can reach the Helm community and developers via the following channels:

• Kubernetes Slack:
  • #helm-users
  • #helm-dev
  • #charts
• Mailing List:
  • Helm Mailing List
• Developer Call: Thursdays at 9:30-10:00 Pacific (meeting details)

Contribution

If you're interested in contributing, please refer to the Contributing Guide before submitting a pull request.

Code of conduct

Participation in the Helm community is governed by the Code of Conduct.

chart-testing's Issues

changes to readme file without raising the chart version does not fail

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Version of Helm and Kubernetes:
Helm 2.12.3
Kubernetes 1.11
CT: 2.1.0

What happened:
Linting a chart did not fail on version check when only the readme of the chart was changed.

What you expected to happen:
As "CheckVersionIncrement" was set to "true" i expect linting to fail when the readme file, which will by part of the tgz archive, is changed but the chart version is not raised.

How to reproduce it (as minimally and precisely as possible):

• just change the readme
• ct lint

Anything else we need to know:
Pipeline where we saw this: https://circleci.com/gh/kiwigrid/helm-charts/636

CheckVersionIncrement is always fasle

Is this a request for help?:

No

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Version of Helm and Kubernetes:
Helm 2.12
Kubectl 1.12
ct 2.0.1

What happened:
Were using a CircleCi pipeline (https://github.com/zammad/helm/blob/master/.circleci/config.yml) mainly based on the example from this repo.

When the chart is linted it does not fail when the chart version is not raised. See: https://circleci.com/gh/zammad/helm/150

CheckVersionIncrement is set to false even if it should be true by default.
Also the explicit setting of "check-version-increment: true" in ct.yaml is ignored.

What you expected to happen:

• linting fails if version is not raised.

How to reproduce it (as minimally and precisely as possible):
Use config from: https://github.com/zammad/helm/blob/master/.circleci

Proposal for `ct` test metrics

Produce ct test metrics file for each tested chart.

1. Main use case would be using ct with https://github.com/helm/chart-releaser.
2. Then helm repos wanted to be part of https://hub.helm.sh can be checked that charts there are tested with ct

Check for mutating matchLabels in apps/v1

matchLabels cannot change in apps/v1 (e.g., StatefulSet). That means things like the chart label cannot change when the version increments. This is no longer allowed.

A way to catch an error on the chart or other common labels would be useful.

Allow setting --wait option when installing charts

It would be nice to be able to specify --wait option when installing charts.

Perhaps it should even be enabled by default.

kubectl rollout status seems to not be failing when pods error

We discovered this issue where the test succeeded but the WordPress pod never started up and errored out, but the CI succeeded and the change got merged: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/helm_charts/7101/pull-charts-e2e/12374/?log#log

This is the PR, we should be able to reproduce it with this chart and manually running kubectl rollout status helm/charts#7101.

If kubectl rollout status is not designed to return a non-zero error when rollouts fail, we may want to grep for error and fail if we get one here: https://github.com/helm/chart-testing/blob/master/lib/chartlib.sh#L273-L305

Decide on and document supported versions

Since #35 / #29 this tool has been rewritten in Go, and the v2.y.z releases as of v2.0.0 are now built from master. However, we still plan to support users who will require more time to upgrade from v1 to v2 for a period of time.

• Decide on support status types (active, security/bug fixes only, EOL, etc), supported version granularity (probably MAJOR versions only), and timeline for each
• Document supported versions
• Document upgrading from < v2
• Ensure branches per version (assuming release-v1 branch for the old version, and master for v2 for now)
• Document process for contributing relative to versions (normally open PRs against master per usual, but for legacy support open v1 PRs against release-v1 branch, etc)

Exit gracefully (run cleanup tasks) when relevant unix signals are received

When an interrupt signal is received, for example, any releases or namespaces that are pending deletion should be deleted. This would be great for local usage of ct install when the user may wish to exit the program as soon as it's clear that a deployment has failed.

Add option to git diff between last 2 local commits

Is this a request for help?:

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
FEATURE REQUEST

Version of Helm and Kubernetes:
helm version: 2.11
kubectl version: 1.11
ct version: 2.0.1

What happened:
Please add an option to git diff between the last 2 local commits instead between local and remote repo.

I'd like to use ct on a local git repo which was cloned from a remote and therefore hase same commit locally as on the remote. Unfortunately the change detection don't works in this case and i only can use --all to get my charts tested.

What you expected to happen:
An option is available to enable change detection to use a diff to HEAD~1 instead to the remote.

How to reproduce it (as minimally and precisely as possible):

• Checkout your repo from remote to a local directory
• run ct lint locally
• "No chart changes detected."

Support for pushing charts to a helm chart repo

Would be nice to have a script allowing to push charts to a helm chart repo. I would also suggest not to include the ci folder with *-values.yaml files in the published chart. There can be some sensitive data that should not be published.

This. Is. Amazing.

In the spirit of "First Post", I wanted to thank you for creating this tool. ❤️

Permission denied when using kind example circle ci job

Is this a request for help?: ues

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug report

What happened:
I copied the setting from the kind example to my chart repository (hope I can make it public tomorrow), modify the required settings and push the changes. When running on circle ci, the job install-charts failed with the following command:

chmod: changing permissions of ‘kind’: Operation not permitted

I think this is a permission issue - kind is downloaded on golang docker image, meaning it is downloaded as root, which causes chmod to fail.
Installing kind using curl seem to solve the problem:

curl -Lo kind https://github.com/kubernetes-sigs/kind/releases/download/0.0.1/kind-linux-amd64

What you expected to happen:
Test to run successfully (or with issues related to my code)

How to reproduce it (as minimally and precisely as possible):
Run the kind example on circle ci

Anything else we need to know:
This is an amazing project, thank you so much!

Upgrade version and add option to specify stable repo for helm init

First, thanks for this repo, I was doing all checks in a Jenkins pipeline which was super ugly so now is really easy to just use the docker image.

Issue:
There is a bug with the current version of helm you are using (v2.10.0) which doesn't allow to do helm init behind proxy. So, it would nice to use latest version(v2.11.0).

Also, while looking into this issue, I'll realize there is an option, --stable-repo-url, for helm init which let you set the stable repo url. It would be even better if we can set this for chart_test.sh. By this way I don't even need to specify a proxy, since I can specify the stable repo url, which is internal and doesn't need a proxy to go to internet.

I was thinking to specify an envar like STABLE_REPO_URL. Let me know if you guys want this so I can do a PR.

maintainer validation ignores config flag

The option --validate-maintainers=false is ignored by ct lint.

Seems like this code block

chart-testing/pkg/chart/chart.go

Line 299 in f632cd5

if err := t.ValidateMaintainers(chart); err != nil {

should be surrounded with if t.config.ValidateMaintainers {}

Version: v2.0.0

Chart repo URLs with query strings cause errors

Running this with a chart repo that has a query string (e.g. a private blob with SAS token) causes an error. This appears to be because the name of the chart is being taken as all the characters before the last = instead of the first.

.mytestenv file:

CHART_REPOS=( "myrepo=https://myblob.blob.core.windows.net/helm/?se=2020-01-01&sp=rl&sv=2018-01-01&sr=c&sig=redacted-token" )

command:

docker run --rm -v "$(pwd):/workdir" --workdir /workdir quay.io/helmpack/chart-testing:v1.1.0 chart_test.sh --no-install --config .mytestenv

result:

--------------------------------------------------------------------------------
 Environment:
 REMOTE=origin
 TARGET_BRANCH=master
 CHART_DIRS=charts
 EXCLUDED_CHARTS=
 CHART_REPOS=myrepo=https://myblob.blob.core.windows.net/helm/?se=2020-01-01&sp=rl&sv=2018-01-01&sr=c&sig=redacted-token
 TIMEOUT=300
 LINT_CONF=/testing/etc/lintconf.yaml
 CHART_YAML_SCHEMA=/testing/etc/chart_schema.yaml
 VALIDATE_MAINTAINERS=false
 GITHUB_INSTANCE=https://github.com
 CHECK_VERSION_INCREMENT=true
--------------------------------------------------------------------------------

Charts to be installed and tested: charts/my-chart
Initializing Helm client...
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Not installing Tiller due to 'client-only' flag having been set
Happy Helming!
Error: Looks like "https://myblob.blob.core.windows.net/helm/?se=2020-01-01&sp=rl&sv=2018-01-01&sr=c&sig=redacted-token" is not a valid chart repository or cannot be reached: open /root/.helm/repository/cache/myrepo=https:/myblob.blob.core.windows.net/helm/?se=2020-01-01&sp=rl&sv=2018-01-01&sr=c&sig-index.yaml: no such file or directory

Not detecting changes when changing template files

Is this a request for help?:

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG

Version of Helm and Kubernetes:

helm

Client: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"}

kubectl

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T19:44:19Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.5-eks-6bad6d", GitCommit:"6bad6d9c768dc0864dab48a11653aa53b5a47043", GitTreeState:"clean", BuildDate:"2018-12-06T23:13:14Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

ct

Version:	 v2.1.0
Git commit:	 2da25914d061bf4b4b1fc77624cc223dac1ba67c
Date:		 2019-01-07T21:48:19Z
License:	 Apache 2.0

What happened:

λ ct lint --debug
Linting charts...
Using config file:  /app/ct.yaml
------------------------------------------------------------------------------------------------------------------------
 Configuration
------------------------------------------------------------------------------------------------------------------------
Remote: origin
TargetBranch: master
BuildId:
LintConf: lintconf.yaml
ChartYamlSchema: chart_schema.yaml
ValidateMaintainers: true
ValidateChartSchema: true
ValidateYaml: true
CheckVersionIncrement: true
ProcessAllCharts: false
Charts: []
ChartRepos: []
ChartDirs: [stable infrastructure]
ExcludedCharts: []
HelmExtraArgs: --timeout 800
HelmRepoExtraArgs: []
Debug: true
Namespace:
ReleaseLabel:
------------------------------------------------------------------------------------------------------------------------
>>> git merge-base origin/master HEAD
>>> git diff --find-renames --name-only b00fdb7fcc0dbd813517b6523cdecb0a6c01d68b -- stable infrastructure
All charts linted successfully
------------------------------------------------------------------------------------------------------------------------
No chart changes detected.
------------------------------------------------------------------------------------------------------------------------

Running the git command ct uses I do however get changed files listed.

λ git diff --find-renames --name-only b00fdb7fcc0dbd813517b6523cdecb0a6c01d68b -- stable infrastructure
stable/notifications/templates/service.yaml

What you expected to happen:

 Charts to be processed:
------------------------------------------------------------------------------------------------------------------------
 stable/notifications
------------------------------------------------------------------------------------------------------------------------
...
please bump version 🗡 

How to reproduce it (as minimally and precisely as possible):

1. Change something in a charts template files
2. Run ct lint
3. Get sad that lint is not detecting change

Anything else we need to know:
Works with ct 2.0.0 and 2.0.1, also detects changes in Chart.yaml och values.yaml (ie not nested folder?)

Is there any plan for helm v3?

Is this a request for help?: No.

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST.

Anything else we need to know:
I know it might be a bit early to talk about helm v3 support but since helm v3's api changes a lot which makes chart-testing fail in several ways, it would be great to know if there is a plan for helm v3 support. Thanks!

Link to the docker image in Prerequisites is wrong

Minor issue

Link to the docker image in Prerequisites is wrong, it looks like it is missing the http:// part

It is recommended to use the provided Docker image which can be [found on Quay](quay.io/helmpack/chart-testing/).
It comes with all necessary tools installed.

How to pass bash array env vars

Note that CHART_DIRS, EXCLUDED_CHARTS, and CHART_REPOS must be configured as Bash arrays.

I cannot figure out, how to make a bash array env var accessible by the script. From what I read here for example, this doesn't seem to be possible. This little example supports this assumption:

bash-4.4# export CHART_REPOS=("a=http://foo:8080" "b=http://bar.com:8080")
bash-4.4# echo "${CHART_REPOS[*]}"
a=http://foo:8080 b=http://bar.com:8080
bash-4.4# bash -c 'echo "${CHART_REPOS[*]}"'

bash-4.4#

Remove implicit dependence on extra tools/repos

Hey all 👋

First of all, thanks very much for this project. We use the old bash version of this in cert-manager and it works well 😄

The recent Go rewrite is great, and I'm keen to get this setup in our own environment again.

I wonder if there is any desire to remove the dependence on yamale/yamllint?

We're hoping to be able to run this without the need for Docker, so for the time being will need to configure our own build/test environment to grab the appropriate version of these tools.

Can these projects either be included in the Go binary release somehow, or otherwise can their functionality be absorbed into Go?

Proposal: update docker image on quay.io

We have helm 2.13.2 in Dockerfile and 2.12.0 in the latest docker image in quay.io. Can you please rebuild image and upload it to quay.io ?

Does not work with CloudBuild github app trigger on PR

If this is a FEATURE REQUEST, please:
Help make CT work on CloudBuild.

What happened:
I do not use CircleCI atm, jsut cloudbuild, unfortunately CT does not detect that it is in a git repository in cloudbuild and fails.

What you expected to happen:
Did not know what to expect.

How to reproduce it (as minimally and precisely as possible):
Try using CT with Cloudbuild on PR

Anything else we need to know:
CloudBuild github App trigger is in Alpha.
It seems like in CloudBuild .git if not copied over to the build env.
If possible, can CT work without detecting that it is in a git repository?

Automate tests to upgrade individual charts

Community charts were hit with a set of unexpected upgrade bugs. We can speculate about why that didn't come to our attention before, but this also could have been caught (and will catch future issues) by adding automated upgrade tests.

See:

• helm/charts#7578
• helm/charts#7680
• helm/charts#7726

Add .github/ directory with templates for PR/Issues

I believe we should copy helm/charts:.github/ directory and its templates to helm/chart-testing and adjust them to the project needs. Would help in maintainability, though not much is really happening in the repo right now.

https://github.com/helm/charts/tree/master/.github

index out of range if there are files in the root of the repository

BUG REPORT

If the root of a repository is used as the chart dir (.) and there are non-empty directories or files, an index out of range will be raised.

Attaching a PR that reproduces and fixes the issue.

Validate maintainers in GitLab runners

BUG REPORT

Version of Helm and Kubernetes:
Helm v2.12.3
Kubernetes v1.11.7

What happened:

For validating the maintainers it sends a HEAD request to the URL built like: repoUrl/maintainerName.

repoUrl is obtained through the following command:

git ls-remote --get-url origin

In most of the cases, if you run it the output for GitLab would be something like:

https://gitlab.com/my-nice-group/MyNiceProject.git

Which matches perfectly with the regex expression defined in https://github.com/helm/chart-testing/blob/master/pkg/tool/account.go#L27

But if you run that command in a GitLab runner, the output would be:

https://gitlab-ci-token:[MASKED]@gitlab.com/my-nice-group/MyNiceProject.git

So the validation will always fail with the following error:

Error validating maintainers: Head https://gitlab-ci-token/garcal01: dial tcp: lookup gitlab-ci-token on 192.168.0.10:53: server misbehaving

What you expected to happen:
regex expression also matches GitLab runner output

How to reproduce it (as minimally and precisely as possible):
Use helm/chart-testing tool in a GitLab runner validating maintainers

Add option to hide pod logs/resource descriptions unless release tests fail

When testing even a single chart with multiple ci values files, the log output that ctproduces is pretty large. When release tests fail, it would be much easier to figure out the cause if the log output from all the other successful releases could be filtered out.

Proposal: helm 2.13 instead of 2.12 in docker image

What happened:
helm in the ct docker image knows nothing about --atomic option because it was added in the 2.13.0 helm release

What you expected to happen:
latest helm version in the ct docker image

How to reproduce it (as minimally and precisely as possible):

docker run -ti --rm quay.io/helmpack/chart-testing:v2.3.0 sh
helm upgrade release chart --install --wait --atomic --timeout=300

Error: unknown flag: --atomic

--helm-extra-args are not passed through to helm init

Is this a request for help?:
No.

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug report.

Version of Helm and Kubernetes:
helm version:

Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}

kubectl version:

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

ct version:

Version:	 v2.3.3
Git commit:	 e3c32f5aea280676e8dc020ac1fac66bf31c7986
Date:		 2019-04-25T17:30:04Z
License:	 Apache 2.0

What happened:
When the --helm-extra-args flag is set, the value is not passed through to the exec of helm init.

What you expected to happen:
I would expect that --helm-extra-args is also passed through to helm init.

How to reproduce it (as minimally and precisely as possible):
My ct.yaml:

debug: true
remote: origin
target-branch: master
chart-dirs:
  - charts
helm-extra-args: --stable-repo-url https://my-internal-helm-mirror/stable/

Output when running ct lint --config test/ct.yaml --chart-yaml-schema test/chart_schema.yaml --lint-conf test/lintconf.yaml:

Linting charts...
Using config file:  test/ct.yaml
------------------------------------------------------------------------------------------------------------------------
 Configuration
------------------------------------------------------------------------------------------------------------------------
Remote: origin
TargetBranch: master
BuildId: 
LintConf: test/lintconf.yaml
ChartYamlSchema: test/chart_schema.yaml
ValidateMaintainers: true
ValidateChartSchema: true
ValidateYaml: true
CheckVersionIncrement: true
ProcessAllCharts: false
Charts: []
ChartRepos: []
ChartDirs: [charts]
ExcludedCharts: []
HelmExtraArgs: --stable-repo-url https://my-internal-helm-mirror/stable/
HelmRepoExtraArgs: []
Debug: true
Upgrade: false
SkipMissingValues: false
Namespace: 
ReleaseLabel: 
------------------------------------------------------------------------------------------------------------------------
>>> git rev-parse --is-inside-work-tree
>>> git merge-base origin/master HEAD
>>> git diff --find-renames --name-only 274897e87a4b68277f3ac6d41ad5c407f54215f8 -- charts

------------------------------------------------------------------------------------------------------------------------
 Charts to be processed:
------------------------------------------------------------------------------------------------------------------------
 common => (version: "0.0.5", path: "charts/common")
------------------------------------------------------------------------------------------------------------------------

>>> helm init --client-only
Creating /root/.helm 
Creating /root/.helm/repository 
Creating /root/.helm/repository/cache 
Creating /root/.helm/repository/local 
Creating /root/.helm/plugins 
Creating /root/.helm/starters 
Creating /root/.helm/cache/archive 
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Error: Looks like "https://kubernetes-charts.storage.googleapis.com" is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp 172.217.22.48:443: connect: connection timed out
Error linting charts: Error initializing Helm: Error waiting for process: exit status 1
------------------------------------------------------------------------------------------------------------------------
No chart changes detected.
------------------------------------------------------------------------------------------------------------------------
Exited with code 1

Anything else we need to know:
Related to #33

Check for mutating labels in StatefulSet spec.VolumeClaimTemplate.metadata.labels

See helm/charts#7803

Related #19, #99

Publish a pre-release docker image from master

This would make end user testing a bit easier before cutting new official releases.

Chart linting git-crypt issue

Hi!

We currently use git-crypt for storing creds.
How can we prevent ecnrypted files from validating?

Olegs-MacBook-Pro:k8s mukolaich$ docker run --rm -v "$(pwd):/workdir" --workdir /workdir gcr.io/kubernetes-charts-ci/chart-testing:v1.0.2 chart_test.sh --no-install --config .conf

--------------------------------------------------------------------------------
 Environment:
 REMOTE=origin
 TARGET_BRANCH=SRE100-500k8s_for_sat
 CHART_DIRS=sat/
 EXCLUDED_CHARTS=sat/s3sync/
 CHART_REPOS=
 TIMEOUT=300
 LINT_CONF=/testing/etc/lintconf.yaml
 CHART_YAML_SCHEMA=/testing/etc/chart_schema.yaml
 VALIDATE_MAINTAINERS=true
--------------------------------------------------------------------------------

"git-crypt" clean: line 1: git-crypt: not found
error: external filter '"git-crypt" clean' failed 127
error: external filter '"git-crypt" clean' failed
fatal: sat/blah/blah/blah/blah.yaml: clean filter 'git-crypt' failed

.conf

CHART_DIRS=sat/charts/blah-nginx
EXCLUDED_CHARTS=sat/s3sync/
TARGET_BRANCH=SRE100-500k8s_for_sat

Support for validating users on Gitlab

It would be nice to validate user on Gitlab also. I love this feature, but for people using Gitlab, this feature might be limiting.

I can work on a PR for this, if you think this is a good idea.

Add flag to disable upgrade tests for renamed or removed values files

Right now switching on the --upgrade flag can lead to cases where it's not possible to get ct to succeed, based on the contents of a previous revision.

The upgrade flag uses the previous revision's ci values files instead of the current revision's values files, and thus if the previous revision has a ci values file that does not support upgrades (example), it becomes impossible to get ct to pass unless you bump the version in Chart.yaml a major version.

One approach to this problem is to add a flag that will tell ct not to pay attention to values files from the previous revision that have been removed or renamed in the current revision. This will allow simply deleting bad values files at the current revision to get builds passing.

A downside to this would be that some legitimate failure modes could be missed during renames or that folks might be encouraged to simply delete test files when there could be a good reason they're failing.

chart-testing docker image on hub.docker.com

Is this a request for help?:
As https://quay.io/ is down for the last 2 days (http://status.quay.io/) an additional build of the chart-testing docker image on Dockerhub would be nice...

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
FEATURE REQUEST

Version of Helm and Kubernetes:
2.11 / 1.11

What happened:
Download of docker image from https://quay.io failed

What you expected to happen:
Docker image Download works

How to reproduce it (as minimally and precisely as possible):
Download the image

Anything else we need to know:

Latest Docker image is broken

Running latest Docker image produces an error:

-> docker run -v $(pwd)/charts:/charts quay.io/helmpack/chart-testing:latest /usr/local/bin/ct lint --all --chart-dirs /charts --debug --validate-maintainers=false
docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"/usr/local/bin/ct\": permission denied": unknown.
ERRO[0001] error waiting for container: context canceled 

While the same works fine on v.2.2.0 tag:

docker run -v $(pwd)/charts:/charts quay.io/helmpack/chart-testing:v2.2.0 /usr/local/bin/ct lint --all --chart-dirs /charts --debug --validate-maintainers=false

Feature: add force for linting/install on unchanged charts

in some testing cases an option to force linting/install on unchanged charts would be very useful.

Add ct option to install charts with multiple values file one at a time

Is this a request for help?:
No.

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

While installing charts with values files, all the charts are installed at the same time in different namespaces. This results in a lot of resource usage. When there's not enough resource available, some chart deployments fail. It would be great to have an option to install the charts with values files one at a time.

In https://github.com/helm/chart-testing/blob/v2.1.0/pkg/chart/chart.go#L345-L372, all the charts are installed with values together and cleaned up at the very end. An option to install with values, wait for deployments and delete release before installing with the next values file would help when there's a limited resource.

Would be great to know if there's a workaround for this already. If not, I can submit a PR adding an option to add the above behavior.

Proposal: Enable full set of checks for charts not in a parent directory

The Problem

Right now the --charts flag disables chart version validation and change detection. From the usage text:

Specific charts to test. Disables changed charts detection and
version increment checking. May be specified multiple times
or separate values with commas

This flag is also currently the only way to test charts that are not part of a chart "parent directory", which is a known use case (see #101 or goharbor/harbor-helm).

Proposal

1. Add a new flag for testing specific charts that does include change detection

or

2. Make a breaking change to not disable change detection when --charts is used, but support
   the old behavior with --charts ./somechart --all.

Also remove the default value for --chart-dirs, since there doesn't seem to be a great way to override string slice args to be empty (see #104 (comment)).

Support go get for installation

FEATURE REQUEST

Downloading released binaries is pretty easy. Still, for those who already have the go toolchain installed go get would be a nice and familiar way to quickly get started with ct.

Proposal: Re-implement in Go

I'm tinkering with re-implementing chart-testing in Go. I've already thought about this when I wrote it in Bash and, looking at it in hindsight, I think I should have done it right from the start. I probably didn't expect we'd end up with a script containing more than 500 lines of code. Too much for a Bash script IMHO. On the other hand, Python is excellent for scripting, you can debug it, and IDE support is very good. It's easier to write and maintain than Bash.

The project is getting more popular with more PRs and issues coming in. However, adding new features to a Bash script can get messy. There are too many different styles and it's too easy to make mistakes, especially when you don't have a debugger.

Configuration is currently not ideal. We have a mixture of CLI args and environment variables. Config and args parsing would be much easier in Python. Without any special effort, we could support CLI args, env vars, and config files.

Overall, we'd get a better understandable and maintainable code and probably lower the threshold for contributions.

Opinions?

cc @helm/charts-maintainers

---

Update: The re-implementation will happen in Go!

Support custom chart processors via exec-based plugin system

As a way to support custom linting and other chart validation use cases, I propose that we allow users to specify custom chart processors that do not require compilation into ct.

The implementation might look something like this:

type ChartProcessor interface {
	ProcessChart(*Chart) TestResult
}

type ExecManifestProcessor struct {
	Command []string
}

// ProcessChart renders a chart (once for each values file) and executes Command
// passing the rendered manifests as input via Stdin.
//
// Runtime complexity is O(n) where n is the number of CI values files.
func (e ExecManifestProcessor) ProcessChart(chart *Chart) TestResult {
	...
}

Users would then be able to specify custom processors, eg. an OPA based linter, a shell script, etc.

There are two sorts of plugins which could be supported:

1. Those which can accept a set of pre-rendered yaml manifests via Stdin
   • kubectl apply --server-dry-run
   • kubeval
   • conftest
2. Those which accept a helm chart path as input
   • helm-opa plugin
   • helm lint (this is already natively supported by ct but would easily be refactored as a plugin)

The ct configuration should distinguish between these two types, and more may be added in the future (eg. processors which accept JSON manifests as input).

Other Considerations

1. We could just build support for new processors directly into ct. This would be a better experience for end users, but is not scalable. Custom processors may still "graduate" to directly supported default plugins like helm lint.
2. We could explicitly support using ct as a go library and allow others to build their own CLIs that integrate custom processors. This might lead to fragmentation, and isn't a satisfactory solution for those who have simple needs.

Check for mutating metadata.labels in v1beta1

See helm/charts#7726

Related: #19

panic error when exeu ct lint-and-install

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
I use chart-testing v2.2.0 docker： quay.io/helmpack/chart-testing:v2.2.0 to test ct lint and install:
docker exec --interactive ctd "$@" ct lint-and-install --charts stable/mysql,stable/postgresql,stable/etcd-operator,stable/coredns --namespace kube-system

then I get the error:

$HELM_HOME has been configured at /root/.helm.
Not installing Tiller due to 'client-only' flag having been set
Happy Helming!
"incubator" has been added to your repositories
No requirements found in stable/mysql/charts.
Linting chart 'stable/mysql'
Validating /workdir/stable/mysql/Chart.yaml...
Validation success! 👍
Validating maintainers...
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/helm/chart-testing/pkg/tool.parseOutGitRepoDomain(0xc0000142dc, 0x3, 0xc00004e400, 0xc000010330)
/go/src/github.com/helm/chart-testing/pkg/tool/account.go:44 +0x6f
github.com/helm/chart-testing/pkg/tool.AccountValidator.Validate(0xc0000142dc, 0x3, 0xc000014170, 0x9, 0x0, 0x0)
/go/src/github.com/helm/chart-testing/pkg/tool/account.go:29 +0x4d
github.com/helm/chart-testing/pkg/chart.(*Testing).ValidateMaintainers(0xc000084180, 0xc0000da4c0, 0xc, 0xc00007f860, 0x15)
/go/src/github.com/helm/chart-testing/pkg/chart/chart.go:565 +0x202
github.com/helm/chart-testing/pkg/chart.(*Testing).LintChart(0xc000084180, 0xc0000da4c0, 0xc, 0x0, 0x0, 0x0, 0x3, 0x0, 0x0, 0xc000178720)
/go/src/github.com/helm/chart-testing/pkg/chart/chart.go:312 +0x534
github.com/helm/chart-testing/pkg/chart.(*Testing).LintAndInstallChart(0xc000084180, 0xc0000da4c0, 0xc, 0x0, 0x0, 0x0, 0xc0000da4c0, 0xc, 0x0, 0x0)
/go/src/github.com/helm/chart-testing/pkg/chart/chart.go:389 +0x67
github.com/helm/chart-testing/pkg/chart.(*Testing).LintAndInstallChart-fm(0xc0000da4c0, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/helm/chart-testing/pkg/chart/chart.go:257 +0x63
github.com/helm/chart-testing/pkg/chart.(*Testing).processCharts(0xc000084180, 0xc0000e7828, 0xc000097832, 0x922d80, 0xc3c4e0, 0x922da0, 0xc3c4e0)
/go/src/github.com/helm/chart-testing/pkg/chart/chart.go:232 +0x6df
github.com/helm/chart-testing/pkg/chart.(*Testing).LintAndInstallCharts(0xc000084180, 0x3, 0xc000097298, 0x6, 0x0, 0x0)
/go/src/github.com/helm/chart-testing/pkg/chart/chart.go:257 +0x4e
github.com/helm/chart-testing/app/cmd.lintAndInstall(0xc00014d180, 0xc0000aae00, 0x0, 0x4)
/go/src/github.com/helm/chart-testing/app/cmd/lintAndInstall.go:53 +0x20c
github.com/helm/chart-testing/vendor/github.com/spf13/cobra.(*Command).execute(0xc00014d180, 0xc0000aadc0, 0x4, 0x4, 0xc00014d180, 0xc0000aadc0)
/go/src/github.com/helm/chart-testing/vendor/github.com/spf13/cobra/command.go:766 +0x2cc
github.com/helm/chart-testing/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00014ca00, 0xc000098120, 0xc00014d680, 0xc00014d400)
/go/src/github.com/helm/chart-testing/vendor/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/helm/chart-testing/vendor/github.com/spf13/cobra.(*Command).Execute(0xc00014ca00, 0x405690, 0xc000076058)
/go/src/github.com/helm/chart-testing/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/helm/chart-testing/app/cmd.Execute()
/go/src/github.com/helm/chart-testing/app/cmd/root.go:57 +0x27
main.main()
/go/src/github.com/helm/chart-testing/app/main.go:22 +0x20

• docker container rm --force ctd

Version of Helm and Kubernetes:

helm version: v2.12.2 (including in chart-testing image)
kubectl version: v1.14.0 and v1.12.7, both failed with the above error msg
ct version: v2.2.0

Possible switch/config option to allow helm lint with "--strict"

There should be a switch or a config option that allows for the lint to use helm lint --strict to ensure a failure if there are missing values.

https://github.com/helm/chart-testing/blob/master/pkg/tool/helm.go#L53

Perhaps even just changing the line to

return h.exec.RunProcess("helm", "lint", "---strict", chart, values)

Support chart installation to an existing namespace

I am working with an organization that runs a dedicated tiller instance for each set of namespaces owned by a given business unit. None of these tiller instances have cluster-admin permissions, nor can they create/delete arbitrary namespaces. Teams may also maintain their own chart repositories.

Specifying an existing namespace for chart installation would allow teams to consume this library in a more distributed and self-service manner, without the need for a dedicated cluster or access to a more highly privileged tiller instance.

TODO: Push releases to registries

We should push releases to docker hub, quay, and gcr (closer to our testing setup)

Unable to use "ct install" command

Is this a request for help?: No

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes:
kubectl:
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:25:46Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

helm
Client: &version.Version{SemVer:"v2.12.2", GitCommit:"7d2b0c73d734f6586ed222a567c5d103fed435be", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

ct:
Version: v2.2.0
Git commit: 00f3726
Date: 2019-01-23T14:30:52Z
License: Apache 2.0

What happened:
I was trying to use official docker container which has a tag: v2.2.0 with own made chart.
During install phase ct generates random release name which is not valid for helm

Error message:

Installing charts...
Version increment checking disabled.

Configuration

Remote: upstream
TargetBranch: master
BuildId:
LintConf: /etc/ct/lintconf.yaml
ChartYamlSchema: chart_schema.yaml
ValidateMaintainers: false
ValidateChartSchema: false
ValidateYaml: false
CheckVersionIncrement: false
ProcessAllCharts: false
Charts: [./]
ChartRepos: []
ChartDirs: [stable incubator]
ExcludedCharts: []
HelmExtraArgs: --timeout 500 --tiller-namespace kube-system --dry-run --debug
HelmRepoExtraArgs: []
Debug: true
Namespace: kube-system
ReleaseLabel: test

---

Charts to be processed:

./

helm init --client-only
$HELM_HOME has been configured at /root/.helm.
Not installing Tiller due to 'client-only' flag having been set
Happy Helming!
helm dependency build ./
No requirements found in .//charts.
Installing chart './'...
helm install ./ --name .-2cnt7c1osa --namespace kube-system --wait --timeout 500 --tiller-namespace kube-system --dry-run --debug
[debug] Created tunnel using local port: '39759'

[debug] SERVER: "127.0.0.1:39759"

[debug] Original chart version: ""
[debug] CHART PATH: /service-monitors

Error: release name .-2cnt7c1osa is not a valid DNS label: a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for validation is 'a-z0-9?')
kubectl get pods --no-headers --namespace kube-system --selector test=.-2cnt7c1osa --output jsonpath={.items[*].metadata.name}
Error printing logs: Error running process: exit status 1
Deleting release '.-2cnt7c1osa'...
helm delete --purge .-2cnt7c1osa --timeout 500 --tiller-namespace kube-system --dry-run --debug
[debug] Created tunnel using local port: '45833'

[debug] SERVER: "127.0.0.1:45833"

Error: incompatible versions client[v2.12.2] server[v2.11.0]
Error deleting Helm release: Error waiting for process: exit status 1
Error installing charts: Error processing charts

✖︎ ./ > Error waiting for process: exit status 1

What you expected to happen:
Perform lint and install operations without errors

How to reproduce it (as minimally and precisely as possible):

1. Download docker image: quay.io/helmpack/chart-testing:v2.2.0
2. Create simple local chart
3. Try to run next command inside the container in directory with your chart:
   ct install --debug --charts ./ --release-label=test --helm-extra-args=" --timeout 500 --tiller-namespace kube-system --dry-run --debug" --namespace kube-system

Anything else we need to know:

Decide on / request Slack channel for chart-testing

How to test charts in a local directory?

Is this a request for help?:

Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
No

Version of Helm and Kubernetes:

Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:55Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.8-eks-7c34c0", GitCommit:"7c34c0d2f2d0f11f397d55a46945193a0e22d8f3", GitTreeState:"clean", BuildDate:"2019-03-01T22:49:39Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Version:	 unreleased
Git commit:	 unknown
Date:		 unknown
License:	 Apache 2.0

What happened:

$ ct lint-and-install   --chart-dirs .
Linting and installing charts...
Error loading configuration: 'chart_schema.yaml' neither specified nor found in default locations

What you expected to happen:

It should lint charts in the local directory. I have no idea how the chart_schema.yaml should look like.

How to reproduce it (as minimally and precisely as possible):

1. Create a dir.
2. Create a chart inside the dir.
3. Run ct lint --chart-dirs .

Thank you!

Update Same for lintconf.yaml. I believe the program should not require any specific configs to run it locally but just use some predefined settings.