replicatedhq / kots

KOTS provides the framework, tools and integrations that enable the delivery and management of 3rd-party Kubernetes applications, a.k.a. Kubernetes Off-The-Shelf (KOTS) Software.

Kubernetes Off-The-Shelf (KOTS) Software

Replicated KOTS is the collective set of tools that enable the distribution and management of Kubernetes Off-The-Shelf (KOTS) software. The Kots CLI (a Kubectl plugin) is a general purpose, client-side binary for configuring and building dynamic Kubernetes manifests. The Kots CLI also serves as the bootstrapper for the in-cluster Kubernetes application Admin Console kotsadm which can be used to automate the core Kots CLI tasks for managing applications (license verification, configuration, updates, image renaming, version controlling changes, and deployment) as well as additional KOTS tasks (running preflight checks and performing support bundle analysis).

Distributing a KOTS application

Software vendors can package their Kubernetes applications or Helm charts or Operators as a KOTS application in order to distribute the application to cluster operators.

Kots CLI Documentation

Check out the full docs on the cluster operator experience for using the Kots CLI as a Kubectl plugin.

Try Kots

Try Kots as a cluster operator by installing the Replicated sample app (Sentry Pro Example) into an existing Kubernetes cluster. First, install the Kots CLI (a Kubectl plugin) on your workstation:

curl https://kots.io/install | bash

Run kots install

The install command is the recommended way to learn KOTS. Executing the install command will install an application and the kotsadm Admin Console to an existing Kubernetes cluster. This command supports installing Helm charts (without Tiller), standard Kubernetes applications and also Replicated KOTS apps.

Continue with the demo by running the following command:

kubectl kots install sentry-pro

Set a namespace for the admin console and the application components to be installed, and provide a password for the admin console. After this command completes, the kotsadm Admin Console will be running in your cluster, listening on port :8800 on a ClusterIP service in the namespace you deployed the application to. By default this is exposed to your workstation using kubectl port-forward, but you could set up an ingress/load balancer of your own.

NOTE Currently, the kotsadm pod can only be scheduled on nodes with the linux/amd64 platform.

Access the Admin Console

Visit http://localhost:8800 to access the Admin Console, enter the password.

Download the sample license for Sentry Pro & upload it to the console. You'll then be presented with configuration settings, preflight checks and other application options.

If you terminate your terminal session, the port-forward will also terminate. To access the admin console again, just run:

kubectl kots admin-console --namespace sentry-pro

Supportability

Currently, the KOTS CLI supports OSX (including Apple Silicon arm64) and Linux platforms. However, the Kubernetes resources that it creates can only be scheduled on nodes with the linux/amd64 platform.

Community

For questions about using KOTS, there's a Replicated Community forum, and a #kots channel in Kubernetes Slack.

Notifications

By default, KOTS will leverage MinIO as a standalone object store instance to store application archives and support bundles. All communication between KOTS and the MinIO object store is limited to a REST API released under the Apache 2.0 license. KOTS has not modified the MinIO source code. Use of MinIO is currently governed by the GNU AGPLv3 license that can be found in their LICENSE file. To remove MinIO usage for this use case in an existing cluster, an optional install flag --with-minio=false is available for new KOTS installs or upgrades from existing versions. To remove MinIO usage for this use case in an embedded cluster, the disableS3 option is available in the KOTS add-on and can be used for new installs or upgrades.

Software Bill of Materials

Signed SBOMs for KOTS Go dependencies and are included in each release. Use Cosign to validate the signature by running the following command.

cosign verify-blob --key sbom/key.pub --signature sbom/kots-sbom.tgz.sig sbom/kots-sbom.tgz

Development

Github Codespaces

1. Create your own codespace.
2. Clone the KOTS repo:

   git clone [email protected]:replicatedhq/kots.git

3. From the root directory, run:

   make cache
   skaffold dev

4. Visit the Admin Console URL. For VS Code:

kots's Issues

Remove PG as a requirement

One of the primary design goals of kotsadm is to keep the footprint (security, memory, cpu, images) as small as possible. We want to ensure that that Admin Console is always smaller than the application footprint.

Currently postgres is hard to make HA and also hard to manage. Since our use of PG is small, what easier to manage options can we replace it with that is HA and has a smaller footprint?

Panic on `kots install` and `kots admin-console` if ports are in use

If you run kots install or kots admin-console while another such instance is running, kots exits with the following:

panic: Unable to listen on any of the requested ports: [{8800 3000}]

goroutine 12 [running]:
github.com/replicatedhq/kots/pkg/k8sutil.PortForward.func2(0xc0000f2840, 0xc00033a7d0, 0xc00025c7b0, 0xc00036bbb8)
	/go/src/github.com/replicatedhq/kots/pkg/k8sutil/portforward.go:63 +0x82
created by github.com/replicatedhq/kots/pkg/k8sutil.PortForward
	/go/src/github.com/replicatedhq/kots/pkg/k8sutil/portforward.go:61 +0x707

We could handle this better.

kots admin-console command hangs when incorrect namespace is specified

$ k kots admin-console asdasdasd

I'd expected this to return an error that application "asdasdasd" was not found. But currently it just hangs.

KOTS install should support brining an external object store

Right now we install minio or (in the case of a kURL cluster) plug into an embedded object store -- it should be possible during kots install to pass a flag to bring an external object store instance. Unclear how we configure auth for this, and I wonder if we'd like to make this configurable in kotsadm or not.

When pushing images to remote registries, don't push layers that already exist

It appears that when KOTS is pushing an image to a remote registry (using containers/image) even if the layer exists, the bytes are copied to the registry again. This doesn't look intentional, and we should disable this to make use of caching and speed updates.

HelmChart spec, exclude field should be a BoolString, not a string

It would allow more flexibility in templating this field:

https://github.com/replicatedhq/kots/blob/master/kotskinds/apis/kots/v1beta1/helmchart_types.go#L240

Should kotsadm periodically check for updates in the background?

Right now to fetch a new version you need to "check for updates" under the version history tab or with the kots CLI.

Is there any intention to have kotsadm check for updates on a regular cadence? Perhaps one that is configurable/disable-able by the user?

Among other benefits, this seems fairly essential for any sort of gitops workflow to be valuable and eliminate manual actions that aren't merging PRs.

Resetting the admin console password should apply without a restart required

StatusInformers do not work across Namespaces

I've implemented StatusInformers for the multi-namespace app listed in replicatedhq/kotsadm#1352, but they do not appear to detect when the nginx pod in the hardcoded namespace becomes ready.

All kots commands should attempt to validate the kots version

If running kots install, and the version of kots is outdated, we should attempt to check for new versions and prompt if the user wants to update before running the command.

Add basic support for kustomize workflows.

I use kustomization as a final step before creating a release. I would like to push this step out of the upstream and into the base, to take advantage of tools available in kustomize.

For example, I cannot use a templated secret generator currently.

Ideally, this would be done with a kots.io/v1beta1 Kustomize resource, but a stopgap measure would be to detect kustomization.yaml files, include that directory as a resource and copy the content of that directory into the upstream.

Proposal: Add `requireMinimalRBACPrivileges` to the application kind

Currently (in KOTS 1.13.x and earlier). running kots install will attempt to create a cluster role and cluster role binding with rather elevated access. If this fails because the user doesn't have permissions, it will fall back to creating a role and role binding that are scoped to the target namespace.

Instead, we should attempt to create the least privileged role possible to install. The 1.14.x release has the multi-namespace support for an app already included, so there's a path to defining these. But existing applications will not be compatible with this.

If we make this change now (in 1.14.x) there are likely application that depend on the existing behavior. So we need to be cautious about the change and take a different approach.

Proposal: in the kots.io/v1beta1, Kind=Application spec, we should add a new boolean (optional) named: requireMinimalRBACPrivileges. This will default to false when missing (as bools do) so existing applications will continue to create cluster roles with elevated permissions by default.

Where this changes: Instead of creating a cluster role / role binding and falling back to a namespace scoped role, we should fail if cluster role is not allowed and this flag is set to false. Currently, in this case, KOTS is not creating a predictable installation experience and an application vendor might see everything working ok in their environment to be surprised when something fails in a more restricted cluster setup.

Eventually, we should change the default to turn this to true for a new application template. This will encourage the best behavior, but allow an escape, when needed.

KOTS CLI Windows Support?

Hi -- I've heard some rumours that things like kubectl kots install will work on a Windows workstation when downloading the CLI with krew, but its unclear what the official state of Windows support in the kots CLI is. This can be a parent issue to aggregate questions and kick off the discussion, but I imagine we'll spin off individual issues to track different bits of functionality. To start, I'm curious about the state of:

• kubectl kots install
• kubectl kots pull
• kubectl kots admin-console

Generated kustomization.yaml should be sorted

Currently the order of files is not deterministic. This creates extra noise when using a workflow such as GitOps where someone looks at and reviews diffs.

in-place upgrades of kotsadm creates a migration crashloopbackoff

This doesn't seem to cause any problems, but noticed when upgrading from 1.3.0 to 1.4.0.

kubectl logs pod/kotsadm-migrations
Executing query "drop index if exists \"idx_cluster_token\""
Error: pq: cannot drop index idx_cluster_token because constraint idx_cluster_token on table cluster requires it
Usage:
  schemahero apply [flags]
Flags:
      --driver string      name of the database driver to use
  -h, --help               help for apply
      --spec-file string   filename or directory name containing the spec(s) to apply
      --uri string         connection string uri to use
pq: cannot drop index idx_cluster_token because constraint idx_cluster_token on table cluster requires it
Executing query "drop index if exists \"idx_cluster_token\""
Error: pq: cannot drop index idx_cluster_token because constraint idx_cluster_token on table cluster requires it
Usage:
  schemahero apply [flags]
Flags:
      --driver string      name of the database driver to use
  -h, --help               help for apply
      --spec-file string   filename or directory name containing the spec(s) to apply
      --uri string         connection string uri to use
pq: cannot drop index idx_cluster_token because constraint idx_cluster_token on table cluster requires it

KOTS CLI: admin-console command: Allow specifying port if 8800 is taken, or using --port 0 to use any unused port

right now, if you do a cluster install you need to have port 8800 open on your workstation to fire up the port-forward

Enter the namespace to deploy to: kots-dex
  • Deploying Admin Console
    • Creating namespace ✓  
    • Waiting for datastore to be ready ✓  
Enter a new password to be used for the Admin Console: ••••••••
  • Waiting for Admin Console to be ready ✓  
Error: failed to forward port: unable to listen on any of the requested ports: [{8800 3000}]

Two issues here:

1. no flag to set the port, and no way to say "just get any port"
2. No information is presented about how to fix, or what to do next (e.g. When this is resolved, run kubectl kots admin-console -n kots-dex to connect to the admin console or run kubectl kots admin-console -n kots-dex --port PORT to connect to the admin console on a different port)

How can we remove the clusterrole/binding for the operator?

kotsdam-operator is currently deployed using a serviceaccount that links a clusterrole and clusterrolebinding. This is a big security ask for someone to install. If we need this (we do for preflights and support bundle), how can we make this optional, or fall back to a role/binding that is tied to the namespace? This could break on locked down clusters, or openshift.

kots upstream upgrade does not work

When running:

kots upstream upgrade <app> --namespace <ns>

KOTS always returns:

Error: Unexpected response from the API: 500

kots installer should prompt for install location, if /usr/local/bin is not writable

Today, the installer will attempt to write the kubectl-kots binary to /usr/local/bin. If this directory is not writable, the installer will attempt to sudo to write.

Proposal: if the /usr/local/bin directory is writable, proceed with writing the binary there, no changes. If the directory is not writable by the user, prompt for a directory (defaulting to /usr/local/bin). Only then attempt to write and use sudo if needed to write.

This will allow users to install kots to a new directory, and not a privileged location.

`kots download` fails with "error opening tar archive for reading"

After specifying a valid namespace & destination folder, kots download consistently results in the following error:

  • Connecting to cluster ⠦Error: opening tar archive for reading: wrapping file reader: unexpected EOF
opening tar archive for reading: wrapping file reader: unexpected EOF

How can a KOTS application include a podspec that has optional containers

A podspec can contain multiple containers, but it would be nice if a KOTS application could use the template functions and config to dynamically include/exclude containers from a pod spec...

Perhaps this could be accomplished using annotations?

Helm resources aren't removed if chart is included and later excluded

1. I add a Helm chart to my setup. It doesn't have a exclude so it's always included.
2. Chart is included and resources are created.
3. I later add an exclude which evaluates to true.
4. Chart is removed from the YAML but resources aren't cleaned up.

I would've expected something like helm uninstall to be run after every deployment to ensure that the environment matched expected.

kots download should put it a better place.

The default directory for kots download is $home. This is an unexpected place and can make a big mess.

Some better alternative would be:

• The current directory ./
• A directory under home $home/kots

additionalNamespaces: kotsadm-replicated-registry secret not propagated if all containers are public

If an application's workloads all consist of only public images, it appears that the kotsadm-replicated-registry secret does not get propagated to any namespaces listed in additionalNamespaces

See this kotsapp for reference: replicatedhq/kotsapps@master...dexhorthy:multi-namespace

 kubectl kots install multi-namespace/multi-namespace

After install, run kubectl get secret -n hardcoded and see it's missing. If you swap to using a private image, the secret will get propagated.

apiVersion: kots.io/v1beta1
kind: License
metadata:
  name: mnsbigbank
spec:
  licenseID: 1aoJdIu8IJAmM2kCT6bYDSZmtYv
  licenseType: trial
  customerName: MNS BIG BANK
  appSlug: multi-namespace
  channelName: multi-namespace
  licenseSequence: 1
  endpoint: 'https://replicated.app'
  entitlements:
    expires_at:
      title: Expiration
      description: License Expiration
      value: ''
      valueType: String
  isGitOpsSupported: true
  isSnapshotSupported: true
  signature: >-
    eyJsaWNlbnNlRGF0YSI6ImV5SmhjR2xXWlhKemFXOXVJam9pYTI5MGN5NXBieTkyTVdKbGRHRXhJaXdpYTJsdVpDSTZJa3hwWTJWdWMyVWlMQ0p0WlhSaFpHRjBZU0k2ZXlKdVlXMWxJam9pYlc1elltbG5ZbUZ1YXlKOUxDSnpjR1ZqSWpwN0lteHBZMlZ1YzJWSlJDSTZJakZoYjBwa1NYVTRTVXBCYlUweWEwTlVObUpaUkZOYWJYUlpkaUlzSW14cFkyVnVjMlZVZVhCbElqb2lkSEpwWVd3aUxDSmpkWE4wYjIxbGNrNWhiV1VpT2lKTlRsTWdRa2xISUVKQlRrc2lMQ0poY0hCVGJIVm5Jam9pYlhWc2RHa3RibUZ0WlhOd1lXTmxJaXdpWTJoaGJtNWxiRTVoYldVaU9pSnRkV3gwYVMxdVlXMWxjM0JoWTJVaUxDSnNhV05sYm5ObFUyVnhkV1Z1WTJVaU9qRXNJbVZ1WkhCdmFXNTBJam9pYUhSMGNITTZMeTl5WlhCc2FXTmhkR1ZrTG1Gd2NDSXNJbVZ1ZEdsMGJHVnRaVzUwY3lJNmV5SmxlSEJwY21WelgyRjBJanA3SW5ScGRHeGxJam9pUlhod2FYSmhkR2x2YmlJc0ltUmxjMk55YVhCMGFXOXVJam9pVEdsalpXNXpaU0JGZUhCcGNtRjBhVzl1SWl3aWRtRnNkV1VpT2lJaUxDSjJZV3gxWlZSNWNHVWlPaUpUZEhKcGJtY2lmWDBzSW1selIybDBUM0J6VTNWd2NHOXlkR1ZrSWpwMGNuVmxMQ0pwYzFOdVlYQnphRzkwVTNWd2NHOXlkR1ZrSWpwMGNuVmxmWDA9IiwiaW5uZXJTaWduYXR1cmUiOiJleUpzYVdObGJuTmxVMmxuYm1GMGRYSmxJam9pV21sbFZtMUdXakpKTHk4MlowRTVSRkJRWVZBeFJ5dFVMMUI1VVd4emJHNVFLelZDWjFoRGJqTmFWQ3RuZFUxWmR5dHJTVkJSUmpRNVVVbHpiVzF2Wm1GQlVXbExNR1J4VkUxelkySmFiaTlJWWpaMGRXWlVkblJwUTNjNGJTdGtSbGcwTURWV1UwTTNPWEpFVjBkb2JpOU5UVmxoVTJkRmRHdFFiR3hYTjBKU1YyOW1lRVJhSzBOV1YzbExkR05xUTFFMFpHTXhVWFl6VUVSdWNISnJiSFIyUTBoc1pUbDFaR1ozTmtOUWRWcDRVVms1TDFSNFdFWllaVFY1VWt0d1ZVSkNaVkpQWkZBMFRta3ZPWFpCWlcxTmRXbHZMMnh6TlRVdlRtVlhRa3cyVVU0eWRUbDNNbEJsUm1zM1FtNHZhSE00TWpOb1FVNDBaVEJWYm5kUWRIZHliVzE1Y1RSTFdHdFVZMDVTYm5Ob2EwbHBTM2MxYTNkcVpWaE9aMlJUWWxBM1JqUkdOMlJxTDB3MU16ZGxNbkZRVlhGM04wdHZTa05HTUZkR1MyaHFhMjVIZWxKWlZGZzVaR3BKZGpBNU1FTm5hR2wzUFQwaUxDSndkV0pzYVdOTFpYa2lPaUl0TFMwdExVSkZSMGxPSUZCVlFreEpReUJMUlZrdExTMHRMVnh1VFVsSlFrbHFRVTVDWjJ0eGFHdHBSemwzTUVKQlVVVkdRVUZQUTBGUk9FRk5TVWxDUTJkTFEwRlJSVUYyWjJob2IyWnNOVlpTT1hWUk1WY3hORUZRZVZ4dU5HOVZOekF3UWxKSmJYTnFTbnBvTldaR1prbFZkakY1YWtZM1ZsWm5jUzlLU21OVGRubGxTRlZNTmxsTlEySXJRbFpJWVZCTmFIRlFNalpYWW1WUVNWeHVTVVpoYTJwTmNTc3ZjMHBIVW5SSlNrNTFOakZ3YkdWU1FYQnJWM0J5ZEZSclRIZFZObEZGYmtvNVpYVTVkbUpyVDI4clQzRkpOMUpQV0RnM2RETnBNRnh1ZVRoSWNYbEJOR0ZNZDIxcE1sWnhPRTV0U1dwMGJrWklSa0Z4TVdGQ00xRjRlVGhNTlVWNlQza3dkeTlKTjFGbFVVWlZjVkphYVhKM1ptSXlkRVFyYzF4dWIyUXpiMEkzUlc5eE16WkhLMkpPT1VFd1lrSnBaM0JqUmlzdldHUkNSemd2YjFSMGFtbzJabHBXYm5OU1NHcFBUSFZpVDJZMmVrNW9RV3ByYnpkdGRseHVOVWxOTjNvMmVVOWlkRXd4TDFoaVRucHRPVFpZUmtKdVJqRnFVbkZyU1ZGWE1FWkNOMUpQU25SWWQyVlphbkJsVDBJd2MzTlRSMnhPZFhKUmJXWndhMXh1ZEhkSlJFRlJRVUpjYmkwdExTMHRSVTVFSUZCVlFreEpReUJMUlZrdExTMHRMVnh1SWl3aWEyVjVVMmxuYm1GMGRYSmxJam9pWlhsS2VtRlhaSFZaV0ZJeFkyMVZhVTlwU25WaVIxcDNaV3BLVlZZeU1UUmlSRmt5VGxWc01tRkZSbUZrV0doT1RtNWFZVTFJUlhsYWVrMTJUa001U0U5WFZuUmhhM2gxWld0Sk1WbHFaRzlQU0ZKRFdsaE9URnBzUWpCamJtaFRWVk00ZVdWdWFGSlNSazVDWWxVeFExRlhValJVTWtwd1UxVnpOV05YT1hsaVYyTTBWRWRuTTFOSE9VeFRSRlY2V1ZVMVVrNXJOSEpUYW1SNlZWaE9lRk5HVWpSVmVtUkdaRzV2TldOV1JuZFdNVlo1VWtST1dsZEdRbHBrTURsb1pVWktRMWxYTUhKak1VbzBZMWhPY1ZZeWFIUldWbFV4VVRJMWMxSXliRkJpU0dSVVRrZGtNbVJYT1hKVWJHeENUa1JPYTFacmVHbGFWa3BIVWtkdmVHTXdXVEZXTTBaUVZVVkpNVmRYT1Zaa2FsWnVZM2s1V1dOSGJHcFRWVEZRWkd0emQyUldaSGRUYTA1T1VXdHNXVnB1Vm5KVVNFcEdVekpPVjFGcVNtaE5WV1JzWWtkU1QxWnFhRXhVYXpWclRsWm9UVkZYVmxkVk0xWm9VbGRHWVdGdVJrSldiWEJDWkc1Q2FHTkVUbTVVVm1oT1kyeHdkazVWYUc5TlV6bHFVVEp2TkZNeU1VTmxWMFpXVkcxVk1sbHJkSEJhV0VwTVVXeENTVlJWTVVwVmEyTjVWMnQ0VDFWSE5ESlhWMFo0WkVWb01FNUhiSEpOVkVsMlYxUk5NRTFGTVZwTk1rWkdUVlpzTkU1NlZsaGxSMmhzV2xVMVZWUlZSVGxRVTBselNXMWtjMkl5U21oaVJYUnNaVlZzYTBscWIybFpiVkpzV2xSVk1rNVVXWGRaTWxwcFRrUk9hazlYU1hsUFIwcHRUMVJvYkZsWFRtaGFiVVV5VGtSWmFXWlJQVDBpZlE9PSJ9

There is no 'version' command

We pass version information as part of ldflags, but the package referenced does not exist.

https://github.com/replicatedhq/kots/blob/a0539db/deploy/.goreleaser.yml#L19-L22

Support for --set to pass options to a helm chart is not implemented

Remove the version number from the filename in releases

Let's remove the version number from the filename in releases. Currently it's very redundant:

https://github.com/replicatedhq/kots/releases/download/v0.3.0/kots_0.3.0_linux_amd64-0.3.0.tar.gz

Let's make this:

https://github.com/replicatedhq/kots/releases/download/v0.3.0/kots_linux_amd64.tar.gz

KOTS install should support bringing an external postgres instance

Unclear on the design for this -- I'm guessing this looks like a CLI flag on kots install ?

kots install for helm charts should default to the chart name

Currently it defaults to applicationABCDEF

Feature: Support long running operations that can be interrupted by the API being rescheduled

Currently, we execute some long running operations (checking for updates, gitops commits, pushing to registries) in the API. Some of these are sync, some async, as needed. But the API pod executing this function could be rescheduled at any time. Today that means that the user needs to make the request again to try. It would be better if the API could use the database or something else as a durable queue of executing tasks (maybe the K8s api to store this?)

libdevmapper.so.1.02.1: cannot open shared object file: No such file or directory

Environment: Centos 7.7
Kots Version: 1.1.0

Error:

kubectl-kots: error while loading shared libraries: libdevmapper.so.1.02.1: cannot open shared object file: No such file or directory

Workaround:
Creating a symlink to libdevmapper.so.1.02 fixes the issue.

Merge replicatedhq/kotsadm repo into here

We'd like to make this repo (kots) contain both the kots and the kotsadm projects. This needs design -- creating this issue to track work happening on it.

Expose Troubleshoot Collector/Analyzer insights in kots template context

For insights generated by https://troubleshoot.sh, it would be great to access some of the values in the kots CLI template context. Maybe these could be stored in upstream/userdata or something each time preflight checks are executed?

Example

Troubleshoot has a collector/analyzer for "Kubernetes Distro", to check for
it would be great to be able to access collector results in the template context, e.g.

apiVersion: v1
kind: Route
metadata:
  name: host-route
  annotations:
    kots.io/when: '{{repl (eq (TroubleshootCollector "kubernetesDistro") "openshift") }}'
spec:
  host: www.example.com  
  to:
    kind: Service
    name: service-name

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: host-route
  annotations:
    kots.io/when: '{{repl not (eq (TroubleshootCollector "kubernetesDistro") "openshift") }}'
spec:
  backend:
    serviceName: service-name
    servicePort: 8080

kots pull helmchart should not require a license

As discussed over slack this is not the expected behavior.

epetrini@epetrini-mac:~/temp/kots$ kubectl kots pull helm://stable/elasticsearch

Error: failed to parse license from file: failed to read license file: read /Users/epetrini/temp/kots: is a directory
epetrini@epetrini-mac:~/temp/kots$ kubectl kots version
Replicated Kots 1.13.4

Support decrypted ConfigValues in kots download/upload

Currently, kots download returns the user-supplied config values in upstream/userdata/config.yaml. This is an internally-generated file with GVK kots.io/v1beta1, Kind=ConfigValues. This file can be edited and then uploaded with kots upload and KOTS will regenerate the base with the new values.

However, if a config item type is "password", the value is encrypted, making it very difficult to update.

A possible solution here is for kots download to return the password types decrypted, and then kots upload can receive the decrypted/updated value and re-encrypt before saving.

To facilitate this, the ConfigValue spec should be extended to support both value and valuePlaintext. If a value is supplied in kots upload in valuePlaintext, it will be assumed to overwrite the currently encrypted.

kots download should not include the decrypted values by default, unless a flag is passed to include them.

This change will enable automated workflows that are not currently possible today.

kots darwin executable is not signed

Because CI is performed on GitHub Actions, we need to find a way to sign the GoReleaser binary. Looking for examples or "prior art" here.

Long text for boolean does not format well

doesn't look great. This is generated by ``` - name: ha_object_store title: Highly Available Object Store help_text: | This application includes MinIO as an in-cluster Object Store. You can enable a highly available deployment mode to avoid downtime during upgrades or spent recovering if a host node becomes unavailable. type: bool default: "0" ```

Kots cli inconsistencies with arguments & flags

Various kots commands behave differently than the usage strings suggest, and common flags (like namespace) should include common abbreviations.

kots upload usage string suggests 'namespace' is a required arg, but it actually interprets arg[0] as the source directory for upload. I believe the documentation should be updated to match the behavior.

kots download usage suggests that the first argument should be the app-slug, but doesn't provide a helpful error when this argument is omitted.

--namespace should include '-n' abbreviation in all the commands that use this flag.

Allow templating HelmChart namespace

It does not look like template functions in type HelmChart spec.namespace are ever evaluated

kots/pkg/base/replicated.go

Lines 172 to 175 in 46fe2b9

	namespace := kotsHelmChart.Spec.Namespace
	if namespace == "" {
	namespace = "repl{{ Namespace}}"
	}

Explore adding OCI format to airgap packages instead of docker tar

Currently our airgap packages include docker-proprietary .tar files that are exports of each image. As identified in spring-cloud/spring-cloud-dataflow#3858, this creates some challenges and inefficiencies.

We should explore supporting OCI format instead of Docker format here.

Some options to consider:

• https://github.com/pivotal/image-relocation
• kbld
• sheaf

While not every registry can support an OCI image yet, many can. As long as we can fall back to pushing to an image repository that wants to receive a docker image, our packaging could be much more flexible.

kots download documentation isn't accurate.

There is a required option for kubectl kots download that isn't marked as such. Running the following returns only the usage, and does not download the manifests:

• kubectl kots download
• kubectl kots download --dest .
• kubectl kots download --dest . -n <namespace>

If an option is required, it should either be part of the usage and/or have a [required] tag in its description.

For reference here is the current kots download usage:

Download the active Kubernetes manifests from a cluster to the local filesystem so that they can be edited and then reapplied to the cluster with 'kots upload'.

Usage:
  kots download [flags]

Flags:
      --dest string   the directory to store the application in (default "/Users/aaronc")
  -h, --help          help for download
      --overwrite     overwrite any local files, if present
      --slug string   the application slug to download

Global Flags:
...

minio init process is unneeded after migrations completes

In some future version (1.7?) we should remove the initcontainer for minio added as part of #175. It's only required for the first run after upgrading from <=1.4, and so as long as kots is started once on 1.5/1.6, 1.7 would be able to forgo it entirely.

Console output text color ignores user terminal settings

Kots console output can be invisible, depending on the user's terminal background color settings. For example, much of the explanatory text is displayed in white, and ends up totally invisible if console background is also white.

All the console output should be visible, regardless of terminal color profile.

Enable 386 and darwin and windows builds in goreleaser

Starting in v0.9.5 we are only building for linux_amd64. This is due to additional deps needed from importing some CGO libraries. We should reenable all architectures and get them building.

Support for preleases

Currently, a kots prerelease will always use master (alpha image tag) of kotsadm. We intentionally ignore prerelease tags and always use alpha. IIRC, we do this so that interim releases work, but this is starting to make beta releases difficult. The code is here:

kots/pkg/kotsadm/kotsadm_version.go

Line 30 in 5dd232b

return "alpha"

There doesn't seem to be a test preventing us from changing this. @divolgin or @laverya do you know why this code exists?

Test_PullReplicated is flaky in CI, re-enable

kots/integration/replicated/pull_test.go

Line 24 in c787e34

func Test_PullReplicated(t *testing.T) {

Unable to install using '--license-file' on the CLI

$ kubectl kots install cd4pe/unstable --license-file license.yaml
Enter the namespace to deploy to: cd4pe

  • Pulling upstream ✓
  • Creating base ✓
  • Creating midstream ✓
  • Creating downstream "this-cluster" ✓
  • Deploying Admin Console
    • Creating namespace ✓
    • Waiting for datastore to be ready ✓
Enter a new password to be used for the Admin Console: ••••••••••
  • Waiting for Admin Console to be ready ✓
Application name: kotsadm042477586
  • Uploading local application to Admin Console ✗
Error: failed to unmarshal response: invalid character '<' looking for beginning of value

The same seems to work fine if I omit --license-file and use the same file in the console workflow.

My goal is to install the application without GUI intervention.

kots CLI: kots install with --license-file hangs after "Creating base" with no feedback

 k kots install my-kots-app --license-file ~/Desktop/path/to/license.yaml

Output gets to here and then just hangs for minutes

Enter the namespace to deploy to: my-kots-app

  • Pulling upstream ✓  
  • Creating base ✓  

I'm guessing this is starting datastores, apis, etc but we lose all the regular kubectl kots install output.

Update after several minutes I get this:

  • Creating base ✓  
  • Creating midstream ✓  
  • Creating downstream "this-cluster" ✓  
  • Deploying Admin Console
    • Creating namespace ✓  
    • Waiting for datastore to be ready ✓  
Enter a new password to be used for the Admin Console: 

Feature Request: allow installing and starting an application from the command line.

Currently, to install a kots application there are a number of steps that can not be done on the command line. For example, setting the password, inputting config values, and the first deploy. The ask is to enable automatic testing of the user's workflow of installing an application to it up and running without having to open the browser.

Here are the minimum steps to install a running application, and ideally, each of these should be configured at the command line in some way:

• Choose a namespace. Can use -n/--namespace
• Choose a password.
• Choose a license file. Can use --license-file, but this has interesting side effects.
• Configure the application. A --config flag would be a useful start
• Click the deploy button from the version history page.

Ideally, I would like to run a single command to have an application installed in my CI pipeline. For example kubectl kots install app-name -n namespace --pasword supersecret --license-file license.yaml --config config.yaml --deploy. I could then wait for my application to be ready and perform tests.

if :8800 is not available, can we pick a different port?

What changes will the web UI need to support running on a port != 8800?