fluxcd / flux2

Open and extensible continuous delivery solution for Kubernetes. Powered by GitOps Toolkit.

Flux version 2

Flux is a tool for keeping Kubernetes clusters in sync with sources of configuration (like Git repositories and OCI artifacts), and automating updates to configuration when there is new code to deploy.

Flux version 2 ("v2") is built from the ground up to use Kubernetes' API extension system, and to integrate with Prometheus and other core components of the Kubernetes ecosystem. In version 2, Flux supports multi-tenancy and support for syncing an arbitrary number of Git repositories, among other long-requested features.

Flux v2 is constructed with the GitOps Toolkit, a set of composable APIs and specialized tools for building Continuous Delivery on top of Kubernetes.

Flux is a Cloud Native Computing Foundation (CNCF) graduated project, used in production by various organisations and cloud providers.

Quickstart and documentation

To get started check out this guide on how to bootstrap Flux on Kubernetes and deploy a sample application in a GitOps manner.

For more comprehensive documentation, see the following guides:

• Ways of structuring your repositories
• Manage Helm Releases
• Automate image updates to Git
• Manage Kubernetes secrets with Flux and SOPS

If you need help, please refer to our Support page.

GitOps Toolkit

The GitOps Toolkit is the set of APIs and controllers that make up the runtime for Flux v2. The APIs comprise Kubernetes custom resources, which can be created and updated by a cluster user, or by other automation tooling.

You can use the toolkit to extend Flux, or to build your own systems for continuous delivery -- see the developer guides.

Components

• Source Controller
  • GitRepository CRD
  • OCIRepository CRD
  • HelmRepository CRD
  • HelmChart CRD
  • Bucket CRD
• Kustomize Controller
  • Kustomization CRD
• Helm Controller
  • HelmRelease CRD
• Notification Controller
  • Provider CRD
  • Alert CRD
  • Receiver CRD
• Image Automation Controllers
  • ImageRepository CRD
  • ImagePolicy CRD
  • ImageUpdateAutomation CRD

Community

Need help or want to contribute? Please see the links below. The Flux project is always looking for new contributors and there are a multitude of ways to get involved.

• Getting Started?
  • Look at our Get Started guide and give us feedback
• Need help?
  • First: Ask questions on our GH Discussions page.
  • Second: Talk to us in the #flux channel on CNCF Slack.
  • Please follow our Support Guidelines (in short: be nice, be respectful of volunteers' time, understand that maintainers and contributors cannot respond to all DMs, and keep discussions in the public #flux channel as much as possible).
• Have feature proposals or want to contribute?
  • Propose features on our GitHub Discussions page.
  • Join our upcoming dev meetings (meeting access and agenda).
  • Join the flux-dev mailing list.
  • Check out how to contribute to the project.
  • Check out the project roadmap.

Events

Check out our events calendar, both with upcoming talks, events and meetings you can attend. Or view the resources section with past events videos you can watch.

We look forward to seeing you with us!

flux2's Issues

Make tk install write progress report lines to standard error

I am using tk install with dry-run and verbose options to generate the yaml to deploy gitops.
However I have to manually edit out the progress report lines emitted by tk install.
If these lines were written to standard error and the yaml lines written to standard out I wouldn't have to remove them manually.

generating deploy key failed (self-hosted GitLab server)

Hello,

I just tried to bootstrap the toolkit components in a GitLab (self-hosted) repository and got the following error:

$ tk bootstrap gitlab --verbose --namespace=gitops-toolkit --owner=devops --repository=gitops-toolkit --hostname=gitlab.mydomain
► connecting to gitlab.mydomain
✔ repository cloned
✚ generating manifests
✔ components are up to date
► installing components in gitops-toolkit namespace
namespace/gitops-toolkit unchanged
customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/providers.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/receivers.notification.toolkit.fluxcd.io configured
role.rbac.authorization.k8s.io/crd-controller-gitops-toolkit unchanged
rolebinding.rbac.authorization.k8s.io/crd-controller-gitops-toolkit unchanged
clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-gitops-toolkit unchanged
service/notification-controller unchanged
service/source-controller unchanged
service/webhook-receiver unchanged
deployment.apps/helm-controller configured
deployment.apps/kustomize-controller configured
deployment.apps/notification-controller configured
deployment.apps/source-controller configured
networkpolicy.networking.k8s.io/deny-ingress unchanged
deployment "source-controller" successfully rolled out
deployment "kustomize-controller" successfully rolled out
deployment "helm-controller" successfully rolled out
deployment "notification-controller" successfully rolled out
✔ install completed
► configuring deploy key
✗ generating deploy key failed: SSH key scan for host gitlab.mydomain:22 failed, error: dial tcp 1.2.3.4:22: i/o timeout

$ tk --version
tk version 0.0.17

It's because our GitLab http endpoint is accessible via https://gitlab.mydomain and the ssh endpoint via ssh.gitlab.mydomain

tk --version returns incorrect version

Using Toolkit CLI v0.0.2, when tk --version it returns:

$ tk --version
tk version 0.0.1-beta.3

Move HelmRelease to helm.toolkit.fluxcd.io/v1(alpha1) ?

Since HelmRelease moved to a new api version domain (helm.toolkit.fluxcd.io) with #136, should it be v1alpha1 instead of v2alpha1, since this is technically the first version of this CRD, (as identified by the toolkit domain)? Currently it could be confused that this is the second version within the toolkit. I can see both sides, just making sure it was considered.

Use standardized status condition schema

The standardized status condition schema:
kubernetes/enhancements#1624

Is available in kuberentes 1.19 libraries:
https://github.com/kubernetes/apimachinery/blob/release-1.19/pkg/apis/meta/v1/types.go

There are additionally some helper libraries which should be available in an upcoming version:
kubernetes/kubernetes#92717
This for example handles only updating lastTransitionTime if the status actually changes, which
we are not doing currently.

Also observedGeneration is included as a per-condition field.

Prettify generated CLI docs

The CLI documentation is readable at the moment but could use some templating and formatting improvements.

• Adjust things so SEE ALSO are visible menu items
• Fix casing on titles
• Avoid footer note to be showed in ToC
• Run doc generation as part of CI
• Ensure all commands have long descriptions and examples

Normalise controller-runtime logs and metrics

We should have a consistent log structure and error handling procedure across all toolkit controllers.

• source-controller
• kustomize-controller
• helm-controller
• notification-controller

Structure logging:

log := r.Log.WithValues("controller", strings.ToLower(<API Kind>), "request", req.NamespacedName)

Log sample:

{"level":"error","ts":"2020-07-15T11:19:45.031+0300","logger":"controller-runtime.controller","msg":"Reconciler error","controller":"gitrepository","request":"default/podinfo","error":"git clone error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"}

{"level":"error","ts":"2020-07-15T11:19:56.142+0300","logger":"controllers.GitRepository","msg":"unable to update status","controller":"gitrepository","request":"default/podinfo","error":"Operation cannot be fulfilled on gitrepositories.source.fluxcd.io \"podinfo\": the object has been modified; please apply your changes to the latest version and try again"}

{"level":"info","ts":"2020-07-15T11:21:18.954+0300","logger":"controllers.GitRepository","msg":"Reconciliation finished in 1.250148853s, next run in 1m0s","controller":"gitrepository","request":"default/podinfo"}

Broadcast event and return error on reconciliation failures:

if reconcileErr != nil {
	r.event(<Reconciled Object>, recorder.EventSeverityError, reconcileErr.Error())
	return ctrl.Result{Requeue: true}, reconcileErr
}

Note for cluster reconcilers like kustomize or helm controller, requeuing should happen at the specified interval.

Metrics sample:

# HELP controller_runtime_reconcile_errors_total Total number of reconciliation errors per controller
# TYPE controller_runtime_reconcile_errors_total counter
controller_runtime_reconcile_errors_total{controller="gitrepository"} 28

# HELP controller_runtime_reconcile_total Total number of reconciliations per controller
# TYPE controller_runtime_reconcile_total counter
controller_runtime_reconcile_total{controller="gitrepository",result="error"} 28
controller_runtime_reconcile_total{controller="gitrepository",result="requeue_after"} 2

Create get started guide

We should create a docs website with GH pages and add a "get started" guide showcasing the toolkit bootstrap and source/kustomization operations.

Publish roadmap

We should publish a first draft of the toolkit roadmap on the docs website.

Preparation for v0.1.0 release

After working for six months on the GitOps Toolkit we've reached feature parity with Flux v1 read-only and Helm Operator v1. I think it's time to make a v0.1 release and promote the APIs to beta. Version 0.1.x will include breaking changes to v0.0.x, so until we release v0.1 we should freeze gotk to 0.0.28 and bundle all changes into 0.1.

Pre 0.1.0:

• rename the default namespace from gitops-system to gotk-system
• change DependsOn type from []string to []CrossNamespaceDependencyReference
• incorporate DependsOn changes in gotk create [...] commands
• replace Kustomization.Status.Snapshot.Revision with Checksum in Kustomization API
• add support for CAPI remote clusters in Kustomnization API
• add support for S3 Bucket sources
• publish multi-arch ARMv7/v8 container images
• promote source/kustomize/helm/notification APIs from alpha1 to beta1
• update Go to v1.15
• make controller releases

Post 0.1.0:

• remove Experimental from the controllers descriptions and docs
• extend documentation around managing Helm releases
• create a migration guide for Flux v1 read-only users
• create a migration guide for Helm Operator v1 users
• update notices in fluxcd/flux and fluxcd/helm-operator README.mds
• highlight the existence of the project on the fluxcd.io website

Should tk use KUBECONFIG environmental variable if set?

Running tk check I was surprised to see it report an unexpected kubernetes server version.
On investigation it seems that tk supports the specification of a kubeconfig file using --kubeconfig option but if this is not provided it uses ~/.kube/config.
If a user has set the KUBECONFIG environmental variable they might reasonably expect tk to use that value?

Collect all errors for bulk `kubectl` actions before returning

At present when we are for example checking the rollout status of multiple installed components, it will stop and only return the first error if some component's rollout failed.

To provide a user with a better overview of what happened, we should complete the bulk run before bailing so that we can return a full overview of what did and did not succeed.

For bonus points: we would like to minimize the number of shell-outs we make to kubectl, and use the Golang Kubernetes client where possible. An example of this for rollout checks can be found here.

bootstrap branch into a flag

When bootstrapping a cluster it is hardcoded const to master. Hoping this would be updated to default to main and we can make this a flag.

https://github.com/fluxcd/toolkit/blob/master/cmd/tk/bootstrap.go#L42

Error using tk completion command on zshrc

Hi!
I'm following the Getting Started steps and I getting the following error:

. <(tk completion)
/dev/fd/11:type:1874: bad option: -t

If I run tk completion alone then works perfect, but if I try to include it into my ~/.zshrc file I receive the error mentioned above.

Environment:

• darwin
• zsh
• tk version 0.0.11

Add notification-controller/api CLI commands

The gotk CLI should contain commands for the notification-controller/api resources:

• gotk <action> alert-provider
• gotk <action> alert
• gotk <action> receiver

Cover more HelmRelease configuration options

The flux create helmrelease command currently offers the bare minimum of options made available by the API.

We would like to extend the available flags to a balanced set that works from a CLI perspective. Given the verbosity of the API, this may mean that some options will be ignored or that there are certain flags that provide "shortcuts" to a configuration set (remediation strategies is one thing that comes to mind here).

Tasks:

• Discuss / agree on the configuration options that should be covered
• Discuss / agree on the flags that control the options
• Implement the flags

Create log interface

We are currently printing output using our own set of private methods (e.g. logAction, logGenerate). Those methods should be factored out into a log interface (and our own implementation).

This would allow us to move the bootstrap gears into pkg/ so they can be re-used by interested parties, while providing a sane way to stay in control over the output.

GitHub Deployments

Hey, am playing with the idea of how to use GitHub Deployment API to drive deployments in a gitops way. GitHub Deployments API is nice because it provides a UI within GitHub which shows what commits are currently deployed within each environment. This is also useful for doing pull request deployments you can add a URL to the deployment where the deployment can be accessed from which also shows up in the pull request UI.

Was wondering if it makes sense to allow referencing a deployment environment in the GitRepository API. The controller could then fetch the commit sha from the latest deployment that matches that environment, something like:

apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
  name: podinfo
  namespace: default
spec:
  interval: 1m
  url: https://github.com/stefanprodan/podinfo
  ref:
    deployment: production

You could then trigger the controller by creating a new deployment like so:

curl -X POST -H "Authorization: Bearer ${GITHUB_TOKEN} --data '{"ref": "master", "environment": "production"}' https://api.github.com/stefanprodan/podinfo/deployments

The new deployment would include the latest sha from the master branch which can be used to grab the latest source.

This however does require the "cluster controller" (e.g. kustomize-controller) to update the deployment status once it has been applied - not sure where this would fit in with regards to the gitops toolkit.

Another problem I can see is that this is GitHub specific but GitRepository looks like it should work with any Git provider, not sure if other Git providers have a similar concept which could be used or whether this should be a different kind e.g. GitHubRepository or GitHubDeployment

HelmRelease is not created

I'm trying to use the latest version of the toolkit and create some services using HelmRelease. I created the following resources in my repository:

apiVersion: source.toolkit.fluxcd.io/v1alpha1
kind: HelmRepository
metadata:
  name: stable-helm-repo
  namespace: gitops-system
spec:
  interval: 1h
  url: https://kubernetes-charts.storage.googleapis.com

apiVersion: helm.toolkit.fluxcd.io/v2alpha1
kind: HelmRelease
metadata:
  name: grafana
  namespace: grafana
spec:
  interval: 1m
  chart:
    spec:
      chart: grafana
      version: '7.1.x'
      sourceRef:
        kind: HelmRepository
        name: stable-helm-repo
        namespace: gitops-system
      interval: 1m

While I see that HelmRepository resource is correctly created during the synchronization process, but HelmRelease is missing although the source-controller doesn't report any error.

Error bootstrapping toolkit on cluster with HelmRelease CRD v1

Trying to use the toolkit against a brand new K8s cluster

$ tk check --pre
► checking prerequisites
✔ kubectl 1.18.2 >=1.18.0
✔ Kubernetes 1.16.11 >=1.14.0
✔ prerequisites checks passed

But the bootstrap fails:

$ tk bootstrap github \
>  --owner=$GITHUB_USER \
>  --repository=gotk-example \
>  --path=gitops-runtime \
>  --personal
► connecting to github.com
✔ repository created
✔ repository cloned
✚ generating manifests
✔ components manifests pushed
► installing components in gitops-system namespace
namespace/gitops-system created
customresourcedefinition.apiextensions.k8s.io/alerts.notification.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/providers.notification.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/receivers.notification.fluxcd.io created
role.rbac.authorization.k8s.io/crd-controller-gitops-system created
rolebinding.rbac.authorization.k8s.io/crd-controller-gitops-system created
clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-gitops-system created
service/notification-controller created
service/source-controller created
service/webhook-receiver created
deployment.apps/helm-controller created
deployment.apps/kustomize-controller created
deployment.apps/notification-controller created
deployment.apps/source-controller created
networkpolicy.networking.k8s.io/deny-ingress created
The CustomResourceDefinition "helmreleases.helm.fluxcd.io" is invalid: status.storedVersions[0]: Invalid value: "v1": must appear in spec.versions
✗ install failed

Windows support

The toolkit CLI shells-out to kubectl using sh -c as a wrapper, we should create a kubectl helper that works on Windows/macOS/Linux, similar to kustomizer.

Create dev guide

We should create a development and contributing guide to the toolkit docs that covers both controllers and CLI dev workflows.

Automate components updates

Create GitHub actions workflow that opens PRs every time a controller is released. The PR should bump the version in manifests/bases and go.mod.

Workflow:

• fetch latest release from GitHub API
• run go mod edit -require=github.com/fluxcd/source-controller@<VERSION>
• run go mod tidy
• check for changes with git diff
• replace the version in bases/source-controller/kustomization.yaml
• commit changes
• open PR with peter-evans/create-pull-request@v2

Combined changelog

It'd be great to have a combined changelog of what's been happening in the GitOps Toolkit parts, so you don't have to go around chase in all *-controller projects.

Use Kubernetes finalizers for garbage collection

The toolkit controllers should register and trigger pre-delete hooks in their reconcile functions to perform garbage collection:

• source-controller
• kustomize-controller
• helm-controller

Add checksum verification to install script

The install script should download the checksums.txt from the release page and verify the tar.gz before extracting it.

Platform agnostic bootstrapper

A simple and straight forward way to make the toolkit work with any platform is to provide a bootstrapper that just assumes it should connect to some Git server while making use of the local SSH agent.

Command:

tk bootstrap git \
--repository=<GitHub repository name> \
--version=<toolkit version> \
--components=<toolkit controllers> \

Workflow:

• clone the repository
• generate install manifests
• commit and push the install manifests to git using the local SSH agent (override the files if they exist)
• apply the manifests on the target cluster (on install)
• wait for controllers rollout
• generate SSH keys and hosts (on install)
• print generated public key (on install)
• create a source manifest for the specified Git repository (on install)
• create a kustomization manifest for reconciling the manifests (on install)
• commit and push the source/ks manifests to git (on install)
• apply the manifests on the target cluster (on install)
• wait for the kustomization rollout

Bonus points:

Some Git servers allow new repositories to be created on push, we could provide an additional flag (e.g. --init) which creates a new Git repository locally, with the remote set to the provided --repository URL, instead of cloning it. Cloning should however always be attempted first to make the command idempotent.

Beautify roadmap md

@dholbach and I plan to add a progress bar and change the bullets to checkmarks. Checked/finished items will be in grey.

Create FAQ document

It'd be good to create a FAQ document which explains:

• differences from Flux v1
• how some functionality will be different
• etc

Add notification controller to bootstrap

Add notification-controller as a default component.

Improve logging and error messages

Currently most of the log output and/or errors are written as machine readable text while they are aimed to be read by humans. All logged messages need to be revised so that they can easily be read and understood by humans for a smooth UX.

Example of current messaging:
"is team %s error: %w", name, err

Example of human readable version:
"failed to determine if team '%s' is assigned to the repository, error: %w", name, err

Upgrade CRDs to apiextensions.k8s.io/v1 and drop support for Kubernetes <1.16

The apiextensions.k8s.io/v1beta1 version of CustomResourceDefinition has been deprecated in Kubernetes v1.16 and will no longer be served in Kubernetes v1.19.

Upgrade procedure:

• Makefile - set CRD_OPTIONS ?= crd:crdVersions=v1
• Makefile - update controller-gen to v0.3.0
• go.mod - update controller-runtime to v0.6.1 and go to 1.14
• Dockerfile - update go to 1.14

Controllers:

• source-controller
• kustomize-controller
• helm-controller
• notification-controller

Publish linux/amd64 and arm64 images to GitHub Container Registry

To avoid Docker Hub pull limits we should move to GitHub Container Registry and publish AMD64 and ARM64 images for the toolkit controllers:

• source-controller
• kustomize-controller
• helm-controller
• notification-controller

Reference implementation:

• Dockerfile: https://github.com/fluxcd/notification-controller/blob/master/Dockerfile
• GitHub workflow based on docker/build-push-action v2: https://github.com/fluxcd/notification-controller/blob/master/.github/workflows/release.yml

The controller images will be made public under ghcr.io/fluxcd/:

• linux/amd64: ghcr.io/fluxcd/<controller>:semver
• linux/arm64: ghcr.io/fluxcd/<controller>-arm64:semver

Once the controller images are available on GitHub Container Registry, we should add a command flag to gotk install/bootstrap that allows users to chose between amd64 and arm64 e.g.:

gotk install --arch=arm64

Watch all namespaces by default

Currently the toolkit controllers have a mix behaviour: source-controller and kustomize-controller resource watcher is restricted to the runtime namespace e.g. gitops-system while helm-controller and notification-controller are watching all namespaces.

We should introduce a boolean command flag called --watch-all-namespaces across all toolkit controllers that allows toggling the resource watcher from cluster wide to a the runtime namespace only. By default, the controllers will watch resources in all namespaces, users can change the default with gotk bootstrap/install.

When --watch-all-namespaces is enabled, the kustomize-controller service account impersonation can't be enforced by cluster admins, as any cluster user could create Kustomization objects with a cluster-admin SA. In the future we'll be designing a ReconciliationPolicy API that will allow cluster admins to enforce service accounts.

Components affected by this change:

• source-controller
• kustomize-controller
• helm-controller
• notification-controller
• gotk CLI

Failed to install helmreleases CRD

I am trying to install gitops toolkit on a brand new Kind cluster and get error relating to "x-kubernetes-preserve-unknown-fields

tk check --pre
► checking prerequisites
✔ kubectl 1.18.3 >=1.18.0
✔ Kubernetes 1.14.2 >=1.14.0
✔ prerequisites checks passed
a669981@vc2crtp2473106n:$ tk install --export > gitops.yaml
a669981@vc2crtp2473106n:$ kubectl apply -f gitops.yaml
namespace/gitops-system created
customresourcedefinition.apiextensions.k8s.io/alerts.notification.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.fluxcd.io configured
error: error validating "gitops.yaml": error validating data: ValidationError(CustomResourceDefinition.spec.validation.openAPIV3Schema.properties.spec.properties.values): unknown field "x-kubernetes-preserve-unknown-fields" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaProps; if you choose to ignore these errors, turn validation off with --validate=false

tk reconcile helmrelease error

tk version 0.0.18
kubernetes v1.18.8

I'm trying to reconcile a helm release issuing the following command:

tk reconcile helmrelease traefik

But getting the following message:

► annotating HelmRelease traefik in gitops-system namespace
✗ HelmRelease.helm.toolkit.fluxcd.io "traefik" is invalid: spec.chart.spec: Required value

DependsOn doesn't check observedGeneration

Currently when checking HelmReleases and Kustomizations specified in DependsOn, the Ready condition is checked, but not that status.observedGeneration matches the latests metadata.generation. So those resources may not have been reconciled with the latest state yet, which they probably should be. I suppose there could still be race conditions where a controller sees an updated resource before all depended on resources have been updated, but that seems less likely.

Use owner in Git author email

At the moment we use a hardcoded tk@<platform's no-reply email domain> when we make Git commits. This ties the commits to the tk user on the targeted platform, which is not something we are after.

Instead, we should use the owner value, so the commits are tied to the organization or user we perform the operation for.

Improve `gotk reconcile helmrelease` readiness check

As the helm-controller does not reset the Ready==True condition without a state change, the current readiness check for the command may result in a false positive.

To prevent this, the Status fields in the HelmRelease can be consulted to ensure reported state is from after the reconciliation request was made (via Status.LastObservedTime).

Change CRD domains to `*.toolkit.fluxcd.io`

The main reason for this is to be able to offer a migration path to Helm Operator in which the Helm Operator and helm-controller can be run in parallel on the same cluster. Due to Kubernetes' assumptions around Custom Resource Definitions, this is not possible without either:

1. introducing a new Kind for the current helm.fluxcd.io domain
2. changing the domain so that the CRD from the Helm Operator and the helm-controller do not collide

Given that HelmRelease has proven to be the right name for the Kind, we have opted for option 2. And given all controllers are part of the Toolkit, it makes most sense to change the domains for all of them to not create any confusion.

Toolkit projects that require changes:

• source-controller
• helm-controller
• kustomize-controller
• notification-controller
• toolkit

GitOps bootstrap

Currently the toolkit CLI operations are performed directly on the target cluster, we could offer an alternative workflow that allows cluster admins to perform the same actions using git.

The bootstrap command provisions a repository and configures the toolkit controllers to reconcile its content on the target cluster. The bootstrap should be idempotent to support both install and upgrade procedures. At first we will offer this feature for repositories hosted on GitHub, later on we can extend the bootstrap command to other platforms and provide a SSH agent integration (platform agnostic).

Command:

GITHUB_TOKEN=token && \
tk bootstrap github \
--owner=<GitHub user/org> \
--repository=<GitHub repository name> \
--version=<toolkit version> \
--components=<toolkit controllers>

Workflow:

• create repository if it doesn't exists
• clone the repository
• generate install manifests
• commit and push the install manifests to git (override the files if they exist)
• apply the manifests on the target cluster (on install)
• wait for controllers rollout
• generate SSH keys and hosts (on install)
• set GitHub deploy key (on install)
• create a source manifest for the specified GitHub repository (on install)
• create a kustomization manifest for reconciling the manifests (on install)
• commit and push the source/ks manifests to git (on install)
• apply the manifests on the target cluster (on install)
• wait for the kustomization rollout

Unable to deploy GitOps toolkit

I've just tried to deploy the GitOps toolkit using the existing tooling and it seems like the published GitHub registry images cannot be pulled:

❯ docker pull ghcr.io/fluxcd/notification-controller:v0.0.10
v0.0.10: Pulling from fluxcd/notification-controller
df20fa9351a1: Already exists 
207995d28a57: Pulling fs layer 
887ea3b292e5: Pulling fs layer 
224a3681e710: Pulling fs layer 
4076933fed24: Waiting 
error pulling image configuration: unknown blob

Inside Kubernetes, I'm seeing something like this:

Failed to pull image "ghcr.io/fluxcd/notification-controller:v0.0.10": rpc error: code = Unknown desc = failed to pull and unpack image "ghcr.io/fluxcd/notification-controller:v0.0.10": failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:af37d5603969808bb7a44f9e8567024db9ddee736bd742c839cb88ff25c6eaca (application/vnd.docker.container.image.v1+json) from remote: not found

`tk` binary conflicts with Tanka

The toolkit binary tk conflicts with the binary from Tanka (https://tanka.dev/), which is unfortunately operating in the same area and might cause trouble for quite a bit of Kubernetes users.

tk create source --export expects Kubernetes to be there

it seems that the tk create source git --export requires K8s to be provisioned first.

✗ Kubernetes client initialization failed: an error on the server ("") has prevented the request from succeeding

But tk install --export works fine without k8s anyway.

What I expect is that tk create sub-commands with --export would work without K8s.

Support Helm release operations

Add support for Helm releases to tk:

• create
• delete
• get
• export
• reconcile
• suspend
• resume

Change default branch to main

Change the default branch from master to main and delete master for:

• source-controller
• kustomize-controller
• helm-controller
• notification-controller
• image-reflector-controller
• image-automation-controller
• toolkit

Replace openssh keygen/scan with go crypto

We should generate the SSH private/public keys and host keys with crypto/ssh instead of relaying on ssh-keygen and ssh-keyscan. This would trim down the CLI dependencies and make it more accessible.

Bootstrapping clusters to the same path

What is the problem?

Currently looking at managing multiple clusters with the same base line of daemonsets, ingress controllers, etc. I was trying to bootstrap them all to the same path. The issue I am running into is the deploy key is set as tk-<path> this will purge the last clusters deploy key. I was hoping we can potentially be able to change the name of the key so we can bootstrap multiple clusters on github to the same path.

An issue with the gitops-system reconsiliation

All the resources in gitops-system namespace are unexpectedly deleted when I add an additional YAML file into fleet-infra\staging-cluster\ created by the bootstrap process.

Steps to reproduce:

1. Deploy resources using the bootstrap gotk bootstrap github --owner=$GITHUB_USER --repository=fleet-infra --path=staging-cluster --personal --arch=arm64. I'm using --arch=arm64 but I don't think it should affect anything
2. Pull the created repository and add a resource description into fleet-infra\staging-cluster, a namespace definition for example. I'm using 01-namespace.yml containing

apiVersion: v1
kind: Namespace
metadata:
  name: lens-metrics-2

So that the structure looks like this:

├── staging-cluster 
│   └── gitops-system 
│   │  ├── toolkit-components.yaml
│   │  ├── toolkit-kustomization.yaml
│   │  └── toolkit-source.yaml
│   └── lens-metrics-2
│       └── 01-namespace.yml 

4. Commit the changes and push.
5. Observe that pods and all other resources in gitops-system (apart of GitRepository and Kustomization definitions) are deleted and never come back again.

When I delete finalizers from the remaining GitRepository and Kustomization CRDs and they finally go away I'm able to execute the bootstrap command from step two and all the resources come back including the namespace I added at step 3.

Support Helm repository operations

Add support for Helm repositories to tk:

• create
• delete
• export
• get
• reconcile #87