hasura / gitkube

Build and deploy docker images to Kubernetes using git push

Gitkube

Gitkube is a tool for building and deploying Docker images on Kubernetes using git push.

After a simple initial setup, users can simply keep git push-ing their repos to build and deploy to Kubernetes automatically.

When should I use gitkube?

1. Ideal for development where you can push your WIP branch to the cluster to test.
2. Reference implementation for writing git-based automation on your server. Fork this repo and create your own CRD + controller + git remote hook that can do things on the Kubernetes cluster.

Features:

• No dependencies except native tooling (git, kubectl)
• Plug and play installation
• Simple public key based authentication
• RBAC ready - Control access to git remotes using RBAC
• Support for namespace based multi-tenancy - Remotes can only deploy to their own namespace
• No assumptions about repository structure

Getting started

Gitkube will run on any Kubernetes vendor/distribution AS IS. In case you find any difficulties in the setup, please comment on #33

Install gitkube

Using kubectl

kubectl create -f https://raw.githubusercontent.com/hasura/gitkube/master/manifests/gitkube-setup.yaml

#expose gitkubed service
kubectl --namespace kube-system expose deployment gitkubed --type=LoadBalancer --name=gitkubed

Using gitkube CLI (DEPRECATED)

1. Install Gitkube CLI:

   • Linux/MacOS

   curl https://raw.githubusercontent.com/hasura/gitkube/master/gimme.sh | bash

   • Windows (using scoop)

   scoop install gitkube

   Or download the latest release and add it to your PATH.

2. Use Gitkube CLI to install Gitkube on the cluster:

   gitkube install

Workflow

• Local dev: User creates a base git repo for the application with Dockerfile and K8s deployment
• Setting Remote: User defines a spec for Remote containing the rules for git push
• Deploying application: Once a Remote is setup, application can be deployed to K8s using git push <remote> master

Local dev

User should have a git repo with source code and a Dockerfile. User should also create a base K8s deployment for the application.

Setting Remote

A Remote resource consists of 3 parts:

1. authorizedKeys: List of ssh-keys for authorizing git push.
2. registry: Details of docker registry where images are pushed post-build.
3. deployments: Spec for building docker image and updating corresponding K8s deployment.

Here is a typical spec for a Remote:

apiVersion: gitkube.sh/v1alpha1
kind: Remote
metadata:
  name: sampleremote
  namespace: default
spec:

# Insert ssh-keys for allowing users to git push
  authorizedKeys:
  - "ssh-rsa your-ssh-public-key"

# Provide registry details: https://github.com/hasura/gitkube/blob/master/docs/registry.md
  registry:
    url: "docker.io/user"
    credentials:
      secretRef: regsecret                # Name of docker-registry secret

# Define deployment rules
  deployments:
  - name: www                             # Name of K8s deployment which is updated on git push
    containers: 
    - name: www                           # Name of container in the deployment which is built during git push
      path: example/www                   # Docker build context path in the git repo
      dockerfile: example/www/Dockerfile  # Location of Dockerfile for the source code

Deploying application

Once a Remote is created, it gets a git remote URL which you can find in its status spec

$ kubectl get remote sampleremote -o yaml
...
status:
  remoteUrl: ssh://[email protected]/~/git/default-sampleremote
  remoteUrlDesc: ""

Add the generated remoteUrl in git

$ git remote add sampleremote ssh://[email protected]/~/git/default-sampleremote

And finally, git push

$ git push sampleremote master

More examples

Follow this example repo for more workflows with gitkube.

How it works

Gitkube has three components:

1. Remote: Custom resource defined by a K8s CRD
2. gitkube-controller: Controller that manages Remote objects and propogates changes to gitkubed
3. gitkubed: Git host that builds docker image from the repo and rolls out deployment

High-level architecture

Contributing

Gitkube is an open source project licensed under Apache License 2.0

Contributions are welcome.

Community and Support

• Join the Gitkube channel in the Kubernetes Slack group

• 

Maintainers

This project has come out of the work at hasura.io. Current maintainers @Tirumarai, @shahidh_k.

Follow @gitkube to stay updated.

Gitkube logo concept and design by Samudra Gupta.

gitkube's Issues

Build images with genuinetools/img

It will be awesome of gitkube can integrate with something like img so that images can be built without mounting the docker sock.

ref:
https://github.com/genuinetools/img
https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/

There are still some issue to be merged/closed for this to happen, mentioned in the blog above ^

Is there a way to only build docker image, no deployment?

Hi,

I'm new to gitkube. I'm trying to find a way to build docker images and push it to private registry only, without deploying pods in k8s cluster. Is there an option for that or is it even possible?

Cannot generate multiple gitkube remotes on a given cluster using gitkube cli.

Because it errors out on creating an already existing docker-registry secret.

FATA[0036] generating remote failed: error creating docker-registry secret: secrets "regsecret" already exists

e2e teardown should not delete crd and clusterrolebinding

Tests should not affect cluster level resources.

Templating for git hooks

Currently, pre-receive hook is fixed: https://github.com/hasura/gitkube/blob/master/build/gitkubed/pre_receive.sh

We want to make it scriptable to enable : #20

Each hook probably needs variables from the remote like DEPLOYMENTS, REGISTRY_PREFIX.
For pre-receive, we are currently doing a moustache render during startup.

For giving same variables to each hook, we need to run a moustache render on each hook and then place it in the correct directory.

Write godocs

Track latest build tag and patch when image changes for deployments

As mentioned in #71, sometimes a user might need to update the deployment (say kubectl apply) and when it happens, the image name is overwritten.

The controller needs to keep track of latest build tags for deployments in each remote.

When any of these deployments' image field gets updated, controller can patch it to the latest build tag.

So, whenever a new build happends, a relevent key should be updated in Remote.Status. Then controller can watch all the tracked deployments and patch when required.

[cli] generate: ask if deployments are present

To handle multi-repo configurations, prompt if deployments are present or not.

get different version for gitkubed (other than controller)

Schema validation for Remote resource

https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation

Sync gitkubed also

Currently, sync happens on remotes.

For consistency, we should sync gitkubed (service and deployment) also

Helm packaging

Package Gitkube as Helm chart

Support for secret type .dockerconfigjson

First push of gitkube-example on to my local cluster went through fine so I thought of pushing an update and realised that I need to setup the registry to get it working. So, I created a kube secret, with name regsecret, for docker following the instructions on the link in the README of gitkube-example and updated the registry of the example remote to be:

  registry:
    url: "docker.io/sdileep"
    credentials:
      secretKeyRef:
        name: regsecret
        key: .dockercfg

Now when I try to push to the example remote, I get the following error:

remote: WARNING: Error loading config file:/home/default-example/.dockercfg - Invalid Auth config file

So, I updated the registry to be:

  registry:
    url: "docker.io/sdileep"
    credentials:
      secretKeyRef:
        name: regsecret
        key: .dockerconfigjson

.dockerconfigjson field of the secret is the base64 representation of docker credentials - I presumed this is what goes in here.

Now when I try to push to the example remote, I get a timeout. Not sure what the correct config here should be, could you please help? Thanks.

Add instructions on how Docker secret need to be configured

Add instructions on how Docker secret need to be configured in the k8s cluster:

kubectl create secret docker-registry gitkube-secret \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=arungupta \
  --docker-password=<password> \
[email protected]

Instructions for Docker for Desktop

Show how gitkube workflow works with Docker for Desktop

Remove Hasura specific instructions from pre-receive hook

Current pre-receive hook has Hasura specific instructions: https://github.com/hasura/gitkube/blob/master/sshd/pre_receive.sh#L185

Add support for docker-compose

Can we add support for docker-compose via docker stack?
That way we would have:

1. Kubernetes manifests
2. Helm
3. Docker-compose

1, 3 are ideal for developers.

docker stack deploy is pretty much exactly like helm install. Uses a compose.yaml instead of Helm charts.
https://docs.docker.com/engine/reference/commandline/stack_deploy/#description

Upgrade client-go

This will support credentials plugin (useful for EKS authentication):

https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

Info needed on min. node config (at least on GKE)

gitkubed pods are unschedulable on a low config node on GKE (g1-small: 1 vCPU, 1.70 GB ).

Works like a charm on the next available config. Add information about the min. node config needed.

Auto-detect base image to use if no Dockerfile is found

Summary

In layman's terms: It should be able to allow developers to git push to a gitkube-enabled cluster without a Dockerfile - or so called the "Heroku-style" deployment.

Intro

Heroku's architecture is composed of a dedicated builder for the source, where your project requires a Procfile to determine processes to execute, and a declarative app manifest that includes all the required dependencies and postinstall scripts that the builder will run.

Heroku builds an app using an approach called a Stack-Buildpack approach, whereas:

• use a Base image that has all the dev dependencies.
• use a Buildpack: a set of shell scripts that would automate the building and releasing of the code
• trim dev dependencies from the container and release to the runtime environment.

But in the case for gitkube, we do not need to implement the same flow as Heroku since we can take advantage of the rich Docker ecosystem in Kubernetes.

Process flow

Once the remote recieves the source from the push, it should be able to detect the dockerfile and the manifest, but we should look on two situations:

• Situation 1: Dockerfile found and manifest found.
  • Use the current flow to dpeloy app.
• Situation 2: Dockerfile not found but manifest found
  • Try to find manifest (package.json, pom.xml, etc.).
  • Grab image that works for the specific app (package.json is found for example, then use nodejs:latest)
  • then run npm install, and deploy app.
  • run code using npm start or find app.js, index.js, etc.
    • if not found, declare an error and call a back off.

Conclusion

This is not intended to replace how gitkube currently works, rather, it is a enhancement on the current flow, and it allows migrating users from Heroku to their own provisioned Kubernetes cluster.

Feel free to extend this as well, as I only accounted for cases on migrating users from heroku.

Improve e2e test

Currently, the e2e test fails when the following two resources already exist:

Error from server (AlreadyExists): error when creating "STDIN": customresourcedefinitions.apiextensions.k8s.io "remotes.gitkube.sh" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterrolebindings.rbac.authorization.k8s.io "gitkube" already exists

For the CRD, we just need to ignore the error and for clusterrolebindings we can just use the test namespace name.

Add git push support for all apps (daemonset, statefulset)

Support for git push based deployments is currently limited to deployments. This should be extended to:

• daemonset
• statefulset

Add an option to enable local Docker registry

Add an option that allows to setup a local Docker registry in the cluster. This allows a simpler workflow OOTB.

Instructions for different docker registries

Add docs/registry.md with instructions on how to create docker-secret for different registry providers like dockerhub, gcr, quay, etc

Add docs for different types of push use-cases

Push master
Push a test branch
Best way to use this with different kinds of git patterns: https://www.atlassian.com/git/tutorials/comparing-workflows

controller doesn't catch remotes crd being deleted when its running

Who's using Gitkube?

Hi

Gitkube is a new project. We need active feedback. We also want to know who is using gitkube, where it's being used - side projects, inside organizations, some place else?

Please drop in a comment if you are using gitkube and provide any additional details. This will be very helpful!

List out key features, esp. in comparison to similar tools

Features:

• Public key based auth
• No assumptions about directory structure (configure dockerfile path)
• Rollout deployments in any namespace
• Easy to fork and customise
• Super easy to use and configure

Deploy Helm Apps

Requirement

• Given a the deployment model, specify a Helm Application to be deployed.
• "Any" application from https://hub.kubeapps.com/ should be deployed automatically.

Design

• Specify the name of the Helm Chart to deploy
• Identify the type of ingress instance the cluster contains
  • Use default ingress nginx
  • This provides a default ingress to the cloud provider
• Identify if the cluster has tls secret
  • This provides HTTPS access to the service
• Identify if the cluster has external-dns
  • This provides Route53 endpoint to the service

basic e2e test

Remove $ from code blocks

Makes it easier to copy

Deployment Update Flow

Hey

Just wondering what the best workflow is for when i need to update a deployment without wiping out the image? Right now i need to call:

kubectl get deployment app -o yaml

Copy the whole image into into my deployment.yaml file, and then i can do kubectl apply

Basically my question is: is there a better way to handle all of this, and to make it play nice with gitkube?

[gitkubed] Goes to CrashLoopBackOff on Docker for Desktop

start_sshd.sh script needs some modifications due to the difference in ownership of docker.sock on Docker for Mac.

State of Gitkube on Docker for Windows in unknown. Any help in identifying the status is appreciated.

Expose gitkube as NodePort type

I'm trying to experiment with gitkube on a local cluster(not minikube) and hence exposed it as NodePort type service. After creating a remote as per example, I am not able to retrieve the remote URL using $ kubectl get remote example -o json | jq -r '.status.remoteUrl. Am I missing something?

ps: the example worked on GCE without any issues.

Instructions for minikube

Add docs/minikube.md for instructions specific to minikube

Slow the animated gif a little.

Its useful, but there are some parts that move very quickly. Perhaps its worth remaking/editing it to run a little slower so its easier to take in each of the steps.

Add docs for monorepo / multirepo organisation structure

Deploy to a different cluster

Currently, gitkube deploys to the same cluster where it is running. We should provide support for deploying it to a different cluster.

[e2e] update tests to use gitkube cli

Tests can use gitkube cli to install and uninstall as well as manage remotes.

List out supported kubernetes vendors or provide instructions for different kubernetes vendors

Mention specifically what kubernetes distros this works with. I've received lots of questions about whether it works on GKE or not.

Is it support AWS EKS ?

Is this support for AWS EKS service?

sshd server signature should be generated and mounted

Add install instructions for various K8s vendors/distributions

Gitkube runs on most Kubernetes distributions/vendors as per the install instructions in README. But in some cases, there may be more configuration required to correctly setup for e.g. exposing the gitkubed service may require configuring a Loadbalancer manually.

We need to document the instructions (docs/install.md) covering various vendors like GKE, Rancher, kops based installations, etc. In most cases, the instructions should be default instructions as provided in the README and in other cases the additional steps should be mentioned.

Please use this issue to mention any distribution which is not covered by the installation instructions.

Docs on how to update

Now that 0.2.0 is out, how do people upgrade?

I just uninstalled it and installed again, but that resulted in my LB ip changing.

Integrate with ECR

ECR currently does not automatically create repositories on push unlike docker hub. How (where) should we handle this logic ?

Support integration for other git hooks

Currently, only pre-push and pre-receive hooks are supported. This should be extended to other hooks like post-push, etc.

change naming schema for the username and group

Hi,

It seems like naming a remote is subject to certain limitations as groupadd is executed (potentially 32chars?).

Example

apiVersion: gitkube.sh/v1alpha1
kind: Remote
metadata:
  name: app-remote
  namespace: my-really-really-really-long-ns

k logs po/gitkubed
Setting up git remotes
Configuring : my-really-really-really-long-ns-app-remote
Adding user `my-really-really-really-long-ns-app-remote' ...
Adding new group `my-really-really-really-long-ns-app-remote' (1000) ...
groupadd: 'my-really-really-really-long-ns-app-remote' is not a valid group name
adduser: `/usr/sbin/groupadd -g 1000 my-really-really-really-long-ns-app-remote' returned error code 3. Exiting.

Is there any way to overwrite the computed my-really-really-really-long-ns-app-remote identifier so groupadd can run through?

add sshd with dockerfile

ImageInspectError on installing gitkube (gitkube install) on minikube

Environment ?

minikube version
minikube version: v0.26.1

$ gitkube version
v0.2.0

What did I do?

gitkube install

What I Expected to see?

Gitkube running peacefully

What did I see instead?

$ k get po -n kube-system 
NAME                                    READY     STATUS              RESTARTS   AGE
default-http-backend-ddb2s              1/1       Running             2          2d
etcd-minikube                           1/1       Running             0          26m
gitkube-controller-cc586fcb-zfk8b       0/1       ImageInspectError   0          23m
gitkubed-6dc69968c5-zwqg5               1/1       Running             0          23m
kube-addon-manager-minikube             1/1       Running             2          2d
kube-apiserver-minikube                 1/1       Running             0          26m
kube-controller-manager-minikube        1/1       Running             0          26m
kube-dns-86f4d74b45-9qfxb               3/3       Running             7          2d
kube-proxy-kjncs                        1/1       Running             0          25m
kube-scheduler-minikube                 1/1       Running             2          2d
kubernetes-dashboard-5498ccf677-cxhrm   1/1       Running             6          2d
nginx-ingress-controller-csqzz          1/1       Running             4          2d
storage-provisioner                     1/1       Running             6          2d
tiller-deploy-5f789bd9f7-zmt8g          1/1       Running             2          2d

$ k describe po gitkube-controller-cc586fcb-zfk8b -n kube-system
....
....
Events:
  Type     Reason                 Age                From               Message
  ----     ------                 ----               ----               -------
  Normal   Scheduled              21m                default-scheduler  Successfully assigned gitkube-controller-cc586fcb-zfk8b to minikube
  Normal   SuccessfulMountVolume  21m                kubelet, minikube  MountVolume.SetUp succeeded for volume "gitkube-token-klm4g"
  Warning  Failed                 6m (x71 over 21m)  kubelet, minikube  Error: ImageInspectError
  Warning  InspectFailed          1m (x93 over 21m)  kubelet, minikube  Failed to inspect image "hasura/gitkube-controller:v0.2.0": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
`

Helm apply does not work on AKS

I am getting this error in AKS but not in GKE.

Error: forwarding ports: error upgrading connection: Upgrade request required

Opened issue with upstream: helm/helm#4254