typeable / octopod

Kubernetes self-service portal

Octopod is a fully open-source self-service portal which empowers your team to leverage the full power of Helm without the need for technical expertise.

Octopod is a way for you to provide a self-service portal for managing instances of a service in your cluster. Anything you can describe in a Helm Chart, any member of team can deploy. No matter how complex the deployment process is.

The recipe is simple: you provide a url to your Helm repo, Octopod then automatically extracts the possible configuration that the Chart can have, and automatically manages the lifecycle of the services. What users see is just a simple web interface:

🖥 Demo

🧑‍🔬 Try it out for yourself

You can have the installation from the demo above running on your local machine in minutes by running the following command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/typeable/octopod/master/octopod_local_install.sh)"

You can dive deeper into the installation process in our deployment guide.

You can remove everything the command above installs by running the following command:

kind delete cluster --name octopod

Motivation

We created Octopod because we believe that everything we release should be rigorously tested, however, such desires greatly complicate the development workflow leading to longer release cycles.

The members of our team that facilitate rigorously testing are also the member that require the most attention from our DevOps engineers to manage the many staging environments required for a proper testing cycle.

With Octopod we were able to reduce the reliance on our DevOps teams for testing to practically zero, while increasing the DevOps maneuverability of our QA engineers and process.

📑 Documentation

🔭 High-level notes

• 🐙 Overview
• 🧑‍🔬 Project management case study
• 🧑‍💻 Technical case study

🛠️ Technical documentation

• 🏗 Technical architecture
• ⚙️ Control script guide
• 🔧🐙 Octopod deployment guide
• 🔧🚀 Helm-based Octopod project setup
• 🐙🎛 octo CLI user guide
• 🤖 CI integration
• 🔒 Octopod security model

ℹ️ FAQ

How long does it take to set up Octopod?

Deploying Octopod itself can be done in a matter of minutes if you have a Kubernetes cluster at hand. If you want to deploy it locally we have a handy script that sets everything up automatically.

After you have Octopod up and running the only thing you need is a Helm Chart that your team might want to deploy. That's it.

Will Octopod work with my project if it uses X?

Yes. Octopod is project-agnostic. If you can run your project in a Docker container, then you can use Octopod with that project.

Does Octopod work with "vanilla" Kubernetes projects that don't use Helm?

If you do not have Helm Charts for your projects, you will not be able to leverage the simple Octopod installation procedure we have developed.

To use Octopod with your project will need to either:

• Write Helm Charts for your project
• Write a set of control scripts that deploy "vanilla" Kubernetes services

You can always contact us if you need help. A good first place to ask questions is Octopod Discussions.

What do I need to know to set up Octopod?

You need to understand the basics of Kubernetes and be familiar with whatever hosting provider you will be using.

What separates Octopod from exiting solutions like Gitlab Dynamic Environments?

Most existing solutions like Gitlab Dynamic Environments offer you an extension of CI/CD pipelines. All they do is allow you to run bash scripts.

This has several drawback the main of which is the fact that you need to understand the under-the-hood details of the deployment process to use them. Existing solutions don't abstract the deployment process. They package up steps of deployment. The user is still responsible for coordinating the steps. You still need to manually manage the state of your deployed services.

Observability and transparency of the deployment management process is another drawback. To inspect the state of a deployed system you need to drop down to the Kubernetes level. At that point you alienate anyone without DevOps expertise. This is the problem Octopod solves.

Octopod actually abstracts the under-the-hood detail like Helm, Kubernetes and Docker from the user into an intuitive and easy to understand model. This allows anyone to manage and deploy services.

Octopod is great, but my service deployment lifecycle is slightly different.

Octopod was developed in a modular way from the very start. It offers specific extension points called control scripts that allow you to modify the behavior of Octopod. The control scripts can be implemented in any programming language, even Bash. You can read more about control scripts in our control script guide.

We have developed a generic set of control scripts that should work great with any Helm Chart. You can use them as a base for your custom control scripts.

Does Octopod work with my CI?

Yes. If you can run arbitrary executables in your CI, then you will be able to integrate it with Octopod. Integration basically consists of calling our octo CLI tool to perform desired actions. You can find more detail in the CI integration doc.

How come I can't see the deployment logs in Octopod web app?

It's been excluded from the GUI because we don't have a good security story to accompany this feature yet. Some secrets and credentials may leak to the project team using Octopod and, potentially, not everyone should have access to this data.

Octopod says "Failure" next to my deployment. Why is it not working?

There are several places where things can go wrong:

1. Your DevOps engineer could have made mistakes in one of the Control Scripts leading to an improper deployment process.

   To fix this the DevOps engineer can use the octo CLI to see detailed logs collected during the deployment process and diagnose the issue.

2. You could have one of the steps of your deployment failing (for example, a failing database migration).

   If you have supplied a Kubernetes Dashboard URL to Octopod during deployment, then you will have a "Details" button in the Web UI. You can click that button to get a filtered view of the Kubernetes Dashboard where you can see what could have gone wrong and diagnose the issue.

Quotations

Typeable team envisioned an insightful approach to CI/CD, which managed to help us cut the delivery time for new features and projects with Octopod

— Wes Iwanski – VP Technology, Downtown Travel

❓ Still have questions?

If you still have questions, be sure to ask them in our Octopod Discussions.

Star the project of you like it

octopod's Issues

Octopod UI: poor perfomance of the deployment editing window

Under certain conditions (like a lot of overrides is displayed) editing window is too slow. It takes seconds for new override field to show up.

octo CLI: Errors aren't meaningful enough

Case:
execute octo list with wrong authentication header
Actual behavior:
in the result you'll get just the word error:
Expected:
Error message is telling about auth problem

Also note that octo info command will throw an error in this case, but it's hard to read and reason about, because a full request/response is being logged. It's must be a simple error message.

Handle kubectl errors in the Octopod local install script

If something went wrong with the Kind cluster errors could be very confusing, like

==> waiting for ingress-controller to become ready...
Unable to connect to the server: EOF
/bin/bash: line 93: [: ==: unary operator expected
==> waiting for ingress-controller to become ready...
Unable to connect to the server: EOF
/bin/bash: line 93: [: ==: unary operator expected

Control-scripts: Simplify scaling+checking of kubernetes resources

We need to simplify addition of a new k8s resources to control scripts.
I propose the following changes:

• use jsonpatch instead of merge
• make generic function to patch anything using jsonpath parameters
• make generic function to check any resource by some parametrized status field

More feedback from config_check script

Add more feedback from config_check script.
Right now it's only prints error if message isn't available in the registry(sometimes doesn't). It must be much more messages for user in cases of broken overrides etc.
Without it empty red window is displayed which is confusing.

Fix weird sorting by default

newer stagings are shown at the bottom for some reason

Remove tag from deployment

The tag of the deployment should be just an ordinary deployment override.

Open the deployment page after creating a deployment

Warn users about duplicate override keys

Same as #77 but should check for duplicate keys. Duplicates are currently just silently dropped.

Pass maximum information to all control scripts

Right now we only pass the arguments to control scripts that we actually needed in practice. This is not generic enough. We already ran into issues where we needed more information in control scripts.

The task is to pass all available information (that is passed to other scripts) to all control scripts. This probably involves getting rid of the per-script argument generators that we have now.

Use proper CSV parser for info script

Currently, we have a very limited hand-written parser that fails with error when something goes wrong:

octopod/octopod-backend/src/Types.hs

Lines 30 to 40 in 36b22c3

	parseDeploymentMetadata :: [Text] -> IO [DeploymentMetadata]
	parseDeploymentMetadata texts =
	for texts $ \t ->
	case T.findIndex (== ',') t of
	Just i -> do
	let (key, value) = bimap strip (T.tail . strip) $ T.splitAt i t
	pure $ DeploymentMetadata key value
	Nothing ->
	error $
	"Malformed metadata key-value pair " <> T.unpack t
	<> ", should be similar to foo,bar"

The parser is currently only used to parse the output of the info control script, but more scripts that output CSV-formatted data will be added, so the task is to replace the hand-written parser with a library that implements the proper CSV format. (and preferable produce haskell data structures, and not just Maps.

Show loading deployment state in deployment list

When you open the list of deployments, after they are downloaded, there is a split second when the header of the table is shown, but not the content.

We do have untilReady primitive. If we stick it in this function, then we can show a widget while the row is actually loading:

octopod/octopod-frontend/src/Page/Deployments.hs

Line 257 in 4a5ad45

activeDeploymentWidget hReq clickedEv dDyn' = do

To do this we would probably need some "loading deployment row" widget designed (we don't have one right now). Just sticking a spinner in there doesn't seem good enough to me. Having a table of 15 spinners seems silly to me.

---

I wonder if it is possible to still show the general "loading" widget while the rows are being loaded. But to me, it is not obvious at all how to achieve this. We would somehow need to extract the "Reflex-related" layout computation out of where it is actually laid out.

The general "loading" widget is currently displayed here:

octopod/octopod-frontend/src/Page/Deployments.hs

Line 62 in 4a5ad45

widgetHold_ loadingDeploymentsWidget $

Implement proper logging

Our current logging situation is sad. We only have two logging levels. The logging verbosity is not configurable. logging is tedious.

I would suggest using co-log.

The task is to refactor logging:

1. Make logging be as simple as calling a single monadic action. (Right now we have to manually get the logger from the reader, and pass it to the logging function.)
2. Add more logging levels (severity): Info, Debug, Warning, Error (or analogous)
3. Make the output severity configurable via CLI (log filtering)
4. Scan through existing logging and reevaluate their severity using your better judgment

Refactor effects

Refactor Sevant endpoints

Servant endpoints are currently very difficult to change and use. Types errors are hard to understand and producing client functions from the API types is cumbersome.

I propose we use Servant.API.Generic to describe APIs with records. In my experience, it makes APIs a lot easier to work with.

Create new override design markup

Basically, we need to implement the designs from #79, #80, and #81.

1. #79

   • On the "Edit/Create Deployment" page: https://www.figma.com/file/RSiu3eq4NFLoZdkJeJCShJ/Octopod-UI?node-id=994%3A1265

     The button "add an override" is now shown at the top, and new overrides are also added to the top.

     

   • On the "Edit/Create Deployment" page: https://www.figma.com/file/RSiu3eq4NFLoZdkJeJCShJ/Octopod-UI?node-id=994%3A1265

     In the current implementation, we do not differentiate override states. In the new implementation the override fields can have the following states:

     1. Default – the user did not change the override at all – displayed as thin and italic
     2. Edited override value – the user either changed the default value of the override or created a completely custom value – bold
     3. Deleted override – the overriding way specified by default, but the user deleted it – shown as a faded and strikethrough version of the "Default" override
     4. Default non-editable override. This state is not shown explicitly in the design but is present in the UI Kit (https://www.figma.com/file/RSiu3eq4NFLoZdkJeJCShJ/Octopod-UI?node-id=997%3A229). This style needs to have the "Default" text style. It is used to show the user an override key that is specified by default and can not be edited.

     
     

   • On the "All Deployments" page: https://www.figma.com/file/RSiu3eq4NFLoZdkJeJCShJ/Octopod-UI?node-id=994%3A1403

     The following states for overrides should be added (the same as on the "Edit/Create Deployment" page):

     1. Default
     2. Edited

     

   • On the "Deployment" page: https://www.figma.com/file/RSiu3eq4NFLoZdkJeJCShJ/Octopod-UI?node-id=994%3A1749

     The following states for overrides should be added (the same as on the "Edit/Create Deployment" page):

     1. Default
     2. Edited

     

   • On the "Edit/Create Deployment" page, override key field: https://www.figma.com/file/RSiu3eq4NFLoZdkJeJCShJ/Octopod-UI?node-id=989%3A397

     We are adding an override key suggestion popup that appears when the user enters an override key text field. It is filtered down based on the text the user enters. The matched parts of the text should be in bold. Make sure the popup is scrollable and does not go off-page.

     

   • On the "Edit/Create Deployment" page: https://www.figma.com/file/RSiu3eq4NFLoZdkJeJCShJ/Octopod-UI?node-id=989%3A679

     For technical reasons, the two sets of overrides now need to be loaded separately, and the loading can take some time to complete. Also, for technical reasons, some of the overrides are preloaded. We show the preloaded overrides and let the user interact with them while we load the rest.

     The loading process is displayed as three additional "block" overrides with spinners:

     

2. #80

   • On the "Create/Edit Deployment" page: https://www.figma.com/file/RSiu3eq4NFLoZdkJeJCShJ/Octopod-UI?node-id=983%3A148

     The "Save" procedure can now take a considerable amount of time to run. For this reason, we need to add a "loading" state to the button.

     

3. #81

   • On the "Create/Edit Deployment" page: https://www.figma.com/file/RSiu3eq4NFLoZdkJeJCShJ/Octopod-UI?node-id=994%3A1908

     We now show additional errors next to the key text field. (I think all of the styling needed already exists in the current version.)

     

Design error states in deployment overrides

There are two error states deployment keys can have. They both need to be designed:

• Duplicate key values. (Duplicate values are not allowed)
• Empty key values. (Empty key values are not allowed)

Keyboard navigation for override key search

Submit create/edit deployment form when Enter is pressed

Update helm repos

Currently after loading a helm repo in the control scripts, the repo is never updated unless the server container is restarted.

We probably need some form of cronjob that updates the loaded repos.

Document new control scripts

document control scripts added in #88 and #89

Make high numbers of overrides display better in Web UI

Test control scripts with private registry

We need to test how contol scripts works wit private registries, especially with the ones with authorization.
Cases:

• Harbor with authentication. Map pull secret to Octopod.
• Docker v2 registry with basic auth. Same as above just mount secret with docker.json

Warn user about empty override keys

Prevent the user from submitting overrides with empty keys. Should show an error and highlight the offending overrides.

The validation should probably also be applied on the backend.

Cache default overrides

The "default overrides" and "keys" control scripts can potentially take a while to run and the UI depends on them. We should introduce a caching mechanism in the backend to speed up the queries.'

See #73 for more detail.

tag_check -> overrides_check

Since in #75 we make tag be just a normal deployment override and to allow more generic validation capabilities we should change the tag_check script to be an overrides_check script which receives the full set of user override edits applied to the default override values.

The exit code should still signal success.

All of the stdout of the overrides_check should be passed to the frontend to display to the user if there was an error.

Remove spellcheck from config fields

UI: App Overrides list is not visible

App Overrides list is going outside of a screen border and you need to scroll to see it.

Add /ping to get version information after deploying.

No way to remove deployment in "cleanup failed" state

If cleanup script has failed in non recoverable fashion and deployment got to the 'cleanup failed' state there is not way to delete it.
There must be --force flag in the cleanup command, which will delete all traces of deployment from database ignoring exit code from cleanup control script

Reimplement header auth

Currently, we abuse the servant-auth library to provide authentication for octo CLI.

The authentication is very simple: we just check the Authorization header in every request for equality.

Using servant-auth for this is overkill and the library doesn't handle the use case very well – it expects the payload to be JSON-encoded, etc.

Additionally, it has a large dependency footprint that we can avoid.

octopod/octopod-api/src/Octopod/PowerAPI.hs

Line 77 in 90fa7b8

data AuthHeaderAuth

octopod/octopod-api/src/Octopod/PowerAPI.hs

Line 49 in 90fa7b8

Auth '[AuthHeaderAuth] () :> "api" :> "v1"

octopod/octopod-backend/src/Octopod/PowerAPI/Auth/Server.hs

Lines 14 to 23 in 90fa7b8

	newtype AuthHeader = AuthHeader ByteString
	instance IsAuth AuthHeaderAuth () where
	type AuthArgs AuthHeaderAuth = '[AuthHeader]
	runAuth _ _ (AuthHeader h) = AuthCheck $ \req ->
	pure $ case lookup "Authorization" $ requestHeaders req of
	Just v | v == h -> Authenticated ()
	_ -> Indefinite
	deriving anyclass instance ToJWT ()

octopod/octo-cli/src/Octopod/PowerAPI/Auth/Client.hs

Lines 15 to 28 in 90fa7b8

	class IsAuth a where
	data AuthContext a
	applyAuth :: AuthContext a -> Request -> Request
	instance (IsAuth a, HasClient m api) => HasClient m (Auth '[a] x :> api) where
	type Client m (Auth '[a] x :> api) = AuthContext a -> Client m api
	clientWithRoute pm Proxy req val =
	clientWithRoute pm (Proxy :: Proxy api) (applyAuth val req)
	hoistClientMonad pm _ f cl = hoistClientMonad pm (Proxy :: Proxy api) f . cl
	instance IsAuth AuthHeaderAuth where
	data AuthContext AuthHeaderAuth = AuthHeaderAuthCtx String
	applyAuth (AuthHeaderAuthCtx h) req = addHeader "Authorization" h req

---

The task is to remove the servant-auth dependency while keeping the current behavior. The functionality seems simple enough that we can implement it without any additional dependencies. But, if an existing library arises that fits the task well, I am not against using it.

Design loading animations for create/edit deployment screens

We have run into an issue where previously fast requests can now take noticeable time to complete. We need to design a loading screen:
After clicking the "Save" button on "Create deployment" and "Edit deployment" screens:

Duplicate key values. (Duplicate values are not allowed)

Refactor calling control scripts

We need a unified way of calling control scripts. I would strongly suggest using typed-process.

The way of calling control scripts should automatically handle errors in the following way:

• if the control script exited with code 0 and if the current logging level is DEBUG then output the script call, script output, and exit code to logs.
• if the control script exited with code other than 0 and if the current logging level is no more restrictive than WARN then output all of the same things

Improve archiving

Currently, we delete the helm deployment when archiving. This can cause problems like generating new different passwords.

A better way of doing archiving might be scaling the deployment to 0 replicas.

To do this we need to:

• Change the archive control script to scale to 0 instead of delete
• Create an unarchive script that scales to a non-zero number of replicas
• Call the unarchive script from the Octopod backend.

Redesign overrides frontend

Currently the overrides screen looks like this:

The new functionality added to the overrides is:

1. We now have default key-value pairs that need to be displayed to the user. The user can edit the default values (making them not default anymore), he can delete either existing or custom values (the deleted default overrides still need to be indicated somehow with the ability to add them back in), and the user can add completely new overrides.
   The designs we need:

   • default override
   • edited default
   • deleted default override
   • custom new override

2. We now also prove a list of keys that can be used (in addition to having the ability to add custom keys). I think it might make sense to show the default keys as hints or autocomplete.
   The design we need:

   • Key hints when adding new overrides

3. The default overrides can take a while to load, so we need to add a loading animation. The "deployment" and "application" overrides are loaded separately, so their loading state should also be separate. This will probably not influence the design, but changing deployment overrides will force us to relax the application overrides.
   The design we need:

   • loading state for the set of overrides

4. The same changes will affect the "All deployments" and "Deployment" screens. They currently look like this:

   

   For compactness reasons on the "All deployments" screen, we don't want to show the unchanged default key-value pairs. We only want to show the edits the user made (deleted a default value, edited a default value, added new values)

   

   On the "Deployment" screen we want to show the full set of key-value pairs: the default values with all of the edits applied. We want to show the exact same information as discussed in point 1.

   The design we need:

   • On "All deployments" screen:
     • edited default on
     • deleted default override
     • custom new override
   • On "Deployment" screen:
     • default override
     • edited default
     • deleted default override
     • custom new override

Add override control scripts

Add the following control scripts:

1. default deployment overrides
2. keys for deployment overrides
3. default application overrides – should take final deployment overrides as one of the arguments
4. key for app overrides – should take final deployment overrides as one of the arguments

The output should be read from stdout in the CSV format.

The default arguments should be displayed to the user in the web UI. The user can then choose to edit or delete existing overrides or add new overrides. The structure for the override edits was introduced in #68 although not fully integrated (they are currently used as the old override system).

We should only store the edits the user made. To get the full config we would run the "default overrides" script again and apply the saved user edits on top.

The keys should just be passed to the frontend and used as suggestions to the user.

Stream control script output

Some control scripts can take a while to load, and debugging them can be difficult if we only get the output after they finish running.

It would make debugging control scripts a lot easier if we could observe the output of control scripts before they finish running.

Integrate Helm Chart into our release procedure

We have the helm charts published in the gh-pages branch. It is updated manually and out of sync with the release of Octopod itself.

We need to either:

1. Integrate the helm chart release into our release checklist: https://github.com/typeable/octopod/blob/master/Release_checklist.md
2. Or (more preferable), automate the whole Helm Chart release in our release job:
   

   octopod/.github/workflows/build.yaml

   Lines 109 to 152 in 43d6f53

   	release:
   	name: Release
   	if: github.event_name == 'workflow_dispatch' && github.event.inputs.release == 'true'
   	needs: [octo-macOS, octo-linux, docker-images]
   	runs-on: "ubuntu-latest"
   	steps:
   	- uses: actions/checkout@v2
   	- name: Common setup
   	uses: ./.github/actions/common_setup
   	with:
   	cachixSigningKey: ${{ secrets.CACHIX_SIGNING_KEY }}
   	- name: Push Docker Release Images to DockerHub
   	uses: ./.github/actions/push_docker_images
   	with:
   	tag: ${{ github.event.inputs.version }}
   	docker_username: ${{ secrets.DOCKERHUB_USERNAME }}
   	docker_password: ${{ secrets.DOCKERHUB_TOKEN }}
   	- name: Push Docker latest Images to DockerHub
   	uses: ./.github/actions/push_docker_images
   	with:
   	tag: latest
   	docker_username: ${{ secrets.DOCKERHUB_USERNAME }}
   	docker_password: ${{ secrets.DOCKERHUB_TOKEN }}
   	- uses: actions/download-artifact@v2
   	- name: Zip and move
   	run: |
   	chmod +x octo-cli-macos/octo
   	zip octo-cli-macos octo-cli-macos/octo
   	chmod +x octo-cli-linux/octo
   	zip octo-cli-linux octo-cli-linux/octo
   	mv octo-cli-image/* .
   	mv octopod-server-image/* .
   	- uses: "marvinpinto/action-automatic-releases@latest"
   	with:
   	repo_token: "${{ secrets.GITHUB_TOKEN }}"
   	automatic_release_tag: "latest"
   	prerelease: false
   	title: "${{ github.event.inputs.version }}"
   	files: |
   	*.zip
   	*.tar.gz

Document control script release procedure

The task is to fill in this TODO:

octopod/Release_checklist.md

Lines 4 to 5 in 43d6f53

	2. If there were changes to control scripts:
	1. _**\<TODO\>**_

Ideally, we would integrate the control script release into the release GitHub Action job:

octopod/.github/workflows/build.yaml

Lines 109 to 152 in 43d6f53

	release:
	name: Release
	if: github.event_name == 'workflow_dispatch' && github.event.inputs.release == 'true'
	needs: [octo-macOS, octo-linux, docker-images]
	runs-on: "ubuntu-latest"
	steps:
	- uses: actions/checkout@v2
	- name: Common setup
	uses: ./.github/actions/common_setup
	with:
	cachixSigningKey: ${{ secrets.CACHIX_SIGNING_KEY }}
	- name: Push Docker Release Images to DockerHub
	uses: ./.github/actions/push_docker_images
	with:
	tag: ${{ github.event.inputs.version }}
	docker_username: ${{ secrets.DOCKERHUB_USERNAME }}
	docker_password: ${{ secrets.DOCKERHUB_TOKEN }}
	- name: Push Docker latest Images to DockerHub
	uses: ./.github/actions/push_docker_images
	with:
	tag: latest
	docker_username: ${{ secrets.DOCKERHUB_USERNAME }}
	docker_password: ${{ secrets.DOCKERHUB_TOKEN }}
	- uses: actions/download-artifact@v2
	- name: Zip and move
	run: |
	chmod +x octo-cli-macos/octo
	zip octo-cli-macos octo-cli-macos/octo
	chmod +x octo-cli-linux/octo
	zip octo-cli-linux octo-cli-linux/octo
	mv octo-cli-image/* .
	mv octopod-server-image/* .
	- uses: "marvinpinto/action-automatic-releases@latest"
	with:
	repo_token: "${{ secrets.GITHUB_TOKEN }}"
	automatic_release_tag: "latest"
	prerelease: false
	title: "${{ github.event.inputs.version }}"
	files: |
	*.zip
	*.tar.gz

Validate override keys

see #94 for more detail