some-natalie / kubernoodles

k8s runners for GitHub Actions in the enterprise, made for humans

Hi there 👋

I'm Natalie, a DevSecOps engineer and consultant experienced in developer experience and consolidation within a wide array of security-focused environments. I work at the intersection of technology, people, and highly-regulated industries as a Principal Federal Solutions Engineer at Chainguard!

📝 I write about tech, what I'm working on, and what I'm playing with on my blog. Here's what I've been up to lately:

1. Reducing CVEs in actions-runner-controller: (Kubernoodles, part 9 of ?) - The same actions-runner-controller you know and love (or curse), but with many fewer CVEs to generate compliance paperwork.
2. GraphQL patterns to know: Handy snippets, tips, and tricks for working in GraphQL I struggled for so you don't have to.
3. Intro to GraphQL using custom fields in GitHub Projects: Getting started with the new GitHub Issues and Projects API, exclusively in GraphQL, doesn't have to be so difficult.

💼 Day to day, I work with

• Linux (mostly RHEL and Ubuntu these days) - fedora-acs-override
• Kubernetes - kubernoodles, self-hosted GitHub Actions runners made for humans
• Software development in highly regulated industries (ITAR, CMMC, NIST 800-171, DoD IL 2-6)
• Python - Advanced Security CSV export
• Go - gh-org-admin-promote
• GitHub Enterprise - enterprise security team, audit and compliance reporting
• but mostly, I work with people on all of the above! 💖

You can find me in our work Slack sharing all sorts of neat things you can do with all that fun stuff and probably find out how I've broken and maybe fixed something too. 😀

👾 I play with

• All sorts of handy Raspberry Pi projects, including
  • Kodi set up on a television for local media (build directions)
  • OpenWRT router (build directions)
  • Pi-hole, for DNS and ad-blocking (build directions)
  • Ubiquiti UniFi network controller, in Docker of course
• I'm getting into the Flipper Zero lately - it's so handy and mischievous! (some fun uses)
• Video games in a Windows VM on my Fedora desktop with libvirt, KVM, and a custom Linux kernel to pass hardware to it. It's got about 5% or so performance drop (just looking at frame rates) over a native install. You should check it out - code and write-up on how it works.

I have an awesome life outside of tech, so while I have a few projects that I enjoy (👆), nothing above is close to where I spend most of my time / energy. If you need anything of mine above fixed, please feel free to fork it and send me a pull request! ❤️

Heads up!

• 🌱 I’m currently studying to sit for my OSCP certification and learning to write better conference talk proposals.
• 🎤 Public speaking is fun! Check out what I've been up to here.
• 😄 Pronouns: she/her
• ❓ Looking for my résumé? It's here, but you can also find some of what I've been up to in my profile. If you want to know about where else I've worked and went to school, you should go to LinkedIn.
• 💬 Want to chat? I'm on Mastodon.

kubernoodles's Issues

Create docs on ingress control

Create some documentation around using an ingress controller.

General outline ideas

• start with HTTP only
• add caching with persistent data volume
• add TLS using lets-encrypt

Bump dependencies and scripts

ARC has some updates to logger, entrypoint, etc. scripts from where I'd left them. Bump these. :)

Add docs about D-in-D architecture

Docker-in-Docker isn't the most fun way to do things, but having docs on why would be valuable

Bump dumb-init

Bump Yelp's dumb-init to the latest version

Lint the dockerfiles

Lint the dockerfiles with hadolint ... clean everything up and ✨

Revisit rootless podman container problems

Problem - podman works, but podman run does not when used by the runner

Idea - use --userns=keep-id at run to keep uid maps

Links that could be helpful

• https://www.redhat.com/sysadmin/rootless-containers-podman
• https://github.com/containers/podman/blob/main/rootless.md
• https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md
• https://docs.podman.io/en/latest/markdown/podman-run.1.html?highlight=userns#userns-mode
• https://www.redhat.com/sysadmin/debug-rootless-podman-mounted-volumes

Logs

##[debug]Evaluating condition for step: 'Container Action test'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Container Action test
##[debug]Loading inputs
##[debug]Loading env
Run ./tests/container
Building docker image
  Dockerfile for action: '/runner/_work/kubernoodles/kubernoodles/./tests/container/Dockerfile'.
  /usr/bin/docker build -t 60e[2](https://github.com/some-natalie/kubernoodles/runs/7346411518?check_suite_focus=true#step:6:2)26:5f9714a018e65ed72405d6d56d722dc4 -f "/runner/_work/kubernoodles/kubernoodles/./tests/container/Dockerfile" "/runner/_work/kubernoodles/kubernoodles/tests/container"
  Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
  Resolved "python" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
  Trying to pull docker.io/library/python:[3](https://github.com/some-natalie/kubernoodles/runs/7346411518?check_suite_focus=true#step:6:3)-slim...
  STEP 1/5: FROM python:3-slim
  Getting image source signatures
  Copying blob sha256:[4](https://github.com/some-natalie/kubernoodles/runs/7346411518?check_suite_focus=true#step:6:4)0c89643d0cd670[5](https://github.com/some-natalie/kubernoodles/runs/7346411518?check_suite_focus=true#step:6:5)484[6](https://github.com/some-natalie/kubernoodles/runs/7346411518?check_suite_focus=true#step:6:6)d3f5[7](https://github.com/some-natalie/kubernoodles/runs/7346411518?check_suite_focus=true#step:6:7)e5f73755ded7c0f951a4c5b9f392c4067[8](https://github.com/some-natalie/kubernoodles/runs/7346411518?check_suite_focus=true#step:6:8)4bf62
  Copying blob sha256:e37ebf440f7f53eb0584605f7c63a[9](https://github.com/some-natalie/kubernoodles/runs/7346411518?check_suite_focus=true#step:6:9)5b42583a1372913da9061b6fdb7b535663
  Copying blob sha256:40c89643d0cd67054846d3f57e5f73755ded7c0f951a4c5b9f392c406784bf62
  Copying blob sha256:e37ebf440f7f53eb0584605f7c63a95b42583a1372913da9061b6fdb7b535663
  Copying blob sha256:461246efe0a75316d99afdbf348f7063b57b0caeee8daab775f1f08152ea36f4
  Copying blob sha256:912bc51860fbe91d759008e764b6db[10](https://github.com/some-natalie/kubernoodles/runs/7346411518?check_suite_focus=true#step:6:11)299b829ec9789fa873ef9f3dced3390b
  Copying blob sha256:461246efe0a75316d99afdbf348f7063b57b0caeee8daab775f1f08152ea36f4
  Copying blob sha256:912bc51860fbe91d759008e764b6db10299b829ec9789fa873ef9f3dced3390b
  Copying blob sha256:07053eece5a202737fa1c0ee49737a22007fad69699cc0953a9ec4276b33ec7c
  Copying blob sha256:07053eece5a202737fa1c0ee49737a22007fad69699cc0953a9ec4276b33ec7c
  Copying config sha256:ba94a8d[11](https://github.com/some-natalie/kubernoodles/runs/7346411518?check_suite_focus=true#step:6:12)761b3d47a0035819c32c0d42a43bc104734c8ce2a303da8d7f6e700
  Writing manifest to image destination
  Storing signatures
  STEP 2/5: COPY test.py /app/test.py
  --> 142a30aafff
  STEP 3/5: WORKDIR /app
  --> d5[12](https://github.com/some-natalie/kubernoodles/runs/7346411518?check_suite_focus=true#step:6:13)516d34e
  STEP 4/5: ENV PYTHONPATH /app
  --> 93e1b2bb5de
  STEP 5/5: CMD ["/app/test.py"]
  COMMIT 60e226:5f9714a018e65ed72405d6d56d722dc4
  --> dca19cf115a
  Successfully tagged localhost/60e226:5f9714a018e65ed72405d6d56d722dc4
  dca19cf115af3b6b38102da6c8fd51922c87ff2e970fa78cd32306c534d05736
/usr/bin/docker run --name e2265f9714a018e65ed72405d6d56d722dc4_d564e3 --label 60e226 --workdir /github/workspace --rm -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e GITHUB_STEP_SUMMARY -e RUNNER_DEBUG -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/runner/_work/_temp/_github_home":"/github/home" -v "/runner/_work/_temp/_github_workflow":"/github/workflow" -v "/runner/_work/_temp/_runner_file_commands":"/github/file_commands" -v "/runner/_work/kubernoodles/kubernoodles":"/github/workspace" 60e226:5f9714a018e65ed72405d6d56d722dc4
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
cannot resolve /github/home: lstat /github: no such file or directory
##[debug]Docker Action run completed with exit code 1
##[debug]Finishing: Container Action test

[NEW RUNNER] - Invalid value: true: Privileged containers are not allowed

Is your feature request related to a problem? Please describe

i tried runner deployment with https://github.com/some-natalie/kubernoodles/blob/main/deployments/ghes/rootless-ubuntu-focal.yml and getting below error

2022-09-21T15:44:16Z ERROR actions-runner-controller.runner Failed to create pod resource {"runner": "actions-runner-system/rootless-ubuntu-focal-mbc22-vgbds", "error": "pods "rootless-ubuntu-focal-mbc22-vgbds" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]"}
github.com/actions-runner-controller/actions-runner-controller/controllers.(*RunnerReconciler).Reconcile

is there any way to run DiD without privileged mode . or any other image / solution

Auto-bump dependencies _within_ Dockerfile

Example - figure out the latest version of Docker Engine or the runner agent, then bump that on a PR

Setup pages to look nice

It's lowest of low effort right now, but a custom domain + let's encrypt + actually putting a bit of effort in would make this much easier to read the docs. :)

Add a RHEL-ish flavor deployment

Add a RHEL-ish flavor deployment - maybe Fedora or UBI based?

Community health files

Create contributing.md and such to finish community health files. According to https://github.com/some-natalie/kubernoodles/community, there's 3 things missing.

• Code of conduct
• Contribution guidelines
• PR template

Add job summaries to each build/test job

Feature

ℹ️ Not yet shipped in GHES

Set up a basic "does it build" test to merge to `main`

Set up a basic "does it build" test to merge to main, to be expanded later

Set up labels for this project

Some of the default issue labels don't really match what I want here. Maybe label things for "tech debt" cleanup, etc.

Move to multi-stage builds

Investigate movement to multi-stage builds to address transient dependency problems.

Rework weekly repo cleanup

Move to actions/delete-package-versions

Remove runner reaper since that's automatic for GHEC now?

Cleanup shell scripts

Make sure shellcheck is clean on all the things!

Set up the super-linter

Set up the super-linter to avoid local linting differences

• Shell
• Docker
• Kubernetes yaml
• Powershell?
• Markdown

[SECURITY] - Cert Manager Mandatory for Openshift 4.X

Describe the problem

A clear and concise description of the problem.

We are trying to setup ARC on Openshift 4.X. Can we install the ARC without using the cert-manager for openshift 4.X?

Move into AKS for demos

Move off my laptop Docker Desktop and into AKS for demos

Needs

• service account creation in AKS
• environment config in GitHub

Reduce Ubuntu image size

2.something GB is a bit big for a default in GHCR ... time to dive

Kata + Firecracker container test

Test buildout and document (if successful) on Kata containers to provide true workload isolation on a STIG baseline

• starting point?

Remove `patched` files

Remove "patched" files if no longer needed

Put together an intro admin guide

Quick overview of implementation, decisions for admins

Set up container scanning

Set up container scanning at build

Add Ubuntu Jammy Jellyfish (22.04 LTS)

Add Ubuntu Jammy Jellyfish (22.04 LTS)

Fix MTU in Rootless Ubuntu Runner

Upstream - actions/actions-runner-controller#1856

Disable automatic runner updates

The runner agent automatically updates itself, which means there's a ton of bandwidth usage on ephemeral pods. Disable this in favor of updating it via the pod images.

[NEW RUNNER] - running DID with privileged: false

Hi,
i was referring to https://github.com/some-natalie/kubernoodles/blob/main/deployments/ghes/rootless-ubuntu-focal.yml and tryign to start Docker inside runner , but our kubernetes cluster policy wont allow running container in prevailed mode .

could you pls suggest any alternative solution / option to run docker or podman with privileged: false option.

2022-09-21T16:07:20Z ERROR Reconciler error {"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "runner": {"name":"rootless-ubuntu-focal-s79gm-nznk2","namespace":"actions-runner-system"}, "namespace": "actions-runner-system", "name": "rootless-ubuntu-focal-s79gm-nznk2", "reconcileID": "b8cc10ad-7d9a-4a2a-ab13-4bef0dda41d7", "error": "pods "rootless-ubuntu-focal-s79gm-nznk2" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]"}

Move to `test` and `latest` tags for repo use

This is a demo repo, so latest is alright, but test should be used for the build/test jobs

Add docs on supply chain management

Add documentation on approaches to supply chain management WRT security, reproducibility, and bandwidth usage

Improve test coverage

Test that the MTU size is set correctly, software expected is installed?

Add rootless Ubuntu runner

Add rootless Ubuntu runner

Delete untagged containers automatically

Delete untagged containers automatically from GHCR

Get off `latest`

tbh, this is mostly about growing up a little bit here

Look into RUNNER_TOOL_CACHE as a read-only fast mount in ARC

Problem - setup-LANGUAGE actions want LANGUAGE-versions, so if not available to the runner, it's going to try (and maybe fail) to get that at each run time from github.com. Copying those files in is delicate, tedious, and makes for gigantic pods.

Potential solution - Try to use a persistent volume claim with readonlymany to address this problem.

Problems with ☝ may include needing a 2-step export (from github.com) and import (into ARC without internet access), still being tedious even with internet access, and does actions/runner support a read-only cache?