The policy-admission is a custom admission controller used to enforce a collection of security and administrative policies across our kubernetes clusters. Each of the authorizers (https://github.com/UKHomeOffice/policy-admission/tree/master/pkg/authorize) are enabled individually via the command option --authorizer=NAME:CONFIG_PATH (note if no configuration path is given we use the default configuration for that authorizer).
$ bin/policy-admission --help
NAME:
policy-admission - is a service used to enforce security policy within a cluster
USAGE:
[global options] command [command options] [arguments...]
VERSION:
v0.1.0 (git+sha: fa934ac)
AUTHOR:
Rohith Jayawardene <[email protected]>
COMMANDS:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--listen INTERFACE network interface the service should listen on INTERFACE (default: ":8443") [$LISTEN]
--tls-cert PATH path to a file containing the tls certificate PATH [$TLS_CERT]
--tls-key PATH path to a file containing the tls key PATH [$TLS_KEY]
--authorizer value enable an admission authorizer, the format is name=config_path (i.e images=config.yaml)
--cluster NAME the name of the kubernetes cluster we are running NAME [$KUBE_CLUSTER]
--namespace NAME namespace to create denial events (optional as we can try and discover) NAME (default: "kube-admission") [$KUBE_NAMESPACE]
--slack-webhook URL slack webhook to send the events to URL [$SLACK_WEBHOOK]
--controller-name NAME controller name also used as prefix in annotations NAME (default: "policy-admission.acp.homeoffice.gov.uk") [$CONTROLLER_NAME]
--enable-logging BOOL indicates you wish to log the admission requests for debugging BOOL [$ENABLE_LOGGING]
--enable-metrics BOOL indicates you wish to expose the prometheus metrics BOOL [$ENABLE_METRICS]
--enable-events BOOL indicates you wish to log kubernetes events on denials BOOL [$ENABLE_EVENTS]
--rate-limit DURATION the time duration to attempt to wrap up duplicate events DURATION (default: 1m30s) [$RATE_LIMIT]
--verbose BOOL indicates you wish for verbose logging BOOL [$VERBOSE]
--help, -h show help
--version, -v print the versionNote, the configuration is auto-reloaded, so you can chunk the configuration files in the configmap and on changes the authorizer will automatically pick on the changes.
The admission controller on /metrics on the listening port produce a series of metrics related to request approvals and denial and a breakdown of the latency per authorizer and request. The feature is enabled via --enable-metrics (albeit defaulting to true).
The admission controller along with creating kubernetes events in specified namespace (via the --enable-events command line option) can also publish denial to a slack channel. Simply pass the --slack-webhook or inject the SLACK_WEBHOOK environment variable. This event will detail Kind, Name, Namespace, Username and the denial message in the event.
An authorizer is enabled via the command line switch --authorizer=name=config_file_path i.e. --authorizer=images=/config/images.yml. The configuration as well as the defaults for all of these can be found in the doc.go in each of the authorizer folders. Each of the authorizer's can be configured to ignore certain namespaces.
Initially the project started off with a series of authorizer's however when the scripts authorizer was added most of coded authorizer's could be replaced with a script.
The scripts authorizer (--authorizer=scripts=config) provides an embedded javascript runtime via github.com/robertkrimen/otto. Both the object and namespace it derives is inject into the script as a javascript object. An explain before for an Ingress resource
function isFiltering(o) {
if (o.kind != "Ingress") {
return false
}
annotations = o.metadata.annotations
if (annotations["ingress.kubernetes.io/class"] != "default") {
return false
}
return true
}
if (isFiltering(object)) {
// do some logic
provider = object.metadata.annotations["ingress.kubernetes.io/provider"]
if (provider != "http") {
deny("metadata.annotations[ingress.kubernetes.io/provider]", "you must use a http provider", provider)
}
}You can find a few more examples in the features folder. By default everyone is allowed, if you wish to deny and object, you can call the deny method, passing the field and reason for denial.
Images provides a means to control which container images are permitted to run within the environment. Applied to both the initContainers and containers of any pods which are created. The configuration for authorizer contains a series of regex's, which are taken as the default policy, however it will also read the annotation policy-admission.acp.homeoffice.gov.uk/images on the pod namespace; a comma separated list of regex's which can add on top of the default policies.
apiVersion: v1
kind: Namespace
metadata:
name: test
annotations:
policy-admission.acp.homeoffice.gov.uk/images: ^docker.io/ukhomehomeoffice/.*$, quay.io/ukhomehomeoffice/.*$The domains authorizer provides one a means to control which hostname's / site are permitted via ingress resources and to control those at a namespace level, ensuring pods from another namespace can't take over a URL from another hosted site. Namespaces are annotated with the policy-admission.acp.homeoffice.gov.uk/domains tag, which is a comma separated list of domains this namespace can create ingress resources for. This hostname's themselves may contain a single wildcard i.e. *.example.com
This authorizer is fairly bespoke, it was added as a number of users were getting the configuration wrong and causing the kube cert manager to fail and hit Letsencrypt limits. The authorizer performs a series of checks against a ingress resource which has been labelled to consume certificates from Letencrypt. This is broken down depending on internal or external ingress ELB's.
For internal:
- we ensure it's not trying to use HTTP as the challenge and has selected dns.
- we ensure the domain name is hosted by us and thus kube-cert-manager can add the TXT record.
For External:
- we ensure the ingress is not attached to an internal ELB.
- we ensure the resource if using DNS the zone is hosted by us.
- we ensure if it's using HTTP that the hostname is a CNAME to our ingress ELB.
Services provides a means to control the kubernetes service types a namespace can use. In general we don't want anyone to be able to open NodePorts or LoadBalancer services. The authorizer takes the default configuration which is ClusterIP only and also reads the policy-admission.acp.homeoffice.gov.uk/services annotation from the namespace to see if anything else is permitted.
The values is generic authorizer used to match one or more attributes targeted via jsonpath and regex the values against a regexp. It can be used to enforce certain labels or annotation's i.e. a namespace must have a contact label etc. The configuration is as below
filter-on: Ingress
matches:
- path: metadata.annotations
key-filter: ingress.kubernetes.io/provider
value: ^http$
## OR on a namespace
filter-on: Namespace
matches:
- path: metadata.annotations
key-filter: maintainers
value: ^.*$
required: trueThe current pod tolerations admission gave more headache then features so we combined the enforcement into an authorizer. The behaviors is as such.
- Check the pod tolerations against the default whitelist defined in the configuration.
- If an annotation exists on the namespace, check the pod against the whitelist
The configuration for the authorizer is
// Config is the configuration for the taint authorizer
type Config struct {
// IgnoreNamespaces is list of namespace to
IgnoreNamespaces []string `yaml:"ignored-namespaces" json:"ignored-namespaces"`
// DefaultWhitelist is default whitelist applied to all unless a namespace has one
DefaultWhitelist []core.Toleration `yaml:"default-whitelist" json:"default-whitelist"`
}An example configuration is;
ignored-namespaces:
- kube-admission
- kube-system
- logging
- sysdig-agent
default-whitelist:
- key: node.alpha.kubernetes.io/notReady
operator: '*'
value: '*'
effect: '*'
- key: node.alpha.kubernetes.io/unreachable
operator: '*'
value: '*'
effect: '*'
- key: dedicated
operator: '*'
value: backend
effect: '*'
- key: dedicated
operator: '*'
value: liberal
effect: '*'
- key: dedicated
operator: '*'
value: strict
effect: '*'For the namespace whitelist annotation the tolerations must be specified in json for:
apiVersion: v1
kind: Namespace
metadata:
name: test
annotations:
policy-admission.acp.homeoffice.gov.uk/tolerations: |
[
{
"key": "dedicated",
"operator": "*",
"value": "compute",
"effect": "*"
},
{
"key": "dedicated",
"operator": "*",
"value": "liberal",
"effect": "*"
},
{
"key": "dedicated",
"operator": "*",
"value": "strict",
"effect": "*"
}
]You can alter the controller prefix by using the --controller-name option which sets the prefix for the annotations across all the authorizers.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent