TLS with user provided secret doesn't seem to work about charts

[rainest]

Simple answer here is that the Kong config doesn't have the cert set up. It needs something roughly along the lines of https://github.com/Kong/charts/blob/main/charts/kong/README.md#certificates but for gateway.env.admin_ssl_cert and gateway.env.admin_ssl_cert_key.

That sets up the server certificate serving on the gateway, but not verification of the client certificate presented by the controller. IIRC that relies on injecting NGINX directives (similar to step 8 in https://docs.nginx.com/nginx-management-suite/admin-guides/configuration/secure-traffic/, the ssl_client_certificate and ssl_verify_client bits, so gateway.env.nginx_admin_ssl_client_certificate pointed to the mounted CA cert and gateway.env.nginx_admin_ssl_verify_client=on). I don't think there was ever any dedicated kong.conf functionality for it, and the community addition of the controller client certificate args just used that.

Did you have a doc you were working from? I forget what we created for the rollout of the ingress chart, and it doesn't look like there's anything in the chart README.md for it. We probably want to just inject it automatically in the long run, but can't really with an umbrella chart--one or the other won't be able to reference generated resources easily.

In classical very backwards fashion I pulled this out by decrypting the comms between the two, which was probably unnecessary (coulda just hit the admin API with curl to see the cert details), but I wanted to confirm how to do it. Golang has no runtime built-in key log dump, and you need to modify code:

ingress-controller-key-dump.diff.txt

go-kong-dump.diff.txt

The controller change is the one actually needed, since go-kong lets you provide your own client and we do. Left as a diff archive since I did a hack job and it's one of those things where we probably shouldn't ship it without a global --enforce-production to just exit immediately if anything like that is on.

From there you can dump traffic and kubectl cp the key dumps for Wireshark's TLS decrypt.

[pmalek]

Thanks for pointing me to these envs!

I managed to make it works with the following values:

gateway:
  admin:
    tls:
      client:
        secretName: "ca-cert"
  replicaCount: 2
  env:
    admin_ssl_cert: /etc/secrets/adminapi-cert/tls.crt
    admin_ssl_cert_key: /etc/secrets/adminapi-cert/tls.key
  secretVolumes:
  - adminapi-cert

controller:
  ingressController:
    image:
      repository: kong/kubernetes-ingress-controller
      tag: "3.0"

    adminApi:
      tls:
        client:
          enabled: true
          # If set to true, you are expected to provide your own secret (see secretName, caSecretName).
          certProvided: true
          # Client TLS certificate/key pair secret name that Ingress Controller will use to authenticate with Kong Admin API.
          secretName: "client-adminapi-cert"
          # CA TLS certificate/key pair secret name that the client TLS certificate is signed by.
          caSecretName: "ca-cert"

    env:
      gateway_discovery_dns_strategy: pod
      log_level: debug
      anonymous_reports: "false"
      kong_admin_tls_skip_verify: "false"

      kong_admin_ca_cert:
        valueFrom:
          secretKeyRef:
             name: ca-cert
             key: tls.crt

and the slightly modified script for the certs:

#!/usr/bin/env bash

set -ex -o pipefail

readonly CACERT=ca.crt
readonly CAKEY=ca.key
readonly TLSKEY=tls.key
readonly TLSCERT=tls.crt
readonly TLSCSR=tls.csr
readonly CLIENTTLSKEY=clienttls.key
readonly CLIENTTLSCERT=clienttls.crt
readonly CLIENTTLSCSR=clienttls.csr

readonly DAYS=365
readonly BITS=4096
readonly NAMESPACE=kong

test -f ${CACERT} || (echo "Creating root CA" && \
  openssl \
  req -x509 -sha256 -days ${DAYS} -newkey rsa:${BITS} -nodes \
  -subj "/C=US/ST=California/L=San Francisco/O=Kong/OU=Kong_Kubernetes/CN=ca.kong.pod" \
  -keyout ${CAKEY} -out ${CACERT})

openssl req -newkey rsa:${BITS} -keyout ${TLSKEY} -nodes \
  -subj "/C=US/ST=California/L=San Francisco/O=Kong/OU=Kong_Kubernetes/CN=*.kong.pod" \
  -out ${TLSCSR}

openssl x509 -req -CA ${CACERT} -CAkey ${CAKEY} -in ${TLSCSR} -out ${TLSCERT} \
  -subj "/C=US/ST=California/L=San Francisco/O=Kong/OU=Kong_Kubernetes/CN=*.kong.pod" \
  -days ${DAYS} -CAcreateserial -extfile domain.ext

openssl req -newkey rsa:${BITS} -keyout ${CLIENTTLSKEY} -nodes \
  -subj "/C=US/ST=California/L=San Francisco/O=Kong/OU=Kong_Kubernetes/CN=client.kong.pod" \
  -out ${CLIENTTLSCSR}

openssl x509 -req -CA ${CACERT} -CAkey ${CAKEY} -in ${CLIENTTLSCSR} -out ${CLIENTTLSCERT} \
  -subj "/C=US/ST=California/L=San Francisco/O=Kong/OU=Kong_Kubernetes/CN=client.kong.pod" \
  -days ${DAYS} -CAcreateserial

function secret()
{
  local NAME=${1}
  local NS=${2}
  local CERT=${3}
  local KEY=${4}
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: "${NAME}"
  namespace: ${NS}
type: kubernetes.io/tls
data:
  tls.crt: $(base64 -i ${CERT})
  tls.key: $(base64 -i ${KEY})
EOF
}

secret client-adminapi-cert ${NAMESPACE} ${CLIENTTLSCERT} ${CLIENTTLSKEY}
secret adminapi-cert ${NAMESPACE} ${TLSCERT} ${TLSKEY}
secret ca-cert ${NAMESPACE} ${CACERT} ${CAKEY}

(domain.ext stays as is)

2023-11-07T18:48:22Z    debug   events  successfully applied Kong configuration to https://10-244-0-200.kong.pod:8444   {"v": 1, "type": "Normal", "object": {"kind":"Pod","namespace":"kong","name":"kong-controller-7cb994b99f-xpxd8","apiVersion":"v1"}, "reason": "KongConfigurationSucceeded"}

[pmalek]

Given the above I wonder if we should close this or consider using this content in a guide in the docs or elsewhere 🤔 ? We don't seem to have a comprehensive guide saying how to configure this. We mostly cover the case of getting and configuring the cert at the ingress level.

[rainest]

Yeah, this basically becomes a docs ticket.

Per Michael we originally had this in https://docs.konghq.com/kubernetes-ingress-controller/2.12.x/guides/using-gateway-discovery/#installation but it hadn't gotten ported yet.