On GKE, how to specify an IP for udp-gateway about stunner

[rg0now]

Using a fix external IP is, theoretically, supported by STUNner, but this is something that is not quite standardized across cloud providers and so we haven't tested this feature extensively yet.

The problem is that the address type you're using in the Gateway spec (NamedAddress) is deprecated and so we currently do not accept NamedAddress as the address type.

I have two suggestions.

First, try to replace the NamedAddress type with IPAddress as in the below, hoping that GKE would accept it:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: udp-gateway
  namespace: stunner-dev
spec:
  gatewayClassName: stunner-gatewayclass
  listeners:
    - name: udp-listener
      port: 3478
      protocol: TURN-UDP
  addresses:
    - type: IPAddress
      value: my_reserved_ip4

The other idea would be to create the LB Service manually (just copy-paste the YAML of the udp-gateway Service) and set the fields spec.externalIPs and spec.loadBalancerIP to the IP address you got from GKE. Would this work? Can you reach STUNner now over that IP? If you do then it is a quick patch to change the code to accept NamedAddress, that would at least fix GKE.

Also, can you please copy-paste the YAML of the service STUNner has created (kubectl -n stunner-dev get svc udp-gateway -o yaml)?

[mikolajkniejski]

svc.Spec.LoadBalancerIP works on GKE. I tested it on my machine and is also recommended on Stack Overflow. And setting external IP breaks LoadBalancers in gke.

[rg0now]

It just occurred to me that you could query the IP address assigned by gcloud compute addresses create ..., let the result be A.B.C.D. Then, set this IP with type IPAddress in the Gateway spec.addresses:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: udp-gateway
  namespace: stunner-dev
spec:
  gatewayClassName: stunner-gatewayclass
  listeners:
    - name: udp-listener
      port: 3478
      protocol: TURN-UDP
  addresses:
    - type: IPAddress
      value: A.B.C.D

It seems the NamedAddress type is not supported outside Google's own in-house load-balancers and, regrettably, we're not Google. Would this solution work for you?

[nhha1602]

Thank you for your helps,

I tried as your comments but I got below error on stunner operator:

2023-10-27T01:35:25.676233438Z  INFO    renderer        creating public service for gateway     {"name": "stunner-dev/udp-gateway", "gateway": "udp-gateway", "service": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner-dev\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner-dev\"},\"annotations\":{\"networking.gke.io/load-balancer-type\":\"Internal\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-dev/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1beta1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"21560782-753a-4e93-9689-14de0157495c\"}]},\"spec\":{\"ports\":[{\"name\":\"udp-listener\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":0}],\"selector\":{\"app\":\"stunner\"},\"type\":\"LoadBalancer\",\"externalIPs\":[\"172.18.100.73\"],\"loadBalancerIP\":\"172.18.100.73\"},\"status\":{\"loadBalancer\":{}}}"}
2023-10-27T01:35:25.676331505Z  INFO    renderer        STUNner dataplane configuration ready   {"generation": 2762, "config": "{version=\"v1alpha1\",admin:{name=\"stunner-daemon\",logLevel=\"all:INFO\",health-check=\"http://0.0.0.0:8086\"},auth:{realm=\"stunner.l7mp.io\",type=\"plaintext\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"stunner-dev/udp-gateway/udp-listener\":{turn://0.0.0.0:3478<32768-65535>,public=-:-,cert/key=-/-,routes=[stunner-dev/livekit-media-plane]}],clusters=[\"stunner-dev/livekit-media-plane\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[172.18.112.32,172.31.2.26]}]}"}
2023-10-27T01:35:25.676385128Z  INFO    cds-server      processing config update event  {"generation": 2762, "update": "update (gen: 2762): upsert-queue: gway-cls: 1, gway: 1, route: 1, svc: 1, confmap: 1, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, svc: 0, confmap: 0, dp: 0"}
2023-10-27T01:35:25.67641283Z   INFO    updater processing update event {"generation": 2762, "update": "update (gen: 2762): upsert-queue: gway-cls: 1, gway: 1, route: 1, svc: 1, confmap: 1, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, svc: 0, confmap: 0, dp: 0"}
2023-10-27T01:35:25.729584964Z  ERROR   updater cannot update service   {"operation": "unchanged", "service": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner-dev\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner-dev\"},\"annotations\":{\"networking.gke.io/load-balancer-type\":\"Internal\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-dev/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1beta1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"21560782-753a-4e93-9689-14de0157495c\"}]},\"spec\":{\"ports\":[{\"name\":\"udp-listener\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":0}],\"selector\":{\"app\":\"stunner\"},\"type\":\"LoadBalancer\",\"externalIPs\":[\"172.18.100.73\"],\"loadBalancerIP\":\"172.18.100.73\"},\"status\":{\"loadBalancer\":{}}}", "error": "cannot upsert service \"stunner-dev/udp-gateway\": services \"udp-gateway\" is forbidden: Use of external IPs is denied by admission control"}
github.com/l7mp/stunner-gateway-operator/internal/updater.(*Updater).ProcessUpdate
        /workspace/internal/updater/updater.go:115
github.com/l7mp/stunner-gateway-operator/internal/updater.(*Updater).Start.func1
        /workspace/internal/updater/updater.go:62

As I see, this error (is forbidden: Use of external IPs is denied by admission control) caused by duplicate by externalIP and loadbalacerIP ? One more thing, that I'm setting this LoadBalancer is Internal - \"networking.gke.io/load-balancer-type\":\"Internal\"

\"type\":\"LoadBalancer\",\"externalIPs\":[\"172.18.100.73\"],\"loadBalancerIP\":\"172.18.100.73\"}

Could you please advise this?

[nhha1602]

Hi,

I just found this link:
(l7mp/stunner-gateway-operator#32 (comment))

and stunner operator code:

// forward the first requested address to Kubernetes
	if len(gw.Spec.Addresses) > 0 {
		if gw.Spec.Addresses[0].Type == nil ||
			(gw.Spec.Addresses[0].Type != nil &&
				*gw.Spec.Addresses[0].Type == gwapiv1a2.IPAddressType) {
			// only the first address can be used because
			// stunner is limited to use a single public address
			// https://github.com/l7mp/stunner-gateway-operator/issues/32#issuecomment-1648035135
			svc.Spec.ExternalIPs = []string{gw.Spec.Addresses[0].Value}
			svc.Spec.LoadBalancerIP = gw.Spec.Addresses[0].Value
		}
	}

So, may be this we set externalIP to LoadBalancerIP will raised above error. Because my network team dose not allow to use externalIP in GKE - they will do NAT the internal LoadBalancerIP to public.

Please advise this.

[davidkornel]

@nhha1602 Hi,

as you mentioned your team does not allow using the externapIP field, thus you would like to set the svc.Spec.LoadBalancerIP field perhaps. I don't know what the answer should be yet however the loadBalancerIP field is deprecated and ignored by most cloud providers. Do you have any proof that GKE supports it?

    // Only applies to Service Type: LoadBalancer.
    // This feature depends on whether the underlying cloud-provider supports specifying
    // the loadBalancerIP when a load balancer is created.
    // This field will be ignored if the cloud-provider does not support the feature.
    // Deprecated: This field was under-specified and its meaning varies across implementations,
    // and it cannot support dual-stack.
    // As of Kubernetes v1.24, users are encouraged to use implementation-specific annotations when available.
    // This field may be removed in a future API version.
    // +optional
    LoadBalancerIP string `json:"loadBalancerIP,omitempty" protobuf:"bytes,8,opt,name=loadBalancerIP"`

We need to figure out first what all these fields mean actually and which of them are supported, etc. I think we need more docs about this topic, if you got some feel free to share with us.

[rg0now]

For a quick fix you can just remove the offending line below, build a new stunner image and deploy that:

svc.Spec.ExternalIPs = []string{gw.Spec.Addresses[0].Value}

Or you can remove the other problematic line and start anew:

svc.Spec.LoadBalancerIP = gw.Spec.Addresses[0].Value

Let us know which one works, if any.

Unfortunately support for static external IPs is so underspeficied in Kubernetes and the Gateway API that we don't know what's the most portable way to implement this. The official advice is to use cloud-provider specific Service annotations but we couldn't find any for GKE, let alone for EKS and the other one thousand cloud providers out there.

[nhha1602]

THank you for your helps,

I remove line: svc.Spec.ExternalIPs = []string{gw.Spec.Addresses[0].Value} and build new image for stunner operator and it worked.

Thank you.

[rg0now]

Thanks! Just for summary: svc.Spec.ExternalIPs and svc.Spec.LoadBalancerIP together are superfluous. I'm leaning towards removing svc.Spec.ExternalIPs, @davidkornel wdyt? Do you remember why we added it in the first place?

[davidkornel]

I'm afraid both of them were added in order to support this feature on different cloud-providers' clusters (without realizing that they would collide and break). Other thing is that these fields are not that used and underspecified. Just reading their API documentation reveals that it is not advised to use these, since they are either not supported or the field itself is deprecated, etc. It seems that the svc.Spec.LoadBalancerIP works on GKE, however, it is most likely to not work on other platforms. TBH this quick 'removing that line' fix might have worked but in the long game, we should do some research on this and implement that annotation-based idea that I proposed in this issue, or something like that.

[rg0now]

This has been addressed in l7mp/stunner-gateway-operator@d955a1b, at least for GKE. Feel free to reopen if bug regresses.