saranshflip
**Describe the bug** On a self-hosted Pixie Cloud deployment, the login flow fails at the final step with a `504 Gateway Timeout` returned by the `cloud-proxy` (openresty/nginx) proxy on `POST /api/auth/login`. Root cause traced to `api-server`'s gRPC client-side load balancer (`github.com/sercand/kuberesolver/v3`, registered via `kuberesolver.RegisterInCluster()` in `src/shared/services/service_flags.go`). It resolves backend addresses (e.g. `auth-service`) via `kubernetes:///<service>.<namespace>:<port>` URIs, which requires watching `pods`/`services`/`endpoints` through the Kubernetes API. `api-server`'s logs are flooded with: ``` kuberesolver: watching ended with error='invalid response code 403', will reconnect again ``` Because the watch never succeeds, `api-server` can never resolve `auth-service`'s pod IP, the gRPC call hangs, and `cloud-proxy` eventually times out with 504. **Note:** I was not able to conclusively confirm this is a reproducible bug in Pixie's code versus an authorization policy specific to our cluster's platform — see the "What we ruled out" section below. Posting here in case others hit the same symptom, and because the missing `discovery.k8s.io/endpointslices` RBAC grant seems like a potential gap regardless. **What we ruled out on our end** - `kubectl auth can-i {get,list,watch} {pods,services,endpoints,endpointslices} --as=system:serviceaccount:plc:default -n plc` → **yes** for everything, including after manually patching `pl-kuberesolver-role` (`k8s/cloud/base/kuberesolver_role.yaml`) to add `discovery.k8s.io/endpointslices`. - No `NetworkPolicy` present in the namespace. - No Calico/Cilium/Istio policies installed on the cluster. - Gatekeeper (OPA) webhooks only intercept `CREATE`/`UPDATE` on specific object kinds (deployments, services, pods, etc.) — never `GET`/`WATCH` — so irrelevant here. - Restarting the `api-server` pod (fresh service-account token) didn't change anything. - Plain DNS resolution of `auth-service.plc.svc.cluster.local` works fine and returns the correct pod IP — it's specifically the K8s-API-watch-based resolver that fails, not DNS. **To Reproduce** 1. Deploy Pixie Cloud (self-hosted) to a K8s v1.33.x cluster. 2. Port-forward `cloud-proxy` and browse to `/auth/login`. 3. Log in with valid credentials (Hydra/Kratos flow completes). 4. On the final `POST /api/auth/login` call, observe `504 Gateway Timeout` in the browser and `kuberesolver: watching ended with error='invalid response code 403'` spamming in `api-server` logs. **Expected behavior** Login completes successfully; `api-server` resolves `auth-service` (and other backends) via `kuberesolver` without 403s. **Questions for maintainers/community** 1. Does `kuberesolver v3.0.0` (as pinned in `go.mod`) ever watch `discovery.k8s.io/v1 EndpointSlices` instead of/in addition to core `v1 Endpoints`? `kuberesolver_role.yaml` only grants `pods`/`services`/`endpoints`, not `endpointslices` — this looks like a potential RBAC gap for clusters/K8s versions where resolution behavior differs. 2. This cluster is on **K8s v1.33.7**, where core `v1 Endpoints` is now deprecated (the API server emits a deprecation warning on every request) in favor of `discovery.k8s.io/v1 EndpointSlice`. Should Pixie's client-side gRPC resolver move to EndpointSlices for forward compatibility with 1.33+? 3. Has anyone else seen `kuberesolver` return 403 on a watch that `kubectl auth can-i` says should be allowed? We suspect this may be an authorization layer specific to our managed cluster platform rather than a Pixie bug, but wanted to check for prior reports of this interaction with `kuberesolver`/client-go. **Logs** `api-server` logs show the repeating `kuberesolver: watching ended with error='invalid response code 403'` message. (`./px collect-logs` not run, since this is a self-hosted deployment and root cause was isolated to a specific pod/service, not the full vizier/cloud log bundle.) **App information** - Pixie Cloud version: `0.1.9` (image tag across all `cloud-*` services) - Pixie CLI version: `0.8.8+Distribution.042e356.20250117230946.1.Homebrew` - K8s cluster version: `v1.33.7` - Node OS/Kernel: Debian GNU/Linux 11 (bullseye), `5.10.0-19-cloud-amd64`, containerd `1.6.36` - Browser: N/A (symptom only visible in-browser; root cause is server-side) **Additional context** Relevant RBAC definition: `k8s/cloud/base/kuberesolver_role.yaml`. `kuberesolver` registration: `src/shared/services/service_flags.go`.