New configuration annotations about gke-autoneg-controller

[erikjoh]

I think there is an opportunity here to make the setup even cleaner. Today, a full setup for a single port would look something like this:

apiVersion: v1
kind: Service
metadata:
  name: fabric-test
  namespace: fabric
  annotations:
    cloud.google.com/neg: '{"exposed_ports":{"5990":{}}}'
    controller.autoneg.dev/neg: '{"backend_services":{"5990":[{"max_rate_per_endpoint":800}]}}'
  labels:
    app: fabric-test
spec:
  selector:
    app: fabric-test
  type: ClusterIP
  clusterIP: None
  ports:
  - name: http
    port: 8080
    protocol: TCP
  - name: grpc
    port: 5990
    protocol: TCP

And it's for example not great that we need to specify the port number 5990 in three different places to make it work.
For our use-case, we usually also have multiple ports listed while we only care about one being exposed (for now at least).

One possible simplification might be if autoneg also could help with setting the cloud.google.com/neg annotation on the service.
So the flow would look something like:

1. autoneg sees some controller.autoneg.dev annotation(s)
2. autoneg sets the cloud.google.com/neg annotation accordingly
3. ingress-gce sees the cloud.google.com/neg annotation and creates the NEGs and sets cloud.google.com/neg-status
4. autoneg sees the cloud.google.com/neg-status and finally can add the NEGs to the backend service accordingly

This flow together with utilization of named ports can probably simplify things all the way down to something like this:

apiVersion: v1
kind: Service
metadata:
  name: fabric-test
  namespace: fabric
  annotations:
    controller.autoneg.dev/port-name: grpc
    controller.autoneg.dev/max-rate-per-endpoint: 800
  labels:
    app: fabric-test
spec:
  selector:
    app: fabric-test
  type: ClusterIP
  clusterIP: None
  ports:
  - name: http
    port: 8080
    protocol: TCP
  - name: grpc
    port: 5990
    protocol: TCP

And for other cases we could have options like for example:
controller.autoneg.dev/port-names: grpc,http, controller.autoneg.dev/port-names: '["grpc","http"']
controller.autoneg.dev/port: 5990, controller.autoneg.dev/ports: '["5990","8080"']

[soellman]

@fdfzcq Do you and/or Fabric have thoughts here? With the new service name defaulting, I think you would need only a single annotation (controller.autoneg.dev/max-rate-per-endpoint:<number>).

[fdfzcq]

@soellman Sorry I totally missed this question 😅 We will discuss in the team and get back to you!

[soellman]

Erik I think that's a great idea! Users can still manually set the cloud.google.com/neg and controller.autoneg.dev/neg annotations as they like.