cloudflare / cloudflare-ingress-controller

A Kubernetes ingress controller for Cloudflare's Argo Tunnels

Argo Tunnel Ingress Controller

About

Argo Tunnel Ingress Controller provides Kubernetes Ingress via Argo Tunnels. The controller establishes or destroys tunnels by monitoring changes to resources.

Argo Tunnel offers an easy way to expose web servers securely to the internet, without opening up firewall ports and configuring ACLs. Argo Tunnel also ensures requests route through Cloudflare before reaching the web server so you can be sure attack traffic is stopped with Cloudflare’s WAF and Unmetered DDoS mitigation and authenticated with Access if you’ve enabled those features for your account.

• Argo Smart Routing
• Argo Tunnel

Deploy

kubectl apply -f deploy/argo-tunnel.yaml

Update the ServiceAccount namespace and bindings to deploy in an alternate namespace.

Without role based access control (RBAC).

kubectl apply -f deploy/argo-tunnel-no-rbac.yaml

With Helm.

helm install --name anydomain cloudflare/argo-tunnel

Note: replicas >1 requires load-balancers

Guides & Reference

• Argo Tunnel: Reference
• Argo Tunnel: Quick Start
• Deploying with Helm
• Setup Your First Tunnel
• Setup Tunnels to Subdomains
• Setup High Availability with Load Balancers
• Supported Command-Line Options, Annotations & Labels
• Monitoring & Analytics

Contributing

Thanks in advance for any and all contributions!

• Before contributing, please familiarize yourself with the Code of Conduct
• See contributing to setup your environment.
• Checkout the issues and roadmap.

Join the community

The Cloudflare community forum is a place to discuss Argo, Argo Tunnel, or any Cloudflare product.

cloudflare-ingress-controller's Issues

Create Cloudflare Warp Helm chart

Imho it would be a good to create a Helm chart for the warp ingress controller and publish it @ https://github.com/kubernetes/charts

Upstream chart could be maintained in a chart dir in this repo.

As a starting point this one could be used: https://github.com/dcos-labs/charts/tree/master/stable/cloudflare-warp-ingress

Mirror support for cloudflared tags

Cloudflared supports marking tunnels with user defined tags. Sometimes, these tags are not just metadata, but feature enable flags under the covers.

• https://developers.cloudflare.com/argo-tunnel/reference/arguments/#tag

The plan is to expose some mechanism to support tags. Unlike, cloudflared, many of the user desired tags can be populated automatically from ingress spec information (host, service, port, etc.). Attempting to specify information as part of an annotation would force users back into a one ingres-per-spec situation (which is not acceptable).

Options to configure tunnel defaults

A tunnel assumes these values; we would like to drive the configuration through command-line flags.

	tunnelRepairDelay  = 40 * time.Millisecond
	tunnelRepairJitter = 1.0
	tunnelTagLimit     = 32

Support for ingress with multiple paths

Any plan to fix this?

1 controller.go:145] Cannot create tunnel for ingress with multiple paths

Collect metrics from warp tunnels

Add additional metrics from the ingress controller itself.

Incorrect chart name in README.md

The command:

helm install --name $RELEASE_NAME --namespace $NS \
    --set rbac.install=$USE_RBAC \
    --set secret.install=true,secret.domain=$DOMAIN,secret.certificate_b64=$CERT_B64 \
    tc/argo-tunnel-ingress

Refers to tc/argo-tunnel-ingress which is invalid.

Running helm search argo returns:

NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                       
tc/argo-ingress                 0.5.0                           Installs the cloudflare argo tunnel ingress con...
tc/cloudflare-warp-ingress      0.5.0                           Installs the cloudflare argo tunnel ingress con...

I think README.md should be updated to reflect this change.

Health checks

How do set up health check with multiple Argo Tunnels in LB?

Support for multiple namespaces

Is to support multiple namespaces (aka, like a traditional ingress controller) in the roadmap?
Alternatively, how can I tunnel my NGINX ingress controller inside the warp, and keep configuring only nginx ingresses? Can I set up a wildcard hostname for the warp somehow?

32-bit ARM support

Hello,

I've built a K8s cluster out of Raspberry PIs that I would love to get set up with the argo ingress controller. Any thoughts around including a linux/arm (non-x64) docker image?

The lb-pool annotation doesn't work

The argo.cloudflare.com/lb-pool: <name of lb pool> annotation in the ingress controller definition doesn't actually set the load balancer pool name to the given one, because the flag is treated like a boolean. This means that the lb pools always use the hostname as the pool name.

Service backend needs to be port 80?

I had a service defined with a backend port of 7744. (targetPort: 7744 and port: 7744).

The nginx ingress deals with this OK - and routes the front end ingress (80/443) to the right destination port.

The warp ingress controller seems to assume the service will be on port 80.

Compile errors in master

Pulling master (@ 80e83b8) results in compile errors:

$ dep ensure
$ go build ./...
# github.com/cloudflare/cloudflare-ingress-controller/pkg/controller
pkg/controller/controller.go:53:61: undefined: Config
pkg/controller/controller.go:174:34: undefined: IngressClassKey
pkg/controller/controller.go:176:50: undefined: IngressClassKey
pkg/controller/controller.go:179:39: undefined: IngressClassKey
pkg/controller/controller.go:547:40: undefined: IngressAnnotationLBPool
pkg/controller/controller.go:560:20: undefined: SecretLabelDomain
pkg/controller/controller.go:568:79: undefined: SecretLabelDomain
pkg/controller/controller.go:573:88: undefined: SecretLabelDomain
pkg/controller/controller.go:573:117: undefined: SecretName
pkg/controller/controller.go:574:70: undefined: SecretName
pkg/controller/controller.go:574:70: too many errors

The same errors are present when running go install:

$ go install github.com/cloudflare/cloudflare-ingress-controller/cmd/argot
# github.com/cloudflare/cloudflare-ingress-controller/pkg/controller
pkg/controller/controller.go:53:61: undefined: Config
pkg/controller/controller.go:174:34: undefined: IngressClassKey
pkg/controller/controller.go:176:50: undefined: IngressClassKey
pkg/controller/controller.go:179:39: undefined: IngressClassKey
pkg/controller/controller.go:547:40: undefined: IngressAnnotationLBPool
pkg/controller/controller.go:560:20: undefined: SecretLabelDomain
pkg/controller/controller.go:568:79: undefined: SecretLabelDomain
pkg/controller/controller.go:573:88: undefined: SecretLabelDomain
pkg/controller/controller.go:573:117: undefined: SecretName
pkg/controller/controller.go:574:70: undefined: SecretName
pkg/controller/controller.go:574:70: too many errors

Services using named ports fail with error 'missing subsets for port'

When using a service definition like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: my-app
  name: my-service
  namespace: default
spec:
  ports:
  - name: my-port
    port: 9000
    targetPort: my-app-port
  selector:
    app: my-app
  type: ClusterIP

I get this error time="2019-01-03T01:33:43Z" level=error msg="translator service issue on ingress: default/my-argo-tunnel, host: example.com, path: {Path:/ Backend:{ServiceName:my-service ServicePort:{Type:0 IntVal:9000 StrVal:}}}, err: \"endpoints 'default/my-service' missing subsets for port 'my-app-port'\""

When using a literal targetPort (like targetPort: 8000) things work fine.

This project does not work out of the box

Ive been trying to implement the ingress for over 24 hours now and have given up. the instructions seem to refer to the bug branch's files and the master branch seems to have different named files "nginx.yaml" vs "nginx-ingress.yaml" without any documentation as to why. I also cant tell whether the warp deployment should have a service attached to it or not because one is not packaged with it, but that could also just be some kubernetes magic that I have not discovered yet. Regardless, i cannot get a public ip from the nginx.yaml file when getting the nginx ingress. ive tried using my own service (also a ClusterIP service like in the nginx yaml) to no avail either. No mention of the ConfigMap file anywhere in the docs. Please just update the docs to tell me what i am doing wrong! Thanks

house-keeping: move ./pkg to ./internal

The project doesn't expose code for use by external applications. The /pkg should be /internal.

• https://github.com/golang-standards/project-layout/blob/master/README.md

/internal

Private application and library code. This is the code you don't want others importing in their applications or libraries.

Put your actual application code in the /internal/app directory (e.g., /internal/app/myapp) and the code shared by those apps in the /internal/pkg directory (e.g., /internal/pkg/myprivlib).

/pkg

Library code that's ok to use by external applications (e.g., /pkg/mypubliclib). Other projects will import these libraries expecting them to work, so think twice before you put something here :-)

It's also a way to group Go code in one place when your root directory contains lots of non-Go components and directories making it easier to run various Go tool (as mentioned in the Best Practices for Industrial Programming from GopherCon EU 2018).

See the /pkg directory if you want to see which popular Go repos use this project layout pattern. This is a common layout pattern, but it's not universally accepted and some in the Go community don't recommend it.

Support using an existing Ingress as upstream

When using this project with Istio mTLS the ideal situation would be to send traffic through the Istio Ingress instead of directly to an Endpoint. Ingress resources don't support this mapping so it might make more sense to add a new CRD that maps a warp tunnel to any hostname.

Argo Tunnel (Warp) does not work anymore

Moving Warp to Argo Tunnel this ingress does not work anymore.
Enabled Argo tunnel and LoadBalancer for my domain, but still getting error:

level=error msg="Server error: error creating new pool demo.mydomain.com: authentication error: response: {\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 1002,\n      \"message\": \"the origin list length must be in range [1, 0]: validation failed\"\n    }\n  ],\n  \"messages\": []\n}\n"

Add support for cloudflared flag --proto-loglevel

DaemonSet option in helm chart

Hi,

It will be nice if we can have an option to deploy the controller as a DaemonSet instead of Deployment to run on each node. See the controller.kind option in https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress for an example. If DaemonSet then it implies loadbalancing true.

Regards,
Riaan

insert CODE_OF_CONDUCT.md

Add a CODE_OF_CONDUCT.md

After an unexpected disconnect, the ingress controller fails to reconnect and doesn't retry

My ingress controller has a tunnel that's established and running for several days. Unexpectedly, it logs a "connection reset by peer" message and tries to reestablish the tunnel. This fails because, it appears, Cloudflare is responding that a tunnel with the same name exists. The ingress controller then stops trying. After the previous tunnel times out (several minutes), I can delete the argo pod to force a new one to be created, and the tunnel is established again.

Logs:

time="2018-08-24T11:19:55Z" level=warning msg="frame read error" dir=read error="read tcp 10.5.65.143:42868->198.41.192.47:7844: read: connection reset by peer" subsystem=mux
time="2018-08-24T11:19:55Z" level=info msg="Stopping mux metrics updater" dir=metrics subsystem=mux
time="2018-08-24T11:19:55Z" level=error msg="Serve tunnel error" connectionID=0 error="read tcp 10.5.65.143:42868->198.41.192.47:7844: read: connection reset by peer"
time="2018-08-24T11:19:55Z" level=info msg="Retrying in 1s seconds"
time="2018-08-24T11:19:57Z" level=info msg="Connected to IAD"
time="2018-08-24T11:19:57Z" level=info msg="Stopping mux metrics updater" dir=metrics subsystem=mux
time="2018-08-24T11:19:57Z" level=error msg="Register tunnel error from server side" connectionID=0 error="Server error: Another tunnel with the same hostname and client-id <client-id> is active."

Should the ingress controller have a backoff loop but continue to retry tunnel registration when this happens?

This is 0.5.2 of the Helm chart.

ingress.class is being ignored

Just gave 0.6 a whirl and it appears that ingress.class support is broken, I've updated the cmdline to ---ingress-class=argo-tunnel and the controller is attempting to build tunnels for other ingresses that do not match its ingress.class, this is a log entry for my Jenkins ingress, which does not have a kubernetes.io/ingress.class defined and should be untouched by the argo controller:

time="2018-11-29T17:33:06Z" level=error msg="link exited with error (*errors.errorString) 'Server error: Failed to update CNAME', repairing ..." hostname=jenkins.<MYDOMAIN> origin="jenkins-web.jenkins:80"

High CPU consumption

Environment
Kubernetes cluster set up via kops
Setup explained in the post (as of 2017-12-15), which created a pod (copy-paste from running configuration):

{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "warp-controller-3266622471-0wff5",
    "generateName": "warp-controller-3266622471-",
    "namespace": "default",
    "selfLink": "/api/v1/namespaces/default/pods/warp-controller-3266622471-0wff5",
    "uid": "cf97545d-e1a9-11e7-805b-0a424d12d852",
    "resourceVersion": "5580173",
    "creationTimestamp": "2017-12-15T15:08:30Z",
    "labels": {
      "pod-template-hash": "3266622471",
      "run": "warp-controller"
    },
    "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"warp-controller-3266622471\",\"uid\":\"cf8b3a31-e1a9-11e7-805b-0a424d12d852\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"5580142\"}}\n",
      "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container warp-controller"
    },
    "ownerReferences": [
      {
        "apiVersion": "extensions/v1beta1",
        "kind": "ReplicaSet",
        "name": "warp-controller-3266622471",
        "uid": "cf8b3a31-e1a9-11e7-805b-0a424d12d852",
        "controller": true,
        "blockOwnerDeletion": true
      }
    ]
  },
  "spec": {
    "volumes": [
      {
        "name": "cloudflare-warp-cert",
        "secret": {
          "secretName": "cloudflare-warp-cert",
          "defaultMode": 420
        }
      },
      {
        "name": "default-token-bbdv5",
        "secret": {
          "secretName": "default-token-bbdv5",
          "defaultMode": 420
        }
      }
    ],
    "containers": [
      {
        "name": "warp-controller",
        "image": "quay.io/stackpoint/warp-controller:beta",
        "command": [
          "/warp-controller",
          "-v=6"
        ],
        "resources": {
          "requests": {
            "cpu": "100m"
          }
        },
        "volumeMounts": [
          {
            "name": "cloudflare-warp-cert",
            "readOnly": true,
            "mountPath": "/etc/cloudflare-warp"
          },
          {
            "name": "default-token-bbdv5",
            "readOnly": true,
            "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
          }
        ],
        "terminationMessagePath": "/dev/termination-log",
        "terminationMessagePolicy": "File",
        "imagePullPolicy": "Always"
      }
    ],
    "restartPolicy": "Always",
    "terminationGracePeriodSeconds": 30,
    "dnsPolicy": "ClusterFirst",
    "serviceAccountName": "default",
    "serviceAccount": "default",
    "nodeName": "ip-172-20-44-106.ec2.internal",
    "securityContext": {},
    "schedulerName": "default-scheduler",
    "tolerations": [
      {
        "key": "node.alpha.kubernetes.io/notReady",
        "operator": "Exists",
        "effect": "NoExecute",
        "tolerationSeconds": 300
      },
      {
        "key": "node.alpha.kubernetes.io/unreachable",
        "operator": "Exists",
        "effect": "NoExecute",
        "tolerationSeconds": 300
      }
    ]
  },
  "status": {
    "phase": "Running",
    "conditions": [
      {
        "type": "Initialized",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2017-12-15T15:08:30Z"
      },
      {
        "type": "Ready",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2017-12-15T15:08:34Z"
      },
      {
        "type": "PodScheduled",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2017-12-15T15:08:30Z"
      }
    ],
    "hostIP": "172.20.44.106",
    "podIP": "100.96.3.79",
    "startTime": "2017-12-15T15:08:30Z",
    "containerStatuses": [
      {
        "name": "warp-controller",
        "state": {
          "running": {
            "startedAt": "2017-12-15T15:08:33Z"
          }
        },
        "lastState": {},
        "ready": true,
        "restartCount": 0,
        "image": "quay.io/stackpoint/warp-controller:beta",
        "imageID": "docker-pullable://quay.io/stackpoint/warp-controller@sha256:4fe8ec7d8847f959438be57198b7da69c77f5ecfc264a16aea62ff2e98ab413a",
        "containerID": "docker://f1f3b4af3a81063e811c5c8e8eebb419400d43fa2bb65e612e92a6f492be576d"
      }
    ],
    "qosClass": "Burstable"
  }
}

and an ingress definition:

kind: Deployment
apiVersion: apps/v1beta1
metadata:
  name: httpbin
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: warp-service-app
    spec:
      containers:
      - name: httpbin
        image: kennethreitz/httpbin:latest
        ports:
        - containerPort: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: warp-service
spec:
  selector:
    app: warp-service-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: warp-service
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: cloudflare-warp
spec:
  rules:
  - host: httpbin.svstaging.net
    http:
      paths:
      - path: /
        backend:
          serviceName: warp-service
          servicePort: 8080

Expected
Low CPU consumption

Actual
Ingress works as expected, but the controller consumes 100% CPU consumption over 12+ hours period, then 80% (probably b/o other pods activation)

While nothing suspicious in the logs:

I1217 12:01:43.398224       1 controller.go:570] Validation ok for starting warp-service/1
I1217 12:01:43.838907       1 reflector.go:276] github.com/cloudflare/cloudflare-warp-ingress/pkg/controller/controller.go:281: forcing resync
I1217 12:01:43.839346       1 controller.go:130] No annotation found for kubernetes.io/ingress.class
I1217 12:01:43.839491       1 controller.go:133] Annotation kubernetes.io/ingress.class=cloudflare-warp
I1217 12:01:48.850426       1 reflector.go:276] github.com/cloudflare/cloudflare-warp-ingress/pkg/controller/controller.go:280: forcing resync
I1217 12:01:48.852048       1 controller.go:266] Watching endpoint default/warp-service
I1217 12:01:48.852282       1 controller.go:545] Start or Stop warp-service
I1217 12:01:48.852426       1 controller.go:570] Validation ok for starting warp-service/1
I1217 12:02:41.835812       1 reflector.go:405] github.com/cloudflare/cloudflare-warp-ingress/pkg/controller/controller.go:280: Watch close - *v1.Endpoints total 0 items received
I1217 12:02:41.838125       1 round_trippers.go:405] GET https://100.64.0.1:443/api/v1/namespaces/default/endpoints?resourceVersion=5809647&timeoutSeconds=453&watch=true 200 OK in 2 milliseconds
I1217 12:02:43.403461       1 reflector.go:276] github.com/cloudflare/cloudflare-warp-ingress/pkg/controller/controller.go:279: forcing resync
I1217 12:02:43.403537       1 controller.go:205] Watching service default/warp-service
I1217 12:02:43.403557       1 controller.go:545] Start or Stop warp-service
I1217 12:02:43.403564       1 controller.go:570] Validation ok for starting warp-service/1
I1217 12:02:43.846662       1 reflector.go:276] github.com/cloudflare/cloudflare-warp-ingress/pkg/controller/controller.go:281: forcing resync
I1217 12:02:43.846704       1 controller.go:130] No annotation found for kubernetes.io/ingress.class
I1217 12:02:43.846711       1 controller.go:133] Annotation kubernetes.io/ingress.class=cloudflare-warp
I1217 12:02:48.859712       1 reflector.go:276] github.com/cloudflare/cloudflare-warp-ingress/pkg/controller/controller.go:280: forcing resync
I1217 12:02:48.859777       1 controller.go:266] Watching endpoint default/warp-service
I1217 12:02:48.859805       1 controller.go:545] Start or Stop warp-service

PS: it could be related to my prior experiments with multi-rule ingress configuration (as discussed here ). While I reverted the ingress configuration to the correct one, the controller process remained the same and could have been affected.

Docker image warp-controller:0.4.2 is not available

Currently, both the warp-controller.deployment.yaml and the helm chart reference warp-controller version 0.4.2 (quay.io/stackpoint/warp-controller:0.4.2).

It appears that this tag does not exist.

Session Affinity

Do Argo Tunnels supports Session Affinity (sticky sessions) in cluster scope?
For example, Nginx Ingress Controller allows specifying several annotations for it:

nginx.ingress.kubernetes.io/session-cookie-name: "__lbroute"
nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"

It's important to us, due we support old web-browsers without WebSockets, using XHR Long Polling as a fallback.

Probing from every data centre is not allowed with this subscription

I get an error similar to this when I try the basic instructions, even though cloudflared --hello-world works for this account and domain name.

Is there a way to get more information out if it to see what I'm doing wrong?

ERRO[0037] Server error: error creating new pool <domain name>: authentication error: response: {
  "result": null,
  "success": false,
  "errors": [
    {
      "code": 1002,
      "message": "Probing from every data centre is not allowed with this subscription. Please choose regions to probe from.: validation failed"
    }
  ],
  "messages": []
}

Unable to configure argo on minikube

Hi

I'm sure i'm missing something during configuration of Argo Ingress for my minikube installation,
but i'm getting following errors after third reinstall of whole configuration:

time="2019-01-12T16:15:09Z" level=error msg="link exited with error (*net.DNSError) 'lookup _warp._tcp.cloudflarewarp.com on 10.96.0.10:53: no such host', repairing ..." hostname=mk-alex.actonica.ru origin="echo.default:80"

time="2019-01-12T16:15:09Z" level=info msg="link repair starts in 23.014208ms" hostname=mk-alex.actonica.ru origin="echo.default:80"

time="2019-01-12T16:15:09Z" level=info msg="ResolveEdgeIPs err"

time="2019-01-12T16:15:09Z" level=error msg="link exited with error (*net.DNSError) 'lookup _warp._tcp.cloudflarewarp.com on 10.96.0.10:53: no such host', repairing ..." hostname=mk-alex.actonica.ru origin="echo.default:80"

time="2019-01-12T16:15:09Z" level=info msg="link repair starts in 25.343819ms" hostname=mk-alex.actonica.ru origin="echo.default:80"

time="2019-01-12T16:15:09Z" level=info msg="ResolveEdgeIPs err"

In my case i'm configuring subdomain, so i follow Argo Tunnels for Subdomains article.

rm -rf ~/.minikube

minikube start
helm init
helm repo update   

helm install --name anydomain --namespace default \
    --set rbac.create=true \
    --set controller.ingressClass=argo-tunnel \
    --set controller.logLevel=6 \
    cloudflare/argo-tunnel

kubectl create secret generic actonica.ru --from-file="$HOME/.cloudflared/cert.pem"

awk '/BEGIN.*TUNNEL/{mark=1}/END.*TUNNEL/{print;mark=0}mark' ~/.cloudflared/cert.pem >> mk-alex.pem
kubectl create secret generic mk-alex.actonica.ru --from-file="mk-alex.pem"

kubectl apply -f argo-sample.yaml 
kubectl apply -f argo-tunnel-sample.yaml 

argo-sample.yaml.txt
argo-tunnel-sample.yaml.txt

I could miss some steps that I did...

What i'm doing wrong?

User cannot list resource in API group at the cluster scope

Hi,

After following the doc, I get this when I log the ingress-controller

E1029 16:16:28.196566       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:319: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:cloudflare-argo" cannot list resource "services" in API group "" at the cluster scope
E1029 16:16:28.273495       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:default:cloudflare-argo" cannot list resource "endpoints" in API group "" at the cluster scope
E1029 16:16:28.666380       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:cloudflare-argo" cannot list resource "ingresses" in API group "extensions" at the cluster scope

I really don't understand what I did wrong.

Here is my deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: dwoom-account-frontend
  name: dwoom-account-frontend
  namespace: default
spec:
  progressDeadlineSeconds: 60
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: dwoom-account-frontend
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: dwoom-account-frontend
    spec:
      containers:
      - image: registry.gitlab.com/dwoom/account/frontend:dev-build
        imagePullPolicy: Always
        name: dwoom-account-frontend
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

My service

kind: Service
apiVersion: v1
metadata:
  name: dwoom-account-frontend-argo
spec:
  selector:
    app: dwoom-account-frontend
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

And my warp-controller

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: argo-tunnel
  name: dwoom-account-frontend-ingress
  namespace: default
spec:
  rules:
  - host: account.dwoom.com
    http:
      paths:
      - backend:
          serviceName: dwoom-account-frontend
          servicePort: 80

I'm new to kubernetes and halm, so all of this is overwhelming. Any idea where I made an error?

meaningful ingress updates are ignored

An ingress can be updated in place to change significant parameters such as the hostname. At present those updates are ignored. The tunnel should be removed and recreated.

GRPC support

Do you have any plans for supporting GRPC backend services with Argo tunneling?

Allow setting ingress.class

To allow multiple ingresses for multiple domains we need to be able to set the ingress.class on the ingress controller to map service -> controller with multiple agro Ingress controllers in a cluster.

Getting started example does not work

I followed the getting started tutorial in the readme and can't get the example working.

The domain I had was the TLD associated with cloudflare account, and I used httpbin.${mydomain.com} as the host configuration in an ingress deployment file.

FWIW, when I used cloudflared directly to tunnel, I was able to see a dns record in my cloudflare account and it also worked.

However, no such record got created when I tried to setup argo tunnel via the ingress controller. I tried using both the TLD as well as a subdomain for the domain passed in as cloudflare-argo/domain and neither approach worked.

Here are the logs from argo tunnel pod when I try to apply the ingress kubectl create -f argo.ingress.yaml

I0813 02:30:46.467708       1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
I0813 02:30:46.467780       1 controller.go:610] creating tunnel for ingress httpbin, default/httpbin/httpbin
I0813 02:30:46.476709       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/namespaces/default/secrets/cloudflared-cert 200 OK in 8 milliseconds
I0813 02:30:46.478739       1 controller.go:636] created tunnel for ingress httpbin,  default/httpbin/httpbin
I0813 02:30:46.478779       1 controller.go:699] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
I0813 02:30:46.478841       1 controller.go:723] Starting tunnel to url httpbin.default:80
I0813 02:30:46.485156       1 round_trippers.go:436] GET https://10.96.0.1:443/apis/extensions/v1beta1/namespaces/default/ingresses/httpbin 200 OK in 6 milliseconds
I0813 02:30:46.489599       1 round_trippers.go:436] PUT https://10.96.0.1:443/apis/extensions/v1beta1/namespaces/default/ingresses/httpbin/status 200 OK in 3 milliseconds
I0813 02:30:46.489817       1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
I0813 02:30:46.489849       1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel

And here are the logs when I delete it kubectl delete -f argo.ingress.yaml

I0813 02:31:10.058995       1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
I0813 02:31:10.059101       1 controller.go:770] Removing tunnel default/httpbin/httpbin
I0813 02:31:10.059132       1 controller.go:502] Error processing delete:default/httpbin/httpbin: lookup cftunnel.com on 10.96.0.10:53: no such host
I0813 02:31:10.064582       1 controller.go:770] Removing tunnel default/httpbin/httpbin
I0813 02:31:10.064626       1 controller.go:502] Error processing delete:default/httpbin/httpbin: Tunnel not found for key default/httpbin/httpbin
I0813 02:31:10.075053       1 controller.go:770] Removing tunnel default/httpbin/httpbin
E0813 02:31:10.075128       1 controller.go:508] Dropping object "delete:default/httpbin/httpbin" out of the queue: Tunnel not found for key default/httpbin/httpbin

Error accessing kubernetes service and edge

I'm struggling setting up argo. I did everything I could find in the readme and available guides.

My cluster is 3 masters and 3 nodes on bare metal with k8s version 1.10.5

The argo pod shows some weird logs:

I0801 10:47:27.085989       1 main.go:67] Starting Controller
I0801 10:47:27.086127       1 controller.go:318] Starting ArgoController
I0801 10:47:27.180904       1 reflector.go:202] Starting reflector *v1.Service (1m0s) from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320
I0801 10:47:27.180971       1 reflector.go:240] Listing and watching *v1.Service from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320
I0801 10:47:27.181051       1 reflector.go:202] Starting reflector *v1.Endpoints (1m0s) from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321
I0801 10:47:27.181095       1 reflector.go:240] Listing and watching *v1.Endpoints from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321
I0801 10:47:27.182046       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0  in 0 milliseconds
I0801 10:47:27.182048       1 reflector.go:202] Starting reflector *v1beta1.Ingress (1m0s) from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322
I0801 10:47:27.182077       1 reflector.go:240] Listing and watching *v1beta1.Ingress from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322
E0801 10:47:27.182134       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:27.182590       1 round_trippers.go:436] GET https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0  in 0 milliseconds
I0801 10:47:27.182623       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0  in 1 milliseconds
E0801 10:47:27.182657       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322: Failed to list *v1beta1.Ingress: Get https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
E0801 10:47:27.182719       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:28.182462       1 reflector.go:240] Listing and watching *v1.Service from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320
I0801 10:47:28.183406       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0  in 0 milliseconds
I0801 10:47:28.183558       1 reflector.go:240] Listing and watching *v1beta1.Ingress from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322
E0801 10:47:28.183571       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:28.184596       1 round_trippers.go:436] GET https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0  in 0 milliseconds
I0801 10:47:28.184638       1 reflector.go:240] Listing and watching *v1.Endpoints from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321
E0801 10:47:28.184917       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322: Failed to list *v1beta1.Ingress: Get https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:28.185477       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0  in 0 milliseconds
E0801 10:47:28.185547       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:29.184150       1 reflector.go:240] Listing and watching *v1.Service from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320
I0801 10:47:29.184929       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0  in 0 milliseconds
E0801 10:47:29.185074       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:29.280610       1 reflector.go:240] Listing and watching *v1beta1.Ingress from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322
I0801 10:47:29.280609       1 reflector.go:240] Listing and watching *v1.Endpoints from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321
I0801 10:47:29.281325       1 round_trippers.go:436] GET https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0  in 0 milliseconds
E0801 10:47:29.281412       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322: Failed to list *v1beta1.Ingress: Get https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:29.281503       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0  in 0 milliseconds
E0801 10:47:29.281575       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:30.185451       1 reflector.go:240] Listing and watching *v1.Service from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320
I0801 10:47:30.186064       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0  in 0 milliseconds
E0801 10:47:30.186144       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:30.281631       1 reflector.go:240] Listing and watching *v1beta1.Ingress from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322
I0801 10:47:30.282365       1 round_trippers.go:436] GET https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0  in 0 milliseconds
E0801 10:47:30.282463       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322: Failed to list *v1beta1.Ingress: Get https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:30.282811       1 reflector.go:240] Listing and watching *v1.Endpoints from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321
I0801 10:47:30.283371       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0  in 0 milliseconds
E0801 10:47:30.283493       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:31.186497       1 reflector.go:240] Listing and watching *v1.Service from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320
I0801 10:47:31.187647       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0  in 0 milliseconds
E0801 10:47:31.187806       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:31.282760       1 reflector.go:240] Listing and watching *v1beta1.Ingress from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322
I0801 10:47:31.283555       1 round_trippers.go:436] GET https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0  in 0 milliseconds
E0801 10:47:31.283661       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322: Failed to list *v1beta1.Ingress: Get https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:31.283734       1 reflector.go:240] Listing and watching *v1.Endpoints from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321
I0801 10:47:31.284342       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0  in 0 milliseconds
E0801 10:47:31.284408       1 reflector.go:205] github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused
I0801 10:47:32.188083       1 reflector.go:240] Listing and watching *v1.Service from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:320
I0801 10:47:32.283929       1 reflector.go:240] Listing and watching *v1beta1.Ingress from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:322
I0801 10:47:32.285053       1 reflector.go:240] Listing and watching *v1.Endpoints from github.com/cloudflare/cloudflare-ingress-controller/pkg/controller/controller.go:321
I0801 10:47:32.333878       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0 200 OK in 145 milliseconds
I0801 10:47:32.358810       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0 200 OK in 73 milliseconds
I0801 10:47:32.359157       1 round_trippers.go:436] GET https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0 200 OK in 74 milliseconds
I0801 10:47:32.362904       1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
I0801 10:47:32.362945       1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
I0801 10:47:32.480775       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/services?resourceVersion=212035&timeoutSeconds=582&watch=true 200 OK in 96 milliseconds
I0801 10:47:32.480888       1 round_trippers.go:436] GET https://10.96.0.1:443/apis/extensions/v1beta1/ingresses?resourceVersion=216501&timeoutSeconds=481&watch=true 200 OK in 118 milliseconds
I0801 10:47:32.484518       1 shared_informer.go:122] caches populated
I0801 10:47:32.580634       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/endpoints?resourceVersion=217218&timeoutSeconds=499&watch=true 200 OK in 93 milliseconds
I0801 10:47:32.680532       1 shared_informer.go:122] caches populated
I0801 10:47:32.680674       1 controller.go:610] creating tunnel for ingress grafana, monitoring/grafana/grafana
I0801 10:47:32.716552       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/namespaces/default/secrets/cloudflared-cert 200 OK in 35 milliseconds
I0801 10:47:32.727808       1 controller.go:636] created tunnel for ingress grafana,  monitoring/grafana/grafana
I0801 10:47:32.727860       1 controller.go:699] Validation ok for running monitoring/grafana/grafana with 1 endpoint(s)
I0801 10:47:32.727903       1 controller.go:723] Starting tunnel to url grafana.monitoring:30802
I0801 10:47:32.780714       1 round_trippers.go:436] GET https://10.96.0.1:443/apis/extensions/v1beta1/namespaces/monitoring/ingresses/grafana 200 OK in 52 milliseconds
I0801 10:47:32.850748       1 round_trippers.go:436] PUT https://10.96.0.1:443/apis/extensions/v1beta1/namespaces/monitoring/ingresses/grafana/status 200 OK in 67 milliseconds
I0801 10:47:32.851040       1 controller.go:610] creating tunnel for ingress nginx, default/nginx/nginx
time="2018-08-01T10:47:32Z" level=error msg="Unable to dial edge" error="Handshake with edge error: EOF"
time="2018-08-01T10:47:32Z" level=info msg="Retrying in 1s seconds"
I0801 10:47:32.885769       1 round_trippers.go:436] GET https://10.96.0.1:443/api/v1/namespaces/default/secrets/cloudflared-cert 200 OK in 34 milliseconds
I0801 10:47:32.886904       1 controller.go:636] created tunnel for ingress nginx,  default/nginx/nginx
I0801 10:47:32.886964       1 controller.go:699] Validation ok for running default/nginx/nginx with 1 endpoint(s)
I0801 10:47:32.886983       1 controller.go:723] Starting tunnel to url nginx.default:80
I0801 10:47:32.922579       1 round_trippers.go:436] GET https://10.96.0.1:443/apis/extensions/v1beta1/namespaces/default/ingresses/nginx 200 OK in 35 milliseconds
I0801 10:47:32.980750       1 round_trippers.go:436] PUT https://10.96.0.1:443/apis/extensions/v1beta1/namespaces/default/ingresses/nginx/status 200 OK in 57 milliseconds
time="2018-08-01T10:47:33Z" level=error msg="Unable to dial edge" error="Handshake with edge error: EOF"
time="2018-08-01T10:47:33Z" level=info msg="Retrying in 1s seconds"
time="2018-08-01T10:47:33Z" level=error msg="Unable to dial edge" error="Handshake with edge error: EOF"
time="2018-08-01T10:47:33Z" level=info msg="Retrying in 2s seconds"
time="2018-08-01T10:47:34Z" level=error msg="Unable to dial edge" error="Handshake with edge error: read tcp 10.244.5.22:41782->198.41.192.2:7844: read: connection reset by peer"
time="2018-08-01T10:47:34Z" level=info msg="Retrying in 2s seconds"
time="2018-08-01T10:47:35Z" level=error msg="Unable to dial edge" error="Handshake with edge error: EOF"
time="2018-08-01T10:47:35Z" level=info msg="Retrying in 4s seconds"
time="2018-08-01T10:47:36Z" level=error msg="Unable to dial edge" error="Handshake with edge error: EOF"
time="2018-08-01T10:47:36Z" level=info msg="Retrying in 4s seconds"
time="2018-08-01T10:47:39Z" level=error msg="Unable to dial edge" error="Handshake with edge error: read tcp 10.244.5.22:45550->198.41.200.2:7844: read: connection reset by peer"
time="2018-08-01T10:47:39Z" level=info msg="Retrying in 8s seconds"
time="2018-08-01T10:47:40Z" level=error msg="Unable to dial edge" error="Handshake with edge error: read tcp 10.244.5.22:41808->198.41.192.2:7844: read: connection reset by peer"
time="2018-08-01T10:47:40Z" level=info msg="Retrying in 8s seconds"
time="2018-08-01T10:47:47Z" level=error msg="Unable to dial edge" error="Handshake with edge error: EOF"
time="2018-08-01T10:47:47Z" level=info msg="Retrying in 16s seconds"
time="2018-08-01T10:47:48Z" level=error msg="Unable to dial edge" error="Handshake with edge error: read tcp 10.244.5.22:41848->198.41.192.2:7844: read: connection reset by peer"
time="2018-08-01T10:47:48Z" level=info msg="Retrying in 16s seconds"

Any hints are much appreciated. :)

Segment README.md into a series of guides

The goal is to split up the README.md into a series of guides that can be more focused and detailed, while leaving a simple landing page.

Add Access Policies

It would be very powerful to be able to define policies for Cloudflare Access within the Ingress controller definition. You will be able to, for example, specify that only employees of your company can access a given deployment.

Imports things under github.com/cloudflare/cloudflare-warp yet that repo is 404 to the general public

For example, https://github.com/cloudflare/cloudflare-warp-ingress/blob/48fc9e03fe1aad83d6c6650237ba6bc502c72b7d/pkg/tunnel/warp.go#L9

Update Cloudflared (Argo Tunnel) dependency

Hello,

From the look of it, you're not following the released of Cloudflared. This ingress controller is using cloudflared version 2018.4.7.

The latest version is 2018.7.0.

For our use case, we would need the websocket to work, but for that to happen, the ingress would need to follow the release of the tunnel.

Can you take the time to update your dependencies and docker image version ?

Websocket authentication errors

I set up a mattermost chat server, which has its own helm chart: https://github.com/mattermost/mattermost-kubernetes

This helm chart includes an Nginx ingress controller, however I wanted to use the cloudflare ingress controller w/ argo tunnels. So, after installing the mattermost helm release, I deleted the default ingress controller and installed our one with this config (note that the service name changes every time):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: argo-tunnel
    argo.cloudflare.com/lb-pool: testpool-name
  name: chat
  namespace: default
spec:
  rules:
  - host: chat.rishabh.ga
    http:
      paths:
      - backend:
          serviceName: wandering-frog-mattermost-app
          servicePort: 8065

This unfortunately doesn't work. I'm able to access the site at chat.rishabh.ga, but I see these logs in the chrome network tab:

websocket re-established connection
websocket_client.jsx:65 websocket closed
websocket_client.jsx:49 websocket re-established connection
websocket_client.jsx:65 websocket closed
websocket_client.jsx:49 websocket re-established connection
websocket_client.jsx:65 websocket closed

These logs in the cloudflare ingress controller pod:

time="2018-07-06T16:30:38Z" level=info msg="200 OK"
time="2018-07-06T16:30:38Z" level=warning msg="All requests should have a CF-RAY header. Please open a support ticket with Cloudflare. GET http://wandering-frog-mattermost-app.default:8065/ HTTP/1.1 "
time="2018-07-06T16:30:38Z" level=info msg="200 OK"
time="2018-07-06T16:30:39Z" level=warning msg="All requests should have a CF-RAY header. Please open a support ticket with Cloudflare. GET http://wandering-frog-mattermost-app.default:8065/ HTTP/1.1 "
time="2018-07-06T16:30:39Z" level=info msg="200 OK"
time="2018-07-06T16:30:39Z" level=warning msg="All requests should have a CF-RAY header. Please open a support ticket with Cloudflare. GET http://wandering-frog-mattermost-app.default:8065/ HTTP/1.1 "
time="2018-07-06T16:30:39Z" level=info msg="200 OK"
time="2018-07-06T16:30:41Z" level=warning msg="All requests should have a CF-RAY header. Please open a support ticket with Cloudflare. GET http://wandering-frog-mattermost-app.default:8065/ HTTP/1.1 "

And these in the mattermost app pod:

{"level":"error","ts":1530893950.0812402,"caller":"app/websocket_router.go:94","msg":"websocket routing error: seq=1 uid= api.web_socket_router.not_authenticated.app_error [details: ]"}

I'm not sure what's causing this. Maybe looking at the default ingress controller defined in the mattermost chart repo would help?

Enable PR and nightly builds from travis-ci

Install travis-ci as an application for the repository and configure it to build PRs and nightlies

Add ability to specify origin certificate secrets as CLI flag or env var

Prior to v0.6.0, domain certificate secrets were found using labels on the secret. In v0.6.0, this was removed in favor of the tls section of each ingress. For our use case, this is a regression. We will have many, many ingresses (in the hundreds) spanning many namespaces (~80 or 90) and adding secrets to each namespace and modifying each ingress is a large amount of work and is hard to maintain as we add more domains.

In the current version, I see one can set a default origin secret via a CLI flag. It would be great to add the ability to specify multiple origin secrets via a CLI flag or env var. This would totally fix our issue with the way origin secrets are handled in v0.6.0.

Update cloudflared (argo tunnel) dependency to 2018.12.1 or higher

Pull forward dependent library to latest release of cloudlfared

• https://github.com/cloudflare/cloudflared/releases

JSON logging

Would it be possible to have cloudflare output json logs instead of the current format?

Document usage of HA lb-pools

Describe typical single-namespace usage, as well as load-balancing across separate clusters.

I cannot pull the 0.6.2 controller with helm

Allow Ingress Url Path based routing

The ingress controller does not support url based routing (the IngressPath attribute Path)

spec:
  rules:
  - host: echo.mydomain.com
    http:
      paths:
      - backend:
      - path: any_path_is_treated_as_slash
          serviceName: echo
          servicePort: http

Thread-Safety issues during shutdown

Race detection reports issues during tunnel shutdown.'

==================
WARNING: DATA RACE
Write at 0x00c0006fe030 by goroutine 74:
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.TestTunnelServiceInitialization()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/client-go/testing/fake.go:102 +0xf49
  testing.tRunner()
      /usr/local/opt/go/libexec/src/testing/testing.go:827 +0x162

Previous read at 0x00c0006fe030 by goroutine 92:
  [failed to restore the stack]

Goroutine 74 (running) created at:
  testing.(*T).Run()
      /usr/local/opt/go/libexec/src/testing/testing.go:878 +0x650
  testing.runTests.func1()
      /usr/local/opt/go/libexec/src/testing/testing.go:1119 +0xa8
  testing.tRunner()
      /usr/local/opt/go/libexec/src/testing/testing.go:827 +0x162
  testing.runTests()
      /usr/local/opt/go/libexec/src/testing/testing.go:1117 +0x4ee
  testing.(*M).Run()
      /usr/local/opt/go/libexec/src/testing/testing.go:1034 +0x2ee
  main.main()
      _testmain.go:64 +0x221

Goroutine 92 (running) created at:
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).Run()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:330 +0x420
==================
==================
WARNING: DATA RACE
Read at 0x00c000026840 by goroutine 74:
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.TestTunnelServiceInitialization()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller_test.go:445 +0x1309
  testing.tRunner()
      /usr/local/opt/go/libexec/src/testing/testing.go:827 +0x162

Previous write at 0x00c000026840 by goroutine 92:
  runtime.mapassign_faststr()
      /usr/local/opt/go/libexec/src/runtime/map_faststr.go:190 +0x0
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).setTunnel()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:83 +0xbe
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).createTunnel()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:625 +0x750
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).processIngress()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:420 +0xca4
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).processNextIngress()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:351 +0x11f
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).runIngressWorker()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:339 +0x38
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).runIngressWorker-fm()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:330 +0x41
  github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x61
  github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xce
  github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x5a

Goroutine 74 (running) created at:
  testing.(*T).Run()
      /usr/local/opt/go/libexec/src/testing/testing.go:878 +0x650
  testing.runTests.func1()
      /usr/local/opt/go/libexec/src/testing/testing.go:1119 +0xa8
  testing.tRunner()
      /usr/local/opt/go/libexec/src/testing/testing.go:827 +0x162
  testing.runTests()
      /usr/local/opt/go/libexec/src/testing/testing.go:1117 +0x4ee
  testing.(*M).Run()
      /usr/local/opt/go/libexec/src/testing/testing.go:1034 +0x2ee
  main.main()
      _testmain.go:64 +0x221

Goroutine 92 (running) created at:
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).Run()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:330 +0x420
==================
I0917 07:24:58.705336   18311 controller.go:334] Stopping ArgoController 
--- FAIL: TestTunnelServiceInitialization (6.01s)
    testing.go:771: race detected during execution of test
==================
WARNING: DATA RACE
Read at 0x00c0001ab950 by goroutine 94:
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.TestTunnelServicesTwoNS()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller_test.go:535 +0x1592
  testing.tRunner()
      /usr/local/opt/go/libexec/src/testing/testing.go:827 +0x162

Previous write at 0x00c0001ab950 by goroutine 110:
  runtime.mapassign_faststr()
      /usr/local/opt/go/libexec/src/runtime/map_faststr.go:190 +0x0
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).setTunnel()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:83 +0xbe
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).createTunnel()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:625 +0x750
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).processIngress()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:420 +0xca4
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).processNextIngress()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:351 +0x11f
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).runIngressWorker()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:339 +0x38
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).runIngressWorker-fm()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:330 +0x41
  github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x61
  github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xce
  github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x5a

Goroutine 94 (running) created at:
  testing.(*T).Run()
      /usr/local/opt/go/libexec/src/testing/testing.go:878 +0x650
  testing.runTests.func1()
      /usr/local/opt/go/libexec/src/testing/testing.go:1119 +0xa8
  testing.tRunner()
      /usr/local/opt/go/libexec/src/testing/testing.go:827 +0x162
  testing.runTests()
      /usr/local/opt/go/libexec/src/testing/testing.go:1117 +0x4ee
  testing.(*M).Run()
      /usr/local/opt/go/libexec/src/testing/testing.go:1034 +0x2ee
  main.main()
      _testmain.go:64 +0x221

Goroutine 110 (running) created at:
  github.com/cloudflare/cloudflare-ingress-controller/internal/controller.(*ArgoController).Run()
      /Users/malberts/go/src/github.com/cloudflare/cloudflare-ingress-controller/internal/controller/controller.go:330 +0x420
==================
I0917 07:25:03.816015   18311 watch.go:109] Stopping fake watcher.
--- FAIL: TestTunnelServicesTwoNS (5.11s)
    testing.go:771: race detected during execution of test

Fatal cloudflared issues are difficult to find

Cloudflared logs are buried and are difficult to find. It would be nice if fatal events (LB errors, etc.) were in the Kubernetes Event Log.

Ingress HTTPS native services

Trying to use argo tunnel with the kubernetes dashboard, only communicates via HTTPS.

Getting errors like this in the argo ingress controller

time="2018-09-04T01:23:49Z" level=error msg="HTTP request error" error="net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x15\\x03\\x01\\x00\\x02\\x02\""

Is there a way to set up argo to be able to work with HTTPS services?

migrate docker image to gcr.io/cloudflare-registry with multi-arch support

Migrate the docker image to an alternate docker repository and provide multi-arch support.

image: "gcr.io/cloudflare-registry/argo-tunnel:0.5.2"

tip: the published manifest allows docker to select the architecture

Task List:

• rename repository name argot ---> argo-tunnel
• update helm chart values ./chart/values.yaml
• update pre-built manifest ./deploy/argo-tunnel.yaml
• update pre-built manifest ./deploy/argo-tunnel-no-rbac.yaml
• add repository argo-tunnel-amd64
• add repository argo-tunnel-arm64v8
• add repository argo-tunnel-ppc64le
• add manifest argo-tunnel

Only serve traffic via Cloudflare access

Rather than trying to define access policies within argo config itself (#32), would it be possible to at least say that hey this particular service can only be accessed through Cloudflare access, and reject all unauthenticated traffic?

I imagine it's possible to set up a reverse proxy in between argo and the service being served and perform JWT verification (https://developers.cloudflare.com/access/setting-up-access/validate-jwt-tokens/), however that's a bunch of extra work.