failed to open dashboard in minikube in some situations where --extra-config is used about linkerd2

[klingerf]

Hi @darren-fu -- thanks for reporting this! Can you provide a bit more info about your kubernetes environment? Are you running in minikube, cloud hosted, etc? Is RBAC enabled?

Fwiw, I ran into this issue in a minikube environment that didn't have RBAC properly configured. The error looked like this in my web browser:

The issue was that some of the containers in the kube-system namespace had failed to start as a result of me starting minikube with RBAC enabled. I fixed it by applying the config found here: kubernetes/minikube#1734 (comment)

Let me know if that works for you. We should certainly make this easier to debug as well.

[darren-fu]

Hi, @klingerf ,yes, I use minikube.

#start minikube
minikube start --extra-config=apiserver.Authorization.Mode=RBAC

# po status
darrenfu:~$ kubectl get po --all-namespaces
NAMESPACE     NAME                          READY     STATUS    RESTARTS   AGE
conduit       controller-598cb87c47-x4lnf   5/5       Running   10         41m
conduit       prometheus-66dcf6fb5-f5dpb    2/2       Running   4          41m
conduit       web-795cd7fdcd-xbrgs          1/1       Running   2          41m
kube-system   kube-addon-manager-minikube   1/1       Running   4          4d
kube-system   kube-dns-6fc954457d-bl5wm     3/3       Running   30         22d
kube-system   kubernetes-dashboard-gtlch    1/1       Running   4          4d
kube-system   kubernetes-dashboard-zvtjc    1/1       Running   10         22d
kube-system   storage-provisioner           1/1       Running   3          4d

I apply the RBAC config you mentioned here:
kubernetes/minikube#1734 (comment)
,BTW, I didn't change anything in this config, just touch a new file role.yaml

# apply RBAC
kubectl apply -f role.yaml

when I try to access dashboard, get same error(try to restart minikube, still get this error)

Error: 'EOF'
Trying to reach: 'http://172.17.0.5:8084/'

As a new user for K8S, I don't how to fix this now, can you give me some help? thx

[franziskagoltz]

Hi @darren-fu,

Sorry that you're still having issues with this. Can you try starting minikube without RBAC enabled? And when you reinstall conduit, do you still get the same error?

We want to check if this is related to your minikube and kubernetes setup or if the issue is related to RBAC use with conduit.

Thank you!

[darren-fu]

@hi @franziskagoltz,
I reinstall conduit as you said, unfortunately still can not open the dashboard page.

darrenfu:~$ minikube start --extra-config=apiserver.AuthorizationMode=AlwaysAllow
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

darrenfu:~$ kubectl get po --all-namespaces
NAMESPACE     NAME                          READY     STATUS    RESTARTS   AGE
conduit       controller-598cb87c47-x4lnf   5/5       Running   15         1d
conduit       prometheus-66dcf6fb5-f5dpb    2/2       Running   6          1d
conduit       web-795cd7fdcd-xbrgs          1/1       Running   3          1d
kube-system   kube-addon-manager-minikube   1/1       Running   5          5d
kube-system   kube-dns-6fc954457d-bl5wm     3/3       Running   33         23d
kube-system   kubernetes-dashboard-gtlch    1/1       Running   5          5d
kube-system   kubernetes-dashboard-zvtjc    1/1       Running   11         23d
kube-system   storage-provisioner           1/1       Running   4          5d

# delete conduit
darrenfu:~$ conduit install | kubectl delete -f -
namespace "conduit" deleted
serviceaccount "conduit-controller" deleted
clusterrole "conduit-controller" deleted
clusterrolebinding "conduit-controller" deleted
service "api" deleted
service "proxy-api" deleted
deployment "controller" deleted
service "web" deleted
service "prometheus" deleted

reinstall conduit

darrenfu:~$ conduit install | kubectl apply -f -
namespace "conduit" created
serviceaccount "conduit-controller" created
clusterrole "conduit-controller" created
clusterrolebinding "conduit-controller" created
service "api" created
service "proxy-api" created
deployment "controller" created
service "web" created
deployment "web" created
service "prometheus" created
deployment "prometheus" created
configmap "prometheus-config" created

darrenfu:~$ kubectl get po --all-namespaces
NAMESPACE     NAME                          READY     STATUS    RESTARTS   AGE
conduit       controller-598cb87c47-b6gkw   5/5       Running   0          31s
conduit       prometheus-66dcf6fb5-xz28k    2/2       Running   0          31s
conduit       web-7cf459866b-wj4mb          1/1       Running   0          31s
kube-system   kube-addon-manager-minikube   1/1       Running   5          5d
kube-system   kube-dns-6fc954457d-bl5wm     3/3       Running   33         23d
kube-system   kubernetes-dashboard-gtlch    1/1       Running   5          5d
kube-system   kubernetes-dashboard-zvtjc    1/1       Running   11         23d
kube-system   storage-provisioner           1/1       Running   4          5d

darrenfu:~$ conduit dashboard
Running `kubectl proxy -p 8001`
Starting to serve on 127.0.0.1:8001

Opening [http://127.0.0.1:8001/api/v1/namespaces/conduit/services/web:http/proxy/] in the default browser

get same error

Error: 'EOF'
Trying to reach: 'http://172.17.0.5:8084/'

again, version

darrenfu:~$ conduit version
Client version: v0.1.1
Server version: v0.1.1
darrenfu:~$ kubectl version --short
Client Version: v1.9.0
Server Version: v1.8.0
darrenfu:~$ minikube version
minikube version: v0.24.1

[franziskagoltz]

HI @darren-fu,

Thanks for the detailed set-up. It seems like the --extra-config=apiserver.AuthorizationMode=AlwaysAllow flag is causing some issues. I was able to reproduce your error when restarting minikube with the extra config.

In the meantime, if you run a minikube delete, (important step to make sure the extra-config flag isn't cached somewhere), followed by minikube start (without any extra flags), you should be able to continue the conduit install process and open the dashboard without any issues.

Let us know how this goes!

[darren-fu]

@franziskagoltz
Great, it works, now I can open the dashboard successfully after run minikube delete.
Thanks for the help, so I can try to add my service to Conduit.
I will close the issue.

[wmorgan]

It seems really strange to me that specifying AlwaysAllow would have any effect. According to https://kubernetes.io/docs/admin/kubelet-authentication-authorization/, AlwaysAllow should be the default behavior, so specifying it should be a noop...

[darren-fu]

@wmorgan @franziskagoltz
I think the problem is not AlwaysAllow.
Actually, I run minikube and Istio first, and I enable some alpha features
--extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota.
Then I use this kube cluster to run Conduit directly without runminikube delete.
FYI, I guess the kube cluster has some problem to run Conduit in this case, maybe some config cause this issue, hope this helps.

[briansmith]

It seems really strange to me that specifying AlwaysAllow would have any effect. According to https://kubernetes.io/docs/admin/kubelet-authentication-authorization/, AlwaysAllow should be the default behavior, so specifying it should be a noop...

See kubernetes/kubeadm#600 (comment).

[briansmith]

@darren-fu wrote:

--extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota

@darren-fu You're comment seems to be missing a closing quote. Could you clarify it please?

I reopened this. We should verify that Conduit actually works correctly in the configurations given by @darren-fu. Besides the one I quoted above in this comment:

• --extra-config=apiserver.AuthorizationMode=AlwaysAllow
• --extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota
• --extra-config=apiserver.AuthorizationMode=AlwaysAllow.

Hopefully this is just about RBAC-related issues that were fixed in more recent versions.

[darren-fu]

@briansmith yes, I missing a closing quote, rhe extra-config I use with minikube as follow:

--extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota"

[klingerf]

I did a bit of investigation for this issue. Here are the versions that I'm running:

$ minikube version
minikube version: v0.25.0
$ kubectl version --short
Client Version: v1.8.6
Server Version: v1.9.0
$ conduit version --client
Client version: v0.3.0

---

When I run:

$ minikube start --extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota"

The command completes, but the kubernetes cluster is unreachable via kubectl:

 $ kubectl get po
The connection to the server 192.168.99.104:8443 was refused - did you specify the right host or port?

So I was unable to test conduit with that configuration. Maybe there's some additional step I need to take to configure kubectl? Usually this happens automatically though.

---

When I run:

$ minikube start --extra-config=apiserver.AuthorizationMode=AlwaysAllow"

The command completes and the kubernetes cluster is reachable via kubectl.

Then I run:

$ conduit install | kubectl apply -f -

That command also completes successfully, and conduit check succeeds:

$ conduit check
kubernetes-api: can initialize the client.......................................[ok]
kubernetes-api: can query the Kubernetes API....................................[ok]
kubernetes-api: is running the minimum Kubernetes API version...................[ok]
conduit-api: can query the Conduit API..........................................[ok]
conduit-api[telemetry]: control plane can use telemetry service.................[ok]

Status check results are [ok]

When I run conduit dashboard, the dashboard opens successfully.

---

Based on these results, I'm concluding that:

• conduit-v0.3.0 installs with the correct RBAC permissions
• minikube-v0.25.0 fixed their kube-dns RBAC issues 🎉

I still don't know if conduit works with any of the other --extra-config options, since I could not get a cluster to start with those options set. @darren-fu Any chance you can do some additional investigation and report back?

[klingerf]

Am going to close this due to inactivity, please re-open if it's still an issue.