"rke up" got "ssh: rejected: administratively prohibited (open failed)" about rke

if you run rke in centos7, you should not use the root user to open the ssh tunnel.you can try the following step to run rke
in all nodes:

1. update openssh to 7.4,and docker version v1.12.6
2. set "AllowTcpForwarding yes" "PermitTunnel yes" to /etc/ssh/sshd_config,and then restart sshd service
3. the host which run rke can ssh to all nodes without password
4. run: "groupadd docker" to create docker group,while docker group is not exist.
5. run: "useradd -g docker yourusername" to create yourusername user and set it's group to docker
6. set the docker.service's MountFlags=shared (vi /xxx/xxx/docker.service)
7. run:"su yourusername" to change current user,and then restart the docker service. so in the user yourusername session the docker.sock will be created in the path /var/run/docker.sock
8. in cluster.yml set the ssh user to yourusername(in setup hosts)

  nodes:
  - address: x.x.x.x
     ...
    user:  yourusername
  - address: x.x.x.x
     ...
    user:  yourusername

9. in cluster.yml set the kubelet to use the systemd cgroup(in setup hosts)

  services:
    kubelet:
      image: rancher/k8s:v1.8.3-rancher2
      extra_args: {"cgroup-driver":"systemd","fail-swap-on":"false"}

now you can run "rke -d up" to setup your k8s cluster.
if you meet "Failed to Save Kubernetes certificates: Timeout waiting for K8s to be ready" when running rke,your can see something here #121

[superseb]

Is this on CentOS/RHEL by any chance?

[galal-hussein]

Can you also make sure that the ssh user you are using has access to docker engine

[iyacontrol]

i also meet the same problem。i even try root user to ssh ，but the rke still failed。

[iyacontrol]

@nevermosby have you solved the problem?

[superseb]

@iyacontrol Please share used OS (cat /etc/os-release)

[iyacontrol]

@superseb
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[iyacontrol]

@superseb Linux 3.10.0-693.el7.x86_64

[nevermosby]

@superseb yeah, it is a centos system, details are listed beblow:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

@galal-hussein I used "root" to run the command so it can be sure I have the access to docker engine.

@iyacontrol I still have this issue with centos.

New progress here:
I tried ubuntu system instead and it did work.
So some different OS configuration can be the root cause.

[galal-hussein]

@nevermosby I will try to reproduce with CentOS 7 to see if the problem is related to specific OS

[galal-hussein]

@nevermosby @iyacontrol @superseb I was able to reproduce the issue, however i am not sure what is the root cause, CentOS seems to reject ssh local forwarding using the root:

- Unit session-33.scope has begun starting up.
Dec 04 23:32:46 hussein-centos-01 sshd[11192]: refused streamlocal port forward: originator  port 0, target /var/run/docker.sock
Dec 04 23:32:46 hussein-centos-01 sshd[11192]: pam_unix(sshd:session): session closed for user root

➜  ssh -nNT -L /tmp/docker.sock:/var/run/docker.sock  [email protected]
channel 1: open failed: administratively prohibited: open failed
channel 1: open failed: administratively prohibited: open failed

however it works for another users other than root, for a work around you can specify other users than root for now

[nevermosby]

@galal-hussein thx for feedback. On the same centos machine, I took ur advice and tried another user other than root but got same error.

[galal-hussein]

@nevermosby can you make sure that the ssh user has access to docker engine, you can add the user to docker group and use it in cluster.yml

[nevermosby]

@galal-hussein yes I am sure I have added the user to docker group and tried "docker push" command, it works.fine.

[galal-hussein]

@nevermosby Can you post the version of openssh currently installed on the centos server

[nevermosby]

@galal-hussein sorry for late reply. The version installed on my centos machine is :

ssh -V

OpenSSH_6.6.1p1,OpenSSL 1.0.1e-fips 11 Feb 2013

[niusmallnan]

@galal-hussein
It seems that pam_unix causes this issue.
Try this: https://serverfault.com/questions/416441/pam-module-causes-flurry-of-ssh-sessions

[galal-hussein]

@niusmallnan No i don't think this is the issue, i just tested with disabling PAM and same error, also i think not using PAM is not recommended in RHEL

[moelsayed]

@nevermosby You need to upgrade your openssh server to 7.x to get support for local stream forwarding.

[nevermosby]

Hi @moelsayed ，thx for remind and I just upgraded my openssh version to 7.4 and enabled "AllowTcpForwarding" for yes:

ssh -V
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017

However, I still got the same error. Maybe the forwarding configuration is not right. Could u give some advise for this?

[subhransusekhar]

Hi @luoshiqian thanks your solutions worked for me on centos 7.4

[galal-hussein]

@luoshiqian Thanks for the steps, OpenSSH doesn't allow ssh tunneling for root users for some reason, closing this issue and will track the root user bug in this issue #136

[ibre5041]

Tried to fix it by enabling the ssh config,AllowTcpForwarding yes, but failed.

Any suggestions?

I had same problem, found you that problem was on mine side. In authorizedkeys in key usage restrictions.