Server does not compile on Mac about ssh3

[francoismichel]

@jabedude @sebastianbauer20 You need to add the CGO_ENABLED=1 environment variable when compiling (I forgot to put that on the README):

CGO_ENABLED=1 go build -o ssh3-server cli/server/main.go

When the target is Linux, I think this is all that is needed (not sure if you cross-compile though).

Concerning running the server on BSD and MacOS, maybe I could figure out rapidly a mean to provide a server that runs on both if I just disable password-based auth on these two systems right now (which is discouraged anyway). If you're interested, I can at least do that, and it could be done rapidly.

And for the long-term, I just need a good way (without CGO if possible) to handle users for these different OSes. os/user provide de by Golang provides nearly everything, except authenticating the user with a password and getting the user's default shell.

[Peter2121]

@francoismichel
If you can fix building of server on FreeBSD just dropping password auth support - it would be an excellent start ;)
Even if you must force /bin/sh as shell.
Other fixes will be appreciated later ;)

I will be happy to test these fixes.

[Peter2121]

Both /usr/local/etc/ssl/cert.pem and /etc/ssl/cert.pem are present, the both files are relative symlinks to /usr/local/share/certs/ca-root-nss.crt:
../../usr/local/share/certs/ca-root-nss.crt for /etc/ssl/cert.pem
../../share/certs/ca-root-nss.crt for /usr/local/etc/ssl/cert.pem

I tested the connection from two different stations - exactly the same error.

Test connection with curl -v :

* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /usr/local/share/certs/ca-root-nss.crt
*  CApath: none

I tried to replace symlink in /usr/local/etc/ssl/ by the real file - it does not change anything.

I would propose you to create a Let's Encrypt certificate and check if it works for you ;)

BTW, I use resty to connect to some APIs on our prod server that uses the same type of certificate (wildcard Let's Encrypt) - it works without problems.

[Peter2121]

With the detailed logging the troubleshooting goes much faster! :)
It was just a typo in certificate name - the previous certificate (expired) was used by ssh3 on server side.

Sorry for noise, there is no certificate problem with a fresh version of certificate.
But the detailed error is appreciated, hope you'll not remove it in the release.

[francoismichel]

Sorry for the missing logging, it's being added progressively, and yes I'll keep it in the release. :-)
I'm changing /bin/bash by /bin/sh (until I correctly parse /etc/passwd) so that BSD support can be bundled in the next release candidate.

[francoismichel]

We currently support MacOS on the client only.
We might support MacOS for the server-side later when we add support for PAM

[jabedude]

just adding here, the server doesn't compile on macOS even when x-compiling for linux:

env GOOS=linux GOARCH=amd64 go build -o ssh3-server cli/server/main.go
go: downloading github.com/creack/pty v1.1.18
# ssh3/linux_server
linux_server/authorized_identities.go:39:50: undefined: linux_util.User
linux_server/authorized_identities.go:131:37: undefined: linux_util.User
linux_server/authorized_identities.go:171:53: undefined: linux_util.User
linux_server/auth.go:80:25: undefined: linux_util.UserPasswordAuthentication
linux_server/handlers.go:51:27: undefined: linux_util.GetUser

[TheFranconianCoder]

just adding here, the server doesn't compile on macOS even when x-compiling for linux:

env GOOS=linux GOARCH=amd64 go build -o ssh3-server cli/server/main.go
go: downloading github.com/creack/pty v1.1.18
# ssh3/linux_server
linux_server/authorized_identities.go:39:50: undefined: linux_util.User
linux_server/authorized_identities.go:131:37: undefined: linux_util.User
linux_server/authorized_identities.go:171:53: undefined: linux_util.User
linux_server/auth.go:80:25: undefined: linux_util.UserPasswordAuthentication
linux_server/handlers.go:51:27: undefined: linux_util.GetUser

Same for building on Windows or Linux (WSL) with GOOS and GOARCH set.

[Peter2121]

Same problem on FreeBSD :(

[TheFranconianCoder]

Makes sense. Got a cross-compiled version from Windows to Linux ARM64 thank to zig:

CGO_ENABLED=1 CC="zig cc -target aarch64-linux-musl" CXX="zig c++ -target aarch64-linux-musl" GOOS=linux GOARCH=arm64 go build -o ssh3-server cli/server/main.go

[francoismichel]

You ask, I deliver. :-)

#56 adds support for other platforms in ssh3-server. The price to pay is disabled user authentication.

I have not tested it yet because I don't have access to those systems right now, but your can already try it out and tell me if it works.

[Peter2121]

Thanks!!
I could build and run ssh3-server on my FreeBSD 13.2 host.
I cannot connect to it with ssh3 client, but maybe this is a network problem, I'll check it shortly and report here.

[francoismichel]

Allright, let me know, I'll probably setup a VM with FreeBSD at some point. It would be nice to run the integration tests on such systems as well, I'll look into it.

[Peter2121]

I solved the network problem, but I still cannot connect to ssh3-server running on a host.
I use the following syntax:

./ssh3 -v -insecure [email protected]:44443/ssh3 -privkey /home/peter/.ssh/id_ssh3
DBG QUIC handshake complete
ERR no suitable identity found

The same key works correctly with 'classic' ssh:

ssh -i /home/peter/.ssh/id_ssh3 [email protected]

The server is started with -v argument as well, but no errors are logged there.

I tried to use ssh-agent - I've got the same error (and it works with 'classic' ssh as well).

[francoismichel]

I think the problem comes from the fact that you put the -privkey arg after the server URL

Could you try changing your command like this and let me know ?

./ssh3 -v -insecure -privkey /home/peter/.ssh/id_ssh3 [email protected]:44443/ssh3

[Peter2121]

Ah, yes, indeed!

I could connect to the host, but there are some glitches though ;)

1. Some errors about keys at the server side:

DBG received request from User-Agent SSH 3.0 francoismichel/ssh3 0.1.3 (major 0, minor 1, patch 3)
DBG parsing ssh authorized key
DBG parsing ssh-rsa identity
DBG parsing ssh authorized key
DBG parsing ssh-rsa identity
DBG parsing ssh authorized key
DBG parsing ssh-rsa identity
DBG parsing ssh authorized key
DBG parsing ssh-rsa identity
DBG parsing ssh authorized key
DBG parsing ssh-rsa identity
ERR invalid private key token: token signature is invalid: crypto/rsa: verification error
ERR invalid private key token: token signature is invalid: crypto/rsa: verification error
ERR invalid private key token: token signature is invalid: crypto/rsa: verification error
ERR invalid private key token: token signature is invalid: crypto/rsa: verification error
INF got request: method: CONNECT, URL: https://mail.flytrace.com:44443/ssh3?user=root
DBG the connection was closed by the application: Application error 0x100 (remote)
INF conversation canceled for conversation id HwGwnVxNe3l8NAcwFIH1Ve3RHfwgoP0eQyU7GI4Uxr4=, user root

Probably, it does not like some keys, but from which file?? Normal sshd does not indicate anything wrong.

2. Problem of shell

It always searches /bin/bash to open a session. In FreeBSD it does not exist, I needed to create a symlink. Probably, you need to parse /etc/password to get user's shell.

3. Problem of environment

Probably, related to (3) - login script was not executed, PATH variable is not correct (/usr/local/bin and /usr/local/sbin must be in PATH).

4. Problem of x509 server certificate

The certificate of my server is wildcard issued by Let's Encrypt (DNS validation). It is a valid certificate, I use it for different network services. But ssh3 is not happy with it:

ERR the peer provided an unknown, insecure certificate, that is not self-signed: x509: invalid signature: parent certificate cannot sign this kind of certificate

I suppose that the same problems will be present on Mac OS X, but I cannot check it at the moment (my MacBook is waiting for a reparation). If I find a Mac OS X to test - I'll put the results here.

[francoismichel]

Thanks for the feedback ! Happy that you could at least connect, let's fix the other problems.

1. Some errors about keys at the server side:

These "errors" appear each time an authorized identity does not verify correctly the user-auth material (privkey signature in your case). So they will happen a lot when you have several lines in your .ssh/authorized_keys or .ssh3/authorized_identities. I should put these logs at Debug ot Trace level.

2. Problem of shell

The Go library already parses /etc/passwd, but they do not use the shell field, it is just discarded. :-( I really wished Golang would parse the shell as well so that I would just rely on that and get rid of more CGO in on the Linux server that currently calls getpwnam. I'll probably have to parse it myself, and it will probably be a copy paste of os/user/lookup_unix.go.

What would be a good temporary fix that works for most non-linux unix systems until I parse /etc/passwd ?

3. Problem of environment

The shell shoud now be run in login mode since effbb00.
But for the PATH variable, I think we need to look at this for FreeBSD: https://man.freebsd.org/cgi/man.cgi?query=setusercontext&sektion=3&format=html.

The default behaviour of OpenSSH is to set the path to be equal to "/usr/bin:/bin:/usr/sbin:/sbin" and we do the same right now, so we shoud try to get the PATH information from another source in the near future (that could be another Github issue though).

4. Problem of x509 server certificate

Could you provide me with all the client log ? If the certificate is a valid LetsEncrypt one, then the first QUIC connection should succeed and you should not see that message. So it might be related to where the certificates are stored on your machine and where the Go TLS library is fetching them. This is probably linked to #40. I am curious to know where they are stored and why the TLS library does not find them, so let me know. :-)

[Peter2121]

What would be a good temporary fix that works for most non-linux unix systems until I parse /etc/passwd ?

Using by default /bin/sh seems to be the best option for me. It is present everywhere, and since FreeBSD 14 it is the default root shell.

we should try to get the PATH information from another source in the near future

The problem will be solved if shell-specific login script will be executed.

Could you provide me with all the client log

2:20PM DBG dialing QUIC host at mail.xxxxxxxxx.com:44443
2023/12/20 14:20:39 failed to increase receive buffer size (wanted: 2048 kiB, got 41 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2:20PM ERR the peer provided an unknown, insecure certificate, that is not self-signed: x509: invalid signature: parent certificate cannot sign this kind of certificate

[francoismichel]

Here are the file and dir paths golang checks to find your root certs.

// Possible certificate files; stop after finding one.
var certFiles = []string{
	"/usr/local/etc/ssl/cert.pem",            // FreeBSD
	"/etc/ssl/cert.pem",                      // OpenBSD
	"/usr/local/share/certs/ca-root-nss.crt", // DragonFly
	"/etc/openssl/certs/ca-certificates.crt", // NetBSD
}

// Possible directories with certificate files; all will be read.
var certDirectories = []string{
	"/etc/ssl/certs",         // FreeBSD 12.2+
	"/usr/local/share/certs", // FreeBSD
	"/etc/openssl/certs",     // NetBSD
}

Does it match with your system certificates location ?

If you curl -v a website (preferably the one with the certificate that does not work), what are the CAfile and CApath variables pointing to ?

[francoismichel]

I am using a letsencrypt certificates on my servers as well, but not wildcard. I just generated a wildcard one with Let's Encrypt and it seems to work correctly from my Fedora client.

I added a bit of logging on the client when the first QUIC connection attempt fails (which should only arrive when stumbling upon invalid certs or self-signed certs not present in the known hosts yet).
You should now see the following error when the client is run in verbose mode: received QUIC crypto error on first connection attempt: [...], I'm interested in what [...] says. :-)

[Peter2121]

Is there a hard technical reason to use 1.21 version of Go for ssh3 (go.mod)? The support of Mac OS X 10.14 is dropped in this version of Go. I have a computer under this version where I can test ssh3, but only Go 1.20 can be installed on this station.

[Peter2121]

So, changing go version to 1.20 I could build ssh3 client and server on Mac OS X 10.14.
The server does not work correctly under a 'normal' user though. This is a Mac OS X permissions problem, there is an error:

fork/exec /bin/sh: operation not permitted

The server works correctly started under root account, I can connect.

[francoismichel]

Is there a hard technical reason to use 1.21 version of Go for ssh3 (go.mod)? The support of Mac OS X 10.14 is dropped in this version of Go. I have a computer under this version where I can test ssh3, but only Go 1.20 can be installed on this station.

No, there is none, we just want to avoid having too much maintenance with go versions that differ too much, and be able to benefit from go improvements faster. I am not strongly against allowing go version n-2, but that can also be done the same way as you did. Additionally, we provide release binaries (with the exception here that #56 is not merged yet and therefore not part of the releases).

The server does not work correctly under a 'normal' user though. This is a Mac OS X permissions problem, there is an error:

fork/exec /bin/sh: operation not permitted

I guess you tried to connect to the same user as the one running the server ?
I have no MacOS device to test it yet, but wouldn't it related to the apple.com.quarantine attribute preventing from executing some programs ?

[Peter2121]

Additionally, we provide release binaries

OK if these binaries will be compatible with 10.13 and 10.14. I don't see any reason to support older versions, but 10.13 and 10.14 still remains popular as there are many changes (not always positive) in 10.15.

I guess you tried to connect to the same user as the one running the server ?

Yes, I tried the connection with the same username

I have no MacOS device to test it yet, but wouldn't it related to the apple.com.quarantine attribute preventing from executing some programs ?

There are many noise about this problem in Internet, I did not dig so deep to really find the correct fix. It seems, it is not related to the ssh3 binary itself.

[francoismichel]

I am okay to lower the minimum required version right now, quite a few Go projects still support it.
But at some point we'll have to move on from it, until then I'll mostly provide support for 1.21 and 1.22. :-)

Thanks for all the feedback and help !

I think I can close the issue and merge #56.