Memory leak using OnBasicAuthWrapper Hook about gmqtt

[xorduna]

Hi,

I digged deeper into the problem and it seems that there is a memory leak when the username / password is incorrect.

From what I have seen, there are two goroutines that never end:

128 @ 0x10096a524 0x10097ae18 0x100e93ebc 0x100e9bbf4 0x10099b744
#	0x100e93ebb	github.com/DrmagicE/gmqtt/server.(*client).writeLoop+0xab	/Users/xorduna/devel/myc-cloud/vendor/github.com/DrmagicE/gmqtt/server/client.go:308
#	0x100e9bbf3	github.com/DrmagicE/gmqtt/server.(*client).serve.func2+0x23	/Users/xorduna/devel/myc-cloud/vendor/github.com/DrmagicE/gmqtt/server/client.go:1458

128 @ 0x10096a524 0x10097bd2c 0x10097bd09 0x100996f08 0x1009b2500 0x100e9bad4 0x10099b744
#	0x100996f07	sync.runtime_Semacquire+0x27				/Users/xorduna/go/go1.19.1/src/runtime/sema.go:62
#	0x1009b24ff	sync.(*WaitGroup).Wait+0x7f				/Users/xorduna/go/go1.19.1/src/sync/waitgroup.go:139
#	0x100e9bad3	github.com/DrmagicE/gmqtt/server.(*client).serve+0x363	/Users/xorduna/devel/myc-cloud/vendor/github.com/DrmagicE/gmqtt/server/client.go:1494

When such error occurs, the goroutine serve() in server/client.go never ends, in fact it gets blocked by the client.wg.Wait()at the end.

I tried to call manually to client.Close() in case there is an error, with the following code:

	if client.err != nil {
		client.Close()
		return
	}

But then the this goroutine

go func() { //write
		client.writeLoop()
		client.wg.Done()
	}()

gets blocked in client.writeLoop().

From what I understand, the defer client.internalClose() should closed the channel where writeLoop() is listening, and also in the writeLoop() when an error is sent, this goroutine should return, but somehow it keeps running.

Can somebody give some light on this problem?

Thank you very much!

Xavi

[xorduna]

Hi,

It seems I got it fixed, adding this piece of code

	if client.err != nil {
		client.Close()
		close(client.close)
	}

just before client.wg.Wait() closes the client.close channel, which causes the writeLoop() to end and finally returning from serve().

I will deploy it in production and create a pull request, but it would be nice to get the opinion of the maintainer.

Thanks!

Xavi

[xorduna]

Opened a pull request: #178

[DrmagicE]

Oh, that is an impressive deep dive. I can reproduce your problem and you are right. Give me some time, I am looking into it.

[DrmagicE]

@xorduna After some digging, I think the root cause is that the server does not close the underlying TCP connection when the client gives an invalid username/password, or lets say, under any authentication failed situations.

client.serve() is blocked by:

gmqtt/server/client.go

Line 1473 in 52e1cba

readWg.Wait()

When clients providing invalid username/password:

• If the client closes the TCP connection after receiving the authentication failed Connack packet, then everything goes fine.
• if the client also does not close the connection, then the readLoop and writeLoop goroutines will be blocked.

This two gorouting will be released when the underlying TCP connection get closed.

So I think this PR #178 may not solve the problem.

I think always close the underlying TCP connection would be the solution, like this:

func (client *client) setError(err error) {
	client.errOnce.Do(func() {
		if err != nil && err != io.EOF {
			zaplog.Error("connection lost",
				zap.String("client_id", client.opts.ClientID),
				zap.String("remote_addr", client.rwc.RemoteAddr().String()),
				zap.Error(err))
			client.err = err
			if client.version == packets.Version5 {
				if code, ok := err.(*codes.Error); ok {
					if client.IsConnected() {
						// send Disconnect
						client.write(&packets.Disconnect{
							Version: packets.Version5,
							Code:    code.Code,
							Properties: &packets.Properties{
								ReasonString: code.ReasonString,
								User:         kvsToProperties(code.UserProperties),
							},
						})
					}
				}
			}
			_ = client.rwc.Close() // add this line
		}
	})
}

But sorry I don't have much time to test it carefully, would you please verify this and raise another PR if it works? Many thanks!

[xorduna]

HI @DrmagicE

Thanks for answering!

I tried your fix, but when executing _ = client.rwc.Close() the connection is closed before sending the error to the client and the client gets an EOF error instead of a Bad username or password.

What about executing close(client.close) instead?

Thanks!

Xavi

[DrmagicE]

@xorduna Hi, sorry for the late reply. I have submitted a pr which should fix the problem.

HI @DrmagicE

Thanks for answering!

I tried your fix, but when executing _ = client.rwc.Close() the connection is closed before sending the error to the client and the client gets an EOF error instead of a Bad username or password.

What about executing close(client.close) instead?

Thanks!

Xavi