lesismal / nbio

Pure Go 1000k+ connections solution, support tls/http1.x/websocket and basically compatible with net/http, with high-performance and low memory cost, non-blocking, event-driven, easy-to-use.

NBIO - NON-BLOCKING IO

Contents

• NBIO - NON-BLOCKING IO
  • Contents
  • Features
    • Cross Platform
    • Protocols Supported
    • Interfaces
  • Quick Start
  • Examples
    • TCP Echo Examples
    • UDP Echo Examples
    • TLS Examples
    • HTTP Examples
    • HTTPS Examples
    • Websocket Examples
    • Websocket TLS Examples
    • Use With Other STD Based Frameworkds
    • More Examples
  • 1M Websocket Connections Benchmark
  • Magics For HTTP and Websocket
    • Different IOMod
    • Using Websocket With Std Server
  • Credits
  • Contributors
  • Star History

Features

Cross Platform

• Linux: Epoll with LT/ET/ET+ONESHOT supported, LT as default
• BSD(MacOS): Kqueue
• Windows: Based on std net, for debugging only

Protocols Supported

• TCP/UDP/Unix Socket supported
• TLS supported
• HTTP/HTTPS 1.x supported
• Websocket supported, Passes the Autobahn Test Suite, OnOpen/OnMessage/OnClose order guaranteed

Interfaces

• Implements a non-blocking net.Conn(except windows)
• SetDeadline/SetReadDeadline/SetWriteDeadline supported
• Concurrent Write/Close supported(both nbio.Conn and nbio/nbhttp/websocket.Conn)

Quick Start

package main

import (
	"log"

	"github.com/lesismal/nbio"
)

func main() {
	engine := nbio.NewEngine(nbio.Config{
		Network:            "tcp",//"udp", "unix"
		Addrs:              []string{":8888"},
		MaxWriteBufferSize: 6 * 1024 * 1024,
	})

	// hanlde new connection
	engine.OnOpen(func(c *nbio.Conn) {
		log.Println("OnOpen:", c.RemoteAddr().String())
	})
	// hanlde connection closed
	engine.OnClose(func(c *nbio.Conn, err error) {
		log.Println("OnClose:", c.RemoteAddr().String(), err)
	})
	// handle data
	engine.OnData(func(c *nbio.Conn, data []byte) {
		c.Write(append([]byte{}, data...))
	})

	err := engine.Start()
	if err != nil {
		log.Fatalf("nbio.Start failed: %v\n", err)
		return
	}
	defer engine.Stop()

	<-make(chan int)
}

Examples

TCP Echo Examples

• echo-server
• echo-client

UDP Echo Examples

• udp-server
• udp-client

TLS Examples

• tls-server
• tls-client

HTTP Examples

• http-server
• http-client

HTTPS Examples

• http-tls_server
• visit: https://localhost:8888/echo

Websocket Examples

• websocket-server
• websocket-client

Websocket TLS Examples

• websocket-tls-server
• websocket-tls-client

Use With Other STD Based Frameworkds

• echo-http-and-websocket-server
• gin-http-and-websocket-server
• go-chi-http-and-websocket-server

More Examples

• nbio-examples

1M Websocket Connections Benchmark

For more details: go-websocket-benchmark

# lsb_release -a
LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

# free
              total        used        free      shared  buff/cache   available
Mem:       24969564    15656352     3422212        1880     5891000     8899604
Swap:             0           0           0

# cat /proc/cpuinfo | grep processor
processor       : 0
processor       : 1
processor       : 2
processor       : 3
processor       : 4
processor       : 5
processor       : 6
processor       : 7
processor       : 8
processor       : 9
processor       : 10
processor       : 11
processor       : 12
processor       : 13
processor       : 14
processor       : 15


# taskset
run nbio_nonblocking server on cpu 0-7

--------------------------------------------------------------
BenchType  : BenchEcho
Framework  : nbio_nonblocking
TPS        : 104713
EER        : 280.33
Min        : 56.90us
Avg        : 95.36ms
Max        : 2.29s
TP50       : 62.82ms
TP75       : 65.38ms
TP90       : 89.38ms
TP95       : 409.55ms
TP99       : 637.95ms
Used       : 47.75s
Total      : 5000000
Success    : 5000000
Failed     : 0
Conns      : 1000000
Concurrency: 10000
Payload    : 1024
CPU Min    : 0.00%
CPU Avg    : 373.53%
CPU Max    : 602.33%
MEM Min    : 978.70M
MEM Avg    : 979.88M
MEM Max    : 981.14M
--------------------------------------------------------------

Magics For HTTP and Websocket

Different IOMod

IOMod	Remarks
IOModNonBlocking	There's no difference between this IOMod and the old version with no IOMod. All the connections will be handled by poller.
IOModBlocking	All the connections will be handled by at least one goroutine, for websocket, we can set Upgrader.BlockingModAsyncWrite=true to handle writing with a separated goroutine and then avoid Head-of-line blocking on broadcasting scenarios.
IOModMixed	We set the Engine.MaxBlockingOnline, if the online num is smaller than it, the new connection will be handled by single goroutine as IOModBlocking, else the new connection will be handled by poller.

The IOModBlocking aims to improve the performance for low online service, it runs faster than std. The IOModMixed aims to keep a balance between performance and cpu/mem cost in different scenarios: when there are not too many online connections, it performs better than std, or else it can serve lots of online connections and keep healthy.

Using Websocket With Std Server

package main

import (
	"fmt"
	"net/http"

	"github.com/lesismal/nbio/nbhttp/websocket"
)

var (
	upgrader = newUpgrader()
)

func newUpgrader() *websocket.Upgrader {
	u := websocket.NewUpgrader()
	u.OnOpen(func(c *websocket.Conn) {
		// echo
		fmt.Println("OnOpen:", c.RemoteAddr().String())
	})
	u.OnMessage(func(c *websocket.Conn, messageType websocket.MessageType, data []byte) {
		// echo
		fmt.Println("OnMessage:", messageType, string(data))
		c.WriteMessage(messageType, data)
	})
	u.OnClose(func(c *websocket.Conn, err error) {
		fmt.Println("OnClose:", c.RemoteAddr().String(), err)
	})
	return u
}

func onWebsocket(w http.ResponseWriter, r *http.Request) {
	conn, err := upgrader.Upgrade(w, r, nil)
	if err != nil {
		panic(err)
	}
	fmt.Println("Upgraded:", conn.RemoteAddr().String())
}

func main() {
	mux := &http.ServeMux{}
	mux.HandleFunc("/ws", onWebsocket)
	server := http.Server{
		Addr:    "localhost:8080",
		Handler: mux,
	}
	fmt.Println("server exit:", server.ListenAndServe())
}

Credits

• xtaci/gaio
• gorilla/websocket
• crossbario/autobahn

Contributors

Thanks Everyone:

• acgreek
• acsecureworks
• arunsathiya
• guonaihong
• isletnet
• liwnn
• manjun21
• om26er
• rfyiamcool
• sunny352
• sunvim
• wuqinqiang
• wziww
• youzhixiaomutou
• zbh255
• IceflowRE
• YanKawaYu

Star History

nbio's Issues

Minimal websocket example

Hi,

I'm trying to write a minimal websocket example but I can't seem to get it working.

Here's the code:

package main

import (
	"net"
	"net/http"

	"github.com/lesismal/nbio/nbhttp/websocket"
)

func main() {
	var wsHandler WSHandler
	server := &http.Server{Addr: "0.0.0.0:8081", Handler: wsHandler}

	ln, err := net.Listen("tcp", "0.0.0.0:8081")
	if err != nil {
		panic(err)
	}

	err = server.Serve(ln)
	if err != nil {
		panic(err)
	}
}

type WSHandler struct {
}

func (h WSHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	defer func() {
		if err := recover(); err != nil {
			println(err)
		}
	}()

	upgrader := websocket.NewUpgrader()
	upgrader.CheckOrigin = func(r *http.Request) bool {
		return true
	}

	conn, err := upgrader.Upgrade(w, r, nil)
	if err != nil {
		println(err)
		return
	}
	wsConn := conn.(*websocket.Conn)
	// wsConn.OnMessage(func(c *websocket.Conn, messageType websocket.MessageType, data []byte) {
	// 	// echo
	// 	c.WriteMessage(messageType, data)
	// })
	wsConn.OnClose(func(c *websocket.Conn, err error) {
		println("OnClose")
	})
	println("OnOpen")
}

When I try to establish a connection, it prints:

2021/09/03 02:01:37 http: response.WriteHeader on hijacked connection from github.com/lesismal/nbio/nbhttp/websocket.(*Upgrader).returnError (upgrader.go:517)
2021/09/03 02:01:37 http: response.Write on hijacked connection from fmt.Fprintln (print.go:265)
(0xa5cfc0,0xc000142f30)

Also, the following code gives error message "wsConn.OnMessage undefined (type *websocket.Conn has no field or method OnMessage)"

	wsConn.OnMessage(func(c *websocket.Conn, messageType websocket.MessageType, data []byte) {
		// echo
		c.WriteMessage(messageType, data)
	})

I want to write proxy server like shadowsockslibev,low memory use 6~7MB?

source:

import (
"fmt"

"github.com/lesismal/nbio"
"strconv"

)

type niosess struct {
isserver bool
step int
parterconn *nbio.Conn
}

func closeConn(c *nbio.Conn){

if session := c.Session(); session != nil {

	sess := session.(*niosess)
	fmt.Println("RemoteAddr ",c.RemoteAddr(),sess.isserver)

	if sess.isserver {
		if sess.parterconn != nil {
			sess.parterconn.SetSession(nil)
			sess.parterconn.Close()
		}

		c.SetSession(nil)
		c.Close()
	} else {
		fmt.Println("is remote close")
		closeConn(sess.parterconn)
	}

}else{
	fmt.Println("non sess close")
	c.Close()
}

}

func Startserver(){

g := nbio.NewGopher(nbio.Config{
	Network:            "tcp",
	Addrs:              []string{":8888"},
	MaxWriteBufferSize: 4 * 1024,
	ReadBufferSize:2048,
	NPoller:1,
	MinConnCacheSize:128,
	Backlog:128,
})

//isClient := true

g.OnOpen(func(c *nbio.Conn)  {
	fmt.Println("OnOpen:", c.RemoteAddr().String())
	if session := c.Session(); session != nil {
		// remote open

	} else{ // server
		sess := &niosess{
			isserver:true,
		}
		c.SetSession(sess)
	}
})

g.OnClose(func(c *nbio.Conn, err error){
	fmt.Println("OnClose:", c.RemoteAddr().String())
	closeConn(c)
})


g.OnData(func(c *nbio.Conn, data []byte) {
	//	c.Write(append([]byte{}, data...))

	if session := c.Session(); session != nil {

		sess := session.(*niosess)
		if sess.isserver {

			fmt.Println("isserver len",len(data),sess.step)

			if sess.step == 0 {
				resp := []byte{5,0}
				c.Write(resp)
				sess.step = 1
			} else if sess.step == 1 {
				data[1] = 0
				nr := len(data)
				hostb := data[3:nr-2]
				portb := data[nr-2:nr]

				isIp := data[3] == 1

				host := ""
				if !isIp {
					host = string(hostb[1:])
				} else{
					host = fmt.Sprintf("%d.%d.%d.%d", hostb[1], hostb[2],hostb[3],hostb[4])
				}

				p := int(portb[0]) * 256 + int(portb[1])


				addr := host+":"+strconv.Itoa(p)
				fmt.Println("host:port",host+":"+strconv.Itoa(p))

			        nbConn, err := nbio.Dial("tcp", addr )
			        if err != nil {
				        fmt.Println("Dial failed: ", err)
				        closeConn(c)
				        return
			        }
    
			        remotesess := &niosess{
				        isserver:false,
				        parterconn:c,
				        step: 5,
			        }
    
			        sess.parterconn = nbConn
			        nbConn.SetSession(remotesess)
			        c.SetSession(sess)
    
			        g.AddConn(nbConn)

				c.Write(data)
				sess.step = 2




			} else {
				if sess.parterconn != nil {
					sess.parterconn.Write(data)
				} else{
					closeConn(c)
				}
			}


		}else{ // is remote

			fmt.Println("remote len",len(data))

			if sess.parterconn != nil {
				sess.parterconn.Write(data)
			} else {
				fmt.Println("parterconn nil")
				closeConn(c)
			}

		}

	} else {
		fmt.Println("non sess recv",data)
	}


})

err := g.Start()
if err != nil {
	fmt.Printf("nbio.Start failed: %v\n", err)
	return
}
defer g.Stop()

g.Wait()

}

this code use memory over 12MB. how to upgrade to use low memory?

WebSocket吞吐量

有优化过WebSocket小包吞吐么? 什么样的配置, 什么样的参数, 大概能跑多少PPS?

autobahn test failures on tls

Here is the latest test report. We are all good when not running through tls. A few tests in section section 9 fail

report-2021-07-03.tar.gz

incompatible with websocketpp

see sample client #22

I ran this test on a macbook but should be no different on linux

details optimization - long term

if possible:

• nbhttp.Parser/ServerProcessor - being testing on poolopt branch
• nbhttp/websocket.Upgrader - being testing on poolopt branch
• replace llib tls.Conn's bytes.Buffer with new implementation with pool - being testing on poolopt branch

Feature request. Websocket compression

is websocket.Conn.SetSession goroutine safe

If I call websocket.Conn.SetSession() in the websocket.Upgrade.openHandler, is that guarantied to be safe; IE websocket.Update.onMessageHandler will not be called until after the openHandler completes such that if in onMessagHandler I call websocket.Conn.Session() I don't worry have to worry about it not being set yet?

It would be nice if such contracts were clearly documented in function header comments. I use VSCode, and the code completion provides the function header comments in the completion popup box so that if the comments stated it, I wouldn't have to review the code.

Please consider going through the code and adding more description to functions; their expected use case and any special consideration/protections/contracts.

Very high CPU consumption when the process is left running after calling .Stop()

I am using nbio on my server.
For reasons, I need to call "nbioServer.Stop()" and leave the server application itself running. Basically what usually (read: with other websocket servers) happens is that the websocket server is closed (so no new connection is served), but the application itself is left running.
What I could observe is that the CPU usage quickly rose (in the 2-3 minutes following the .Stop() call) from the normal 8-10% to 38% and remained there until I finally killed the application, then it quickly went back down.

The following was printed, as expected. No other output followed, yet the CPU increase made me suspicious.

2021/08/17 12:55:04.754 [ERR] Poller[NB_LISTENER_0] Accept failed: accept tcp [::]:PORT: use of closed network connection, exit...

2021/08/17 12:55:04.755 [INF] Gopher[NB] stop

What is going on after .Stop() is called?

Edit:
Go 1.15

relevant libraries, from go.mod
github.com/lesismal/llib v1.0.9 // indirect
github.com/lesismal/nbio v1.1.20 // indirect

Comparison with other event based network libs

Thanks for this lib. Do you have any performance comparison with other frameworks?

Frameworks like:

• evio
• gnet

this may help:
example gnet-benchmarks

很好奇这里为什么注释掉runtime.Gosched()

首先谢谢您的开源框架
在这个地方：

nbio/poller_epoll.go

Line 159 in db13c52

// runtime.Gosched()

我看到代码中将runtime.Gosched()注释掉了
了gnet、gev、netpoll的代码， 都做了类似的处理， 动态地改变参数msec ，并且调用runtime.Gosched()主动让出时间片
字节的开发团队认为：
“当无事件触发，调整 msec=-1 时，直接 continue 会立即再次执行 EpollWait，而由于无事件，msec=-1，当前 goroutine 会 block 并被 P 切换。但是被动切换效率较低，如果我们在 continue 前主动为 P 切换 goroutine，则可以节约时间。”
很好奇为什么作者在这里注释掉了runtime.Gosched()，有什么其他的考虑吗？
（我在2核的服务器上做了benchmark，两者没有太多区别。但是2核的实验结果可能没有多大参考价值）

republished v1.0.0 throws checksum error

looks like release v1.0.0 has been overridden which causes following error during go mod tidy.
somebody has pulled old v1.0.0 which immediately cached by goproxy

mind making v1.0.1 to fix this?

go: github.com/lesismal/[email protected]: verifying go.mod: checksum mismatch
	downloaded: h1:EDP2ruaH9wyJNSsNjWXrYicl7p3gXyA9qhUaz0qet9U=
	sum.golang.org: h1:PYXWhrALQ3/6gWsJuH7LMR3ii0juFFQzj0g/28EOrSI=

a lot of failures when tested again autobahn test suite

Test the websocket_tls server

https://github.com/crossbario/autobahn-testsuite

unzip and open index.html

autobahn-testreport.zip

Should I receive one OnClose for every connection created and closed?

Hi. I am using nbio hard and it is performing very well. But here might be a small issue: I depend on OnClose for properly reverting things to the initial state. For example, I create a bunch of connections (say, 20000) using a synthetic test tool. And then I close all of these connections at once. In such case I seem to often end up with a small percentage of connections not receiving an OnClose event. The problem seems to occur especially when I just kill the client tool, ending all connections without properly closing them down. The vast majority of the connections does still receive it's OnClose event on the server side. But some (say, 279 out of 20000) do not. This could very well be an oddity in my app. But I can't find the issue. NOTE: I have print statements in OnClose. As a result it often takes some significant amount of time to gp through all OnClose events. Could this be causing the issue?
Same context: What do the different OnClose error messages mean: 1. no error, 2. "EOF", 3. "broken pipe" and 4. "connection timed out". What can I derive from these error messages? Do any of them demand special (different) action on my part? Currently I am NOT closing any connection for which I receive an OnClose event (assuming they are closed already). Is this the correct handling of it?

tls测试失败

用了示例中的tls server和client，发现无法正常通信，客户端收不到消息，一直阻塞

projects using nbio

Please leave a message with information about your projects that are using nbio if possible.

tls incompatible with winhttp

I used visual studio 2019 and ran this against websocket_tls server . If I simply change the port to 28000 and replace WINHTTP_FLAG_SECURE with 0, this code is able to connect to the non-tls websocket example

// winhttpExample.cpp : This file contains the 'main' function. Program execution begins and ends there.
//
#pragma comment(lib, "winhttp")

#include <Windows.h>
#include <WinHttp.h>
#include <stdio.h>

int __cdecl wmain()
{
	DWORD dwError = ERROR_SUCCESS;
	BOOL fStatus = FALSE;
	HINTERNET hSessionHandle = NULL;
	HINTERNET hConnectionHandle = NULL;
	HINTERNET hRequestHandle = NULL;
	HINTERNET hWebSocketHandle = NULL;
	BYTE rgbCloseReasonBuffer[123];
	BYTE rgbBuffer[1024];
	BYTE* pbCurrentBufferPointer = rgbBuffer;
	DWORD dwBufferLength = ARRAYSIZE(rgbBuffer);
	DWORD dwBytesTransferred = 0;
	DWORD dwCloseReasonLength = 0;
	USHORT usStatus = 0;
	WINHTTP_WEB_SOCKET_BUFFER_TYPE eBufferType;
	INTERNET_PORT Port = 8888;
	const WCHAR* pcwszServerName = L"localhost";
	const WCHAR* pcwszPath = L"/ws";
	const WCHAR* pcwszMessage = L"Hello world";
	const DWORD cdwMessageLength = ARRAYSIZE(L"Hello world") * sizeof(WCHAR);

	wprintf(L"program started\n");
	//
	// Create session, connection and request handles.
	//
	hSessionHandle = WinHttpOpen(L"WebSocket sample",
		WINHTTP_ACCESS_TYPE_DEFAULT_PROXY,
		NULL,
		NULL,
		0);
	if (hSessionHandle == NULL)
	{
		dwError = GetLastError();
		goto quit;
	}
	wprintf(L"session handle constructed\n");

	hConnectionHandle = WinHttpConnect(hSessionHandle,
		pcwszServerName,
		Port,
		0);
	if (hConnectionHandle == NULL)
	{
		dwError = GetLastError();
		wprintf(L"connection failed %d\n", dwError);
		goto quit;
	}
	wprintf(L"connected\n");

	hRequestHandle = WinHttpOpenRequest(hConnectionHandle,
		L"GET",
		pcwszPath,
		NULL,
		NULL,
		NULL,
		WINHTTP_FLAG_SECURE);
		// 0);
	if (hRequestHandle == NULL)
	{
		dwError = GetLastError();
		goto quit;
	}

	wprintf(L"open request successs\n");
	//
	// Request protocol upgrade from http to websocket.
	//
#pragma prefast(suppress:6387, "WINHTTP_OPTION_UPGRADE_TO_WEB_SOCKET does not take any arguments.")
	fStatus = WinHttpSetOption(hRequestHandle,
		WINHTTP_OPTION_UPGRADE_TO_WEB_SOCKET,
		NULL,
		0);
	if (!fStatus)
	{
		dwError = GetLastError();
		goto quit;
	}

	//
	// Perform websocket handshake by sending a request and receiving server's response.
	// Application may specify additional headers if needed.
	//

	fStatus = WinHttpSendRequest(hRequestHandle,
		WINHTTP_NO_ADDITIONAL_HEADERS,
		0,
		NULL,
		0,
		0,
		0);
	if (!fStatus)
	{
		dwError = GetLastError();
		goto quit;
	}
	wprintf(L"upgraded \n");

	fStatus = WinHttpReceiveResponse(hRequestHandle, 0);
	if (!fStatus)
	{
		dwError = GetLastError();
		goto quit;
	}
	wprintf(L"receive response\n");

	//
	// Application should check what is the HTTP status code returned by the server and behave accordingly.
	// WinHttpWebSocketCompleteUpgrade will fail if the HTTP status code is different than 101.
	//

	hWebSocketHandle = WinHttpWebSocketCompleteUpgrade(hRequestHandle, NULL);
	if (hWebSocketHandle == NULL)
	{
		dwError = GetLastError();
		goto quit;
	}
	wprintf(L"upgrade complete\n");

	//
	// The request handle is not needed anymore. From now on we will use the websocket handle.
	//

	WinHttpCloseHandle(hRequestHandle);
	hRequestHandle = NULL;

	wprintf(L"Succesfully upgraded to websocket protocol\n");

	//
	// Send and receive data on the websocket protocol.
	//

	dwError = WinHttpWebSocketSend(hWebSocketHandle,
		WINHTTP_WEB_SOCKET_BINARY_MESSAGE_BUFFER_TYPE,
		(PVOID)pcwszMessage,
		cdwMessageLength);
	if (dwError != ERROR_SUCCESS)
	{
		goto quit;
	}

	wprintf(L"Sent message to the server: '%s'\n", pcwszMessage);

	do
	{
		if (dwBufferLength == 0)
		{
			dwError = ERROR_NOT_ENOUGH_MEMORY;
			goto quit;
		}

		dwError = WinHttpWebSocketReceive(hWebSocketHandle,
			pbCurrentBufferPointer,
			dwBufferLength,
			&dwBytesTransferred,
			&eBufferType);
		if (dwError != ERROR_SUCCESS)
		{
			goto quit;
		}

		//
		// If we receive just part of the message restart the receive operation.
		//

		pbCurrentBufferPointer += dwBytesTransferred;
		dwBufferLength -= dwBytesTransferred;
	} while (eBufferType == WINHTTP_WEB_SOCKET_BINARY_FRAGMENT_BUFFER_TYPE);

	//
	// We expected server just to echo single binary message.
	//

	if (eBufferType != WINHTTP_WEB_SOCKET_BINARY_MESSAGE_BUFFER_TYPE)
	{
		wprintf(L"Unexpected buffer type\n");
		dwError = ERROR_INVALID_PARAMETER;
		goto quit;
	}

	wprintf(L"Received message from the server: '%.*s'\n", dwBufferLength, (WCHAR*)rgbBuffer);

	//
	// Gracefully close the connection.
	//

	dwError = WinHttpWebSocketClose(hWebSocketHandle,
		WINHTTP_WEB_SOCKET_SUCCESS_CLOSE_STATUS,
		NULL,
		0);
	if (dwError != ERROR_SUCCESS)
	{
		goto quit;
	}

	//
	// Check close status returned by the server.
	//

	dwError = WinHttpWebSocketQueryCloseStatus(hWebSocketHandle,
		&usStatus,
		rgbCloseReasonBuffer,
		ARRAYSIZE(rgbCloseReasonBuffer),
		&dwCloseReasonLength);
	if (dwError != ERROR_SUCCESS)
	{
		goto quit;
	}

	wprintf(L"The server closed the connection with status code: '%d' and reason: '%.*S'\n",
		(int)usStatus,
		dwCloseReasonLength,
		rgbCloseReasonBuffer);

quit:

	if (hRequestHandle != NULL)
	{
		WinHttpCloseHandle(hRequestHandle);
		hRequestHandle = NULL;
	}

	if (hWebSocketHandle != NULL)
	{
		WinHttpCloseHandle(hWebSocketHandle);
		hWebSocketHandle = NULL;
	}

	if (hConnectionHandle != NULL)
	{
		WinHttpCloseHandle(hConnectionHandle);
		hConnectionHandle = NULL;
	}

	if (hSessionHandle != NULL)
	{
		WinHttpCloseHandle(hSessionHandle);
		hSessionHandle = NULL;
	}

	if (dwError != ERROR_SUCCESS)
	{
		wprintf(L"Application failed with error: %u\n", dwError);
		return -1;
	}

	return 0;
}

NewServer(TLS) c.SetReadDeadline(time.Now().Add(conf.KeepaliveTime))

at end of the OnOpen handler, you call c.SetReadDeadline(time.Now().Add(conf.KeepaliveTime)), but on the OnData handler, you commented this line out. Why did you comment it out, and should OnOpen at least be consistent with OnData

large message send doesn't appear to work

#39

Issue running a websocket server on Mac osX 11.2.3

When I try running a server with the default configuration (copied the code from the many examples you provided), the following is printed to the console on my mac:

2021/08/04 23:52:24.632 [INF] Gopher[NB] start listen on: [":8001"]
2021/08/04 23:52:24.632 [ERR] Poller[NB_LISTENER_0] Accept failed: accept tcp [::]:8001: use of closed network connection, exit...

..and the server is effectively not active / can't be reached. I tried also using different ports, specifying "localhost" and so on.
Any idea on what might be going wrong here?

how to use this repo with gin?

our project use gin as http router,and upgrade websocket request in gin handler with websocket lib,
but this repo seem need to register http handle with repo own server.
is there has a way to use this repo like other websocket libs?
besides,i notice the api OnDataFrame,if server should work with half a pack when client send big data with frames?
can i assume data is complete when callling handler OnMessage?

SetSession/Session does not make sense when using TLS

As per doc both SetSession and Session are aimed to be used by end user to set own session context. And approach works fine up until TLS required.

WrapOpen sets tls context as session which breaks flow if user later wanna set own session object.

Would it make sense to actually let tls to implement Session methods and wrap Conn into tlsConn rather than doing it opposite side as end client should not know about any underlying transports rather than just having net.Conn interface.

UDP support

Any plans to add UDP? I have seen lines in gopher.go

	// tcp* supported only by now, there's no plan for other protocol such as udp,
	// because it's too easy to write udp server/client.

but well, e.g. I want to implement DNS proxy which should respond on UDP/TCP port 53,
it is ridiculous to implement another one flow instead using nbio itself.

Side question: as I understand onData() have to return ASAP to avoid blocking the event loop,
is call go handler() inside onData() enough or need to use some goroutine pool (which one actually)?

Good project，Thanks！！！ Use low memory and very fast！！！

TLS corruption when using macos foundation websocket client

sample code:
#25

I know you don't have a mac so not sure how you can reproduce. If can attach a tcpdump

websocket tls client panics

#101

并发请求下，客户端大量重连

看了 can not reconnect on the tls websocket example #7这个，不知道是否是mac设备的原因。

deadlock on connection close

I'm debugging this right now

 0  0x000000000403eee5 in runtime.gopark
    at /Users/adminac/sdk/go1.16.4/src/runtime/proc.go:337
 1  0x0000000004050b25 in runtime.goparkunlock
    at /Users/adminac/sdk/go1.16.4/src/runtime/proc.go:342
 2  0x0000000004050b25 in runtime.semacquire1
    at /Users/adminac/sdk/go1.16.4/src/runtime/sema.go:144
 3  0x0000000004072127 in sync.runtime_SemacquireMutex
    at /Users/adminac/sdk/go1.16.4/src/runtime/sema.go:71
 4  0x0000000004081265 in sync.(*Mutex).lockSlow
    at /Users/adminac/sdk/go1.16.4/src/sync/mutex.go:138
 5  0x000000000433d6a5 in sync.(*Mutex).Lock
    at /Users/adminac/sdk/go1.16.4/src/sync/mutex.go:81
 6  0x000000000433d6a5 in github.com/lesismal/llib/std/crypto/tls.(*Conn).Close
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/std/crypto/tls/conn.go:1577
 7  0x0000000004389ca4 in github.com/lesismal/nbio/nbhttp.NewServerTLS.func7
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/nbhttp/server.go:481
 8  0x000000000431702b in github.com/lesismal/nbio.(*poller).deleteConn
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/poller_kqueue.go:63
 9  0x00000000043141fc in github.com/lesismal/nbio.(*Conn).closeWithErrorWithoutLock
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/conn_unix.go:476
10  0x0000000004311bcc in github.com/lesismal/nbio.(*Conn).Write
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/conn_unix.go:100
11  0x0000000004339a3c in github.com/lesismal/llib/std/crypto/tls.(*Conn).write
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/std/crypto/tls/conn.go:1018
12  0x0000000004339dea in github.com/lesismal/llib/std/crypto/tls.(*Conn).writeRecordLocked
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/std/crypto/tls/conn.go:1090
13  0x000000000433938c in github.com/lesismal/llib/std/crypto/tls.(*Conn).sendAlertLocked
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/std/crypto/tls/conn.go:913
14  0x00000000043362e5 in github.com/lesismal/llib/std/crypto/tls.(*Conn).sendAlert
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/std/crypto/tls/conn.go:926
15  0x00000000043362e5 in github.com/lesismal/llib/std/crypto/tls.(*Conn).readRecordOrCCS
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/std/crypto/tls/conn.go:748
16  0x000000000433ce17 in github.com/lesismal/llib/std/crypto/tls.(*Conn).readRecord
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/std/crypto/tls/conn.go:641
17  0x000000000433ce17 in github.com/lesismal/llib/std/crypto/tls.(*Conn).AppendAndRead
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/std/crypto/tls/conn.go:1528
18  0x000000000438a174 in github.com/lesismal/nbio/nbhttp.NewServerTLS.func8
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/nbhttp/server.go:508
19  0x00000000043177f5 in github.com/lesismal/nbio.(*poller).readWrite
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/poller_kqueue.go:103
20  0x00000000043184c5 in github.com/lesismal/nbio.(*poller).readWriteLoop
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/poller_kqueue.go:199
21  0x0000000004317d7b in github.com/lesismal/nbio.(*poller).start
    at /Users/adminac/gosrc/pkg/mod/github.com/lesismal/[email protected]/poller_kqueue.go:142

it looks like on frame 17, in AppendAndRead, the frame owns the mutex (locks close mutex on conn.go:1489), then on frame 6, it is trying to get the mutex again

v1.2.6-beta.1 nbio tls websocket client (nbio_examples) does not appear to work

v1.2.5-beta.1 buffers appear to be corrupting at high connection count

使用中遇到的问题

协议解析怎么使用，怎么才能让OnData()的数据是一个协议的完整的数据包

Concurrency issues with NewServerTLS()

In some cases my server may want to disconnect an existing websocket connection. It is a case that does not show up in any of your examples. Imagine you would like to do the following here

if qps==1000 {
    wsConn.Close()
}

If I do something like this, the connection will get closed - the client will receive an onClose event. But wsConn.Close() then hangs. Any idea what could be causing this?

autobahn tests failures

9.* tests appear to timeout

Is it possible to set Headers on client-side before Websocket is established?

Hi, thank you for this great library.

I need to set an "Authorization" header on the client side when the client dials to the server (establishes the HTTP connection). I checked the examples and searched the docs but cannot seem to find a way to do it.

Is there any way to do it?

Just for reference to illustrate the point here is how the client side header can be set with Gorilla for example:

import "github.com/gorilla/websocket"

// ...

func connect() error {
	header := http.Header{}
	header["Authorization"] = []string{"myauthstring"}

	var resp *http.Response
	dialer := *websocket.DefaultDialer
	c.conn, resp, err = dialer.Dial("http://example.com/mywebsocketpath", header)
	if err != nil {
		if resp != nil {
			fmt.Printf("Server responded with status=%v\n", resp.Status)
		}
		return err
	}
}

I am looking for the equivalent capability in nbio.

Note: on the server side I am already able to check the value of the header this way and it works fine with nbio:

import "github.com/lesismal/nbio/nbhttp/websocket"

// ...

// somewhere in my init func:
mux := &http.ServeMux{}
mux.HandleFunc("/mywebsocketpath", srv.wsHandler)

// and here is the handler
func myWsHandler(w http.ResponseWriter, r *http.Request) {

	auth := r.Header["Authorization"]
	// check auth header here.

	upgrader := websocket.NewUpgrader()
	conn, err := upgrader.Upgrade(w, r, nil)
	if err != nil {
		log.Fatal("WS Server upgrade:", err)
	}
        // use websocket conn here.
}

Splice

Does this support splice?

memory usage explosion with provided examples

I used your client to stream in 1.5 MB messages at high throughput and I am seeing high memory usage. I'm going to look into optimizing for this scenario, but was wondering if you have any thought of how I could ago about do this.

I guess I could also go about creating a simple example, I'll do that first

Does Support net.Dial() or tls.Dial() for connect to nbio server?

can not reconnect on the tls websocket example

when I have the server running in another shell

$ go run .
2021/05/13 17:11:43 connecting to wss://localhost:8888/wss
2021/05/13 17:11:43 write: hello world
2021/05/13 17:11:43 read : hello world
2021/05/13 17:11:44 write: hello world
2021/05/13 17:11:44 read : hello world
2021/05/13 17:11:45 write: hello world
2021/05/13 17:11:45 read : hello world
2021/05/13 17:11:46 write: hello world
2021/05/13 17:11:46 read : hello world
^Csignal: interrupt
$ go run .
2021/05/13 17:11:49 connecting to wss://localhost:8888/wss
2021/05/13 17:11:49 dial:EOF
exit status 1
$ go run .
2021/05/13 17:11:51 connecting to wss://localhost:8888/wss
2021/05/13 17:11:51 dial:EOF
exit status 1

the server is still running the in the other shell

I am on a mac if that helps

after upgrader.Upgrade(w, r, nil), if client send message right now, and wsConn.OnMessage handler set delay， message will miss，handler should set when upgrader.Upgrade

func onWebsocket(w http.ResponseWriter, r *http.Request) {
upgrader := websocket.NewUpgrader()
upgrader.EnableCompression = true
conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
panic(err)
}
wsConn := conn.(*websocket.Conn)
wsConn.EnableWriteCompression(true)
wsConn.SetReadDeadline(time.Time{})
wsConn.OnMessage(func(c *websocket.Conn, messageType websocket.MessageType, data []byte) {
// echo
if *print {
switch messageType {
case websocket.TextMessage:
fmt.Println("OnMessage:", messageType, string(data), len(data))
case websocket.BinaryMessage:
fmt.Println("OnMessage:", messageType, data, len(data))
}
}
c.WriteMessage(messageType, data)
})
wsConn.OnClose(func(c *websocket.Conn, err error) {
if *print {
fmt.Println("OnClose:", c.RemoteAddr().String(), err)
}
})
if *print {
fmt.Println("OnOpen:", wsConn.RemoteAddr().String())
}
}

autobahn tests failures

seeing a lot of crashes on the autobahn test suite

[nbio-autobahn]: Running test case ID 9.4.4 for agent non-tls from peer tcp4:127.0.0.1:9998
[nbio-server]:   2021/08/10 12:37:47.210 [ERR] taskpool call failed: runtime error: invalid memory address or nil pointer dereference
[nbio-server]:   goroutine 7 [running]:
[nbio-server]:   github.com/lesismal/nbio/taskpool.call.func1()
[nbio-server]:          /go/src/github.com/lesismal/nbio/taskpool/caller.go:19 +0xc7
[nbio-server]:   panic(0x6bfaa0, 0x8ec800)
[nbio-server]:          /usr/local/go/src/runtime/panic.go:965 +0x1b9
[nbio-server]:   github.com/lesismal/nbio/nbhttp/websocket.(*Conn).writeFrame(0x0, 0x102, 0xc004243000, 0x1000, 0x1000, 0x0, 0x766d60, 0xc00004ace0)
[nbio-server]:          /go/src/github.com/lesismal/nbio/nbhttp/websocket/conn.go:318 +0x192
[nbio-server]:   github.com/lesismal/nbio/nbhttp/websocket.(*Conn).WriteFrame(...)
[nbio-server]:          /go/src/github.com/lesismal/nbio/nbhttp/websocket/conn.go:274
[nbio-server]:   github.com/lesismal/nbio/examples/websocket/server_autobahn.newUpgrader.func1(0x0, 0x460002, 0xc004243000, 0x1000, 0x1000)
[nbio-server]:          /go/src/github.com/lesismal/nbio/examples/websocket/server_autobahn/server.go:23 +0x88
[nbio-server]:   github.com/lesismal/nbio/nbhttp/websocket.newConn.func6(0x0, 0x50002, 0xc004243000, 0x1000, 0x1000)
[nbio-server]:          /go/src/github.com/lesismal/nbio/nbhttp/websocket/conn.go:409 +0x6d
[nbio-server]:   github.com/lesismal/nbio/nbhttp/websocket.(*Upgrader).handleDataFrame.func1()
[nbio-server]:          /go/src/github.com/lesismal/nbio/nbhttp/websocket/upgrader.go:408 +0x6d
[nbio-server]:   github.com/lesismal/nbio/nbhttp.(*Parser).Execute.func1()
[nbio-server]:          /go/src/github.com/lesismal/nbio/nbhttp/parser.go:120 +0x62
[nbio-server]:   github.com/lesismal/nbio/taskpool.call(0xc003082048)
[nbio-server]:          /go/src/github.com/lesismal/nbio/taskpool/caller.go:23 +0x5d
[nbio-server]:   github.com/lesismal/nbio/taskpool.(*fixedRunner).taskLoop(0xc000056300)
[nbio-server]:          /go/src/github.com/lesismal/nbio/taskpool/fixedpool.go:41 +0x16d
[nbio-server]:   created by github.com/lesismal/nbio/taskpool.NewFixedPool
[nbio-server]:          /go/src/github.com/lesismal/nbio/taskpool/fixedpool.go:125 +0x167
[nbio-server]:   
[nbio-server]:   

Gopher的listeners的初始化问题

以下代码是不是重复了（Is the following code repeated）？：

1. gopher_std.go/L95，listeners: make([]*poller, len(conf.Addrs))
2. gopher_std.go/L24：g.listeners = make([]*poller, len(g.addrs))

compress after enable, not worked

thanks for your work~

c.SetCompressionLevel(9)
c.EnableWriteCompression(true)
c.WriteMessage(messageType, data)

frequent disconnects when more than one connection

I'm using the websocket over tls on linux and it appears to work ok when the server only has one client connected, but with more than one I am seeing, read timeout errors on the server

OnClose: 172.20.0.1:62684 read timeout

all clients are performing the same transaction. Every half a second they send a message to the server than the server responds with acknowledgment

Keeping many idle clients alive

In another thread you said:

which would cost lots of goroutines when there are lots of connections, problems about mem/gc/schedule/stw comes with the huge number of goroutines.

My current keep-alive implementation is using 1 goroutine per client to send PingMessages in perfect frequency. It is using case <-time.After(): which is simple (almost elegant) and is working really well. But it creates a lot of goroutines.

The alternative "1 goroutine for all clients" implementation, that I can think of, would be way more complex. And I am not certain it would be really so much more efficient. Any thoughts you can share on this?

support configurable buffer pool for all the different layers

1. nbio.Conn Write
2. tls.Conn(already)
3. http Read/Write
4. websocket Read/Write

Request: websocket over tls example.

I need a run websocket framing on top of tls. I may get around to trying to implement in the next few days if I get some free time, but if you know how to implement quickly, that would be great.

Cheers.

使用nbio的项目

如果可以，请留言您正在使用nbio的项目相关信息．

Why does nbio need to lock while reading and writing

nbio/conn_unix.go

Line 59 in db13c52

c.mux.Lock()

I think for one connetion, it always works in only one loop (one goroutine) after being put into epoll

support tls and non-tls servers and clients with the same engine/mux

not sure if all of these together are supported at the same time or if it's better to create multiple servers/engines.

currently my server is using nbio for the tls connection and the stock go http server for the non-tls connection and gorrilla for the outbound websocket connections.

I'm wondering if it is a good idea/possible to use a single common nbio engine for all these scenarios and share the mux for for the tls and non-tls servers.