frevib / io_uring-echo-server

io_uring echo server

io_uring bare minimum echo server

• uses an event loop created with io_uring
• uses liburing https://github.com/axboe/liburing
• Linux 5.7 or higher with IORING_FEAT_FAST_POLL and IORING_OP_PROVIDE_BUFFERS required

Install and run

First install the latest liburing:

git clone https://github.com/axboe/liburing

./configure

make

make install

Then install io_uring-echo-server:

git clone https://github.com/frevib/io_uring-echo-server.git

make io_uring_echo_server

./io_uring_echo_server [port_number]

compare with epoll echo server

https://github.com/frevib/epoll-echo-server

Benchmarks (without buffer selection)

https://github.com/frevib/io_uring-echo-server/blob/io-uring-feat-fast-poll/benchmarks/benchmarks.md

Versions

v1.7

• Use latest liburing from https://github.com/axboe/liburing

v1.6

• Use IORING_OP_PROVIDE_BUFFERS

v1.5

• Use IORING_FEAT_FAST_POLL, which increases performance a lot
• Remove all polling, this is now handled by IORING_FEAT_FAST_POLL

v1.4

Fix bug that massively overstated the performance.

v1.3

Use pre-allocated sqe->user_data instead of dynamically allocating memory.

v1.1

Fix memory leak.

v1.0

Working release.

io_uring-echo-server's Issues

IORING_FEAT_FAST_POLL question

Hi,

I do not see anything in your code that does something differently. what IORING_FEAT_FAST_POLL does and where exactly you use it? (besides testing it in the features)

please get rid of liburing private copy

$ ./io_uring_echo_server 1976
io_uring echo server listening for connections on port: 1976
from io_uring_echo-server main() sizeof(struct io_uring): 160
From io_uring_queue_mmap() sizeof(*ring): 216 sizeof(struct io_uring): 216
Before memset() in io_uring_queue_mmap(): p->sq_entries: 2048
After memset() in io_uring_queue_mmap(): p->sq_entries: 0
io_uring_init_failed...
: Invalid argument

Wild results, cannot reproduce

I have tested your epoll and io_uring examples and I get 250k (req/sec) with your epoll example and only 220k with io_uring. I also get 250k with my own epoll implementation so that confirms we are both using efficient use of epoll.

I'm running on Linux 5.7 Clear Linux - do you have any hints on how I can reproduce your results?

Buffers are not being reused: "bufs in automatic buffer selection empty, this should not happen..."

Hi,

Buffers are not being reused after calling io_uring_prep_provide_buffers(). To reproduce this error change:

#define MAX_CONNECTIONS 4096

to

#define MAX_CONNECTIONS 2

Then telnet to the echo server port, close the connection and repeat three times. The first time all works well as the first two buffers are used but on the third connection attempt io_uring runs out of buffers and returns "No buffer space available". I did some debugging and io_uring starts using buf id 1, then id 0 but on the third attempt rather than reusing any of the two available buffers it fails.

It might be a io_uring bug as the code looks fine. I'm using the 5.11 kernel.

Thanks.

从源码上看两个比较模型不合理

epoll-echo-server 单线程，高并发时，recv、send操作都是串行执行
uring-echo-server 单线程，高并发时，uring_read、uring_write的sqe在内核中是iou-wrk线程池并发执行

How to achieve timeout send/receive?

...

A suggestion

Hi, I'm new to io_uring and I'm recently practicing using io_uring and using your code as a reference. I tried to modify the conn_info structure but found that the program was giving me errors, and it took me a long time to figure out that user_data only holds 64-bit data. I think your code may be misleading to people new to io_uring, although your conn_info structure happens to be 64-bit, but if you can, I suggest you store a pointer to conn_info in user_data, because the above problem really bothered me for two days :(

Why not need to malloc for cqe->user_data or sqe->user_data ( conn_info )?

https://github.com/frevib/io_uring-echo-server/blob/io-uring-op-provide-buffers/io_uring_echo_server.c#L172

Could add a udp_echo server demo ?

io_uring_echo_server becomes unresponsive in an uninterruptible sleep state

I faced the same issue during long running of the io_uring based server. Have u ever addressed this issue or have some insights about it?

redundant text sent

$ telnet 127.0.0.1 12345                                                                                       13:58:11
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
1234567890
1234567890
12345
12345
890

expected:

$ telnet 127.0.0.1 12345                                                                                       13:59:12
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
1234567890
1234567890
12345
12345

Return same amount of bytes - Rust echo bench gives unreliable results

Using a fixed buffer gives unreliable results in with Rust echo bench

crashed when the number of clients exceeded 1100

I tried many times，it crashed every time when the number of clients exceeded 1100.

cargo run --release -- --address 10.18.4.6:12346 --number 1500 --duration 60 --length 500

Use `io_uring_free_probe()` instead of `free()` in liburing 2.4 to avoid runtime errors.

Hello frevib!

I've just started to learn io_uring. I am getting a runtime error while running your io_uring code for echo server. It seems we can no longer use free() to free probes. Here are my environments:

% pacman -Si liburing
Repository      : extra
Name            : liburing
Version         : 2.4-1
Description     : Linux-native io_uring I/O access library
Architecture    : x86_64
URL             : https://git.kernel.dk/cgit/liburing/
Licenses        : LGPL2.1  MIT
Groups          : None
Provides        : liburing.so=2-64  liburing-ffi.so=2-64
Depends On      : glibc
Optional Deps   : None
Conflicts With  : None
Replaces        : None
Download Size   : 160.30 KiB
Installed Size  : 309.36 KiB
Packager        : David Runge <[email protected]>
Build Date      : Thu 08 Jun 2023 07:02:14 PM +08
Validated By    : MD5 Sum  SHA-256 Sum  Signature

% uname -a
Linux relm 6.4.7-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 27 Jul 2023 22:02:18 +0000 x86_64 GNU/Linux

% g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/13.1.1/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc/src/gcc/configure --enable-languages=ada,c,c++,d,fortran,go,lto,objc,obj-c++ --enable-bootstrap --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --with-build-config=bootstrap-lto --with-linker-hash-style=gnu --with-system-zlib --enable-__cxa_atexit --enable-cet=auto --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-default-ssp --enable-gnu-indirect-function --enable-gnu-unique-object --enable-libstdcxx-backtrace --enable-link-serialization=1 --enable-linker-build-id --enable-lto --enable-multilib --enable-plugin --enable-shared --enable-threads=posix --disable-libssp --disable-libstdcxx-pch --disable-werror
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 13.1.1 20230714 (GCC) 

Cheers,
Fw[a]rd.