spacejam / rio Goto Github PK

I am new to this library and I am trying to figure out how does this library help batching syscalls for network I/O. my use case is I am trying to use this library to broadcast a stream of messages m1,m2, m3....to all the connected sockets of my server. so if there are 1000 clients connected to my server I want to broadcast each message to all 1000 and I am trying to accomplish this is in as few syscalls as possible because called syscall.send on every socket descriptor seems to be the bottleneck of my server to scale. Latency is very important for my app.

Below are my questions/comments about rio library

1. does calling ring.write_at makes a syscall? I am assuming the answer is no since it returns a Future however I assume calling ring.write_at.await? will make a syscall. so if I need to batch I need to call ring.write_at for each individual message I want to broadcast and add then call ring.submit_all? but then what I do with each each individual future I get from calling ring.write_at?
2. How do I handle backpressure in my app if the submission queue is full? what I do? just keep retrying forever?

with the following code:

fn main() -> io::Result<()> {
  let ring = rio::new()?;
  let acceptor = TcpListener::bind("127.0.0.1:8080")?;
  let c1 = ring.accept(&acceptor);
  ring.submit_all();
  println!("c1: {:?}", c1.wait());
  Ok(())
}

I get the following panic trace:

thread 'main' panicked at 'failed to submit our expected SQE on ensure_submitted. expected old 0 + submitted 0 to be >= sqe_id 1', /home/geal/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.2/src/io_uring/uring.rs:102:13          
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace 
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "PoisonError { inner: .. }"', /home/geal/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.2/src/io_uring/uring.rs:90:13
stack backtrace:                                                                                                       
   0:     0x55c1f8995dc8 - backtrace::backtrace::libunwind::trace::h86edaa2680be3f32                       
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1:     0x55c1f8995dc8 - backtrace::backtrace::trace_unsynchronized::h020717321cc60d9f                                                                                                                                                      
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2:     0x55c1f8995dc8 - std::sys_common::backtrace::_print_fmt::h95a740d649d8282b                                                                                                                                                          
                               at src/libstd/sys_common/backtrace.rs:77                                      
   3:     0x55c1f8995dc8 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h229d12a248a94d4d
                               at src/libstd/sys_common/backtrace.rs:59
   4:     0x55c1f89b4e4c - core::fmt::write::h7a7c155a9a2fc994
                               at src/libcore/fmt/mod.rs:1052
   5:     0x55c1f8990867 - std::io::Write::write_fmt::hbc3e21ba137de707
                               at src/libstd/io/mod.rs:1428 
   6:     0x55c1f8998485 - std::sys_common::backtrace::_print::h8ecf04ab6aa60d02
                               at src/libstd/sys_common/backtrace.rs:62
   7:     0x55c1f8998485 - std::sys_common::backtrace::print::hbbeb2ccd67fe006e
                               at src/libstd/sys_common/backtrace.rs:49
   8:     0x55c1f8998485 - std::panicking::default_hook::{{closure}}::h30799abc567130ac
                               at src/libstd/panicking.rs:204
   9:     0x55c1f89981c6 - std::panicking::default_hook::h992fc24d479949ec
                               at src/libstd/panicking.rs:224
  10:     0x55c1f8998ae2 - std::panicking::rust_panic_with_hook::hd5c9bb7319c9d846
                               at src/libstd/panicking.rs:470
  11:     0x55c1f89986cb - rust_begin_unwind
                               at src/libstd/panicking.rs:378
  12:     0x55c1f89b36e1 - core::panicking::panic_fmt::hb5178b003b60d015
                               at src/libcore/panicking.rs:85                                                                                                                                                                                 
  13:     0x55c1f89b3503 - core::option::expect_none_failed::ha3a5581dc27ee7da
                               at src/libcore/option.rs:1199
  14:     0x55c1f895ec20 - core::result::Result<T,E>::unwrap::h128df1335a9895c2
                               at /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/result.rs:963
  15:     0x55c1f897f2ff - rio::io_uring::uring::Uring::ensure_submitted::h26a3d49f0b4273bd
                               at /home/geal/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.2/src/io_uring/uring.rs:90
  16:     0x55c1f896584f - rio::completion::Completion<C>::wait_inner::hb9b19c50f41fdda3
                               at /home/geal/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.2/src/completion.rs:88
  17:     0x55c1f8966189 - <rio::completion::Completion<C> as core::ops::drop::Drop>::drop::h52530162c3514f2d
                               at /home/geal/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.2/src/completion.rs:108
  18:     0x55c1f8962215 - core::ptr::drop_in_place::hb39ddc4a50cec2ea
                               at /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ptr/mod.rs:174
  19:     0x55c1f895aeb8 - rio::completion::Completion<C>::wait::h27ed660e57b425e5
                               at /home/geal/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.2/src/completion.rs:76
  20:     0x55c1f895b321 - rio_test::main::he690bc1d1ba0d878
                               at src/main.rs:45
  21:     0x55c1f895afb5 - std::rt::lang_start::{{closure}}::hc8053d381934b416
                               at /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/rt.rs:67
  22:     0x55c1f89985a3 - std::rt::lang_start_internal::{{closure}}::h35e485e19663c96e
                               at src/libstd/rt.rs:52
  23:     0x55c1f89985a3 - std::panicking::try::do_call::h00ecf8f872707631
                               at src/libstd/panicking.rs:303
  24:     0x55c1f899bb67 - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:86
  25:     0x55c1f8998f89 - std::panicking::try::h3f84bc59b258dd0b
                               at src/libstd/panicking.rs:281
  26:     0x55c1f8998f89 - std::panic::catch_unwind::hd5737c3ccacd68f7
                               at src/libstd/panic.rs:394
  27:     0x55c1f8998f89 - std::rt::lang_start_internal::h0b1b741f79488fca                           
                               at src/libstd/rt.rs:51                                                                  
  28:     0x55c1f895af89 - std::rt::lang_start::h9ae5b092ac55de00
                               at /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/rt.rs:67
  29:     0x55c1f895b5aa - main                                                                                        
  30:     0x7fcf8b4d1153 - __libc_start_main       
  31:     0x55c1f895916e - _start                                                                                      
  32:                0x0 - <unknown>                                                                                   
thread panicked while panicking. aborting.                                                                             
sh : ligne 1 : 154044 Instruction non permise (core dumped)cargo run

and for good measure, it ends on a sigill :D

If I remove the submit_all line, it does not panic

I'm working on a similar library for go and was wondering if there is any more details on how back pressure works, I noticed it doesn't use IORING_FEAT_NODROP. I've looked at the code a little bit, but I'm not as familiar with rust.

Hi,
I'm trying to run the minimal example that you give for TCP handling but it get stuck on the first accept(...).wait
I'm running linux kernel 5.7.1 on ubuntu 20.4
If I try to netcat to 127.0.0.1:6666 give a few input and force exit the server, netcat will stop itself.
I've written a simple client with std::net::TcpStream::connect() and the connect works on the client side but the accept never wakes up on the server one.

use std::{
    io,
    net::{TcpListener, TcpStream},
};
use rio::Config;

async fn proxy(ring: &rio::Rio, a: &TcpStream, b: &TcpStream) -> io::Result<()> {
    let buf = vec![0_u8; 512];
    loop {
        let read_bytes = ring.read_at(a, &buf, 0).await?;
        let buf = &buf[..read_bytes];
        ring.write_at(b, &buf, 0).await?;
    }
}

fn main() -> io::Result<()> {
    let ring = rio::new()?;
    let acceptor = TcpListener::bind("127.0.0.1:6666")?;

    extreme::run(async {
        // kernel 5.5 and later support TCP accept
        loop {
            let stream = ring.accept(&acceptor).await?; //haging is here
            dbg!(proxy(&ring, &stream, &stream).await);
        }
    })
}

Hi,
this has been brought up in #3 to a degree but I'd like to suggest adding a CLA that assigns the copyright any of contributions to @spacejam - this would resolve any license ambiguities for contributions while still maintaining the ability to share the code under GPL and MIT for sponsors.

Hi, I get "Cannot allocate memory" error when creating the third instance, even if the first two are dropped. Code

fn main() {
    for _ in 0..4 {
        println!("{:?}", rio::new().map(|_| ()));
    }
}

outputs

Ok(())
Ok(())
Err(Os { code: 12, kind: Other, message: "Cannot allocate memory" })
Err(Os { code: 12, kind: Other, message: "Cannot allocate memory" })

I am using rio 0.7.5 on Linux 5.4.13 and I'm sure there's enough memory.

hello,

I was wondering if you could offer advice about how to store pending Completion instances while you are waiting for them to be completed, potentially doing other work in the meantime and submitting additional tasks to the queue.

the two issues I ran into while attempting a demo event loop design around Rio were:

1. no way to storeCompletion instances returned by my ring, as taking an immutable reference into a collection would prevent the ability to add additional items to it, and
2. no way to check if a Completion is done without blocking (not sure if this is avoidable, I am not a io-uring expert (yet))

I posted some code here, which is where I stopped, unable to figure out how I would go further: https://gist.github.com/jonathanstrong/0d8aaef2fbbb62d2ddcd291f62a03114

my goal is for a thread pool worker to be receiving io jobs over a channel, submitting work to the ring, potentially for several unrelated jobs at once, then processing the results from the ring as they become available, and sending back info/data about the outcome of the work to the thread that sent the job upon that result becoming available.

another, more big picture way of saying this is, how do I store the context (buffer/file) of a Rio submission, other than a named stack variable as in the examples?

thanks for any help you can provide!

Hello, I'm getting this vulnerability when building my application

Does anyone know how to solve it?

This code assumes that the layout of std::net::SocketAddrV{4,6} matches libc::sockaddr, but std makes no such promise. See rust-lang/rust#78802 for more details.

Example fixes: tokio-rs/mio#1388, rust-lang/socket2#120.

Well, not the trait itself, but the bound isn't enough to stop various read methods from violating aliasing rules. Example:

fn main() {
    let data: Vec<u8> = vec![0; 1024];
    crossbeam_utils::thread::scope(|s| {
        let ring = rio::new().expect("create uring");
        let file = std::fs::File::open("file").expect("openat");
        let _c = ring.read_at(&file, &data, 0);
        s.spawn(|_| immut(&data));
    }).unwrap();
}

fn immut(data: &[u8]) {
    loop {
        println!("{:?}", data[0]);
        if data[0] != 0 { break; }
        std::thread::sleep(std::time::Duration::from_millis(1));
    }
}

If you mkfifo fifo and run this, you'll see that once data is written into the fifo, immut will observe a change in data even though it's purported to be behind a shared reference.

hello! I am implementing the connect() operation and need some advice on the API. The call looks like this:

pub fn connect<'a>(                            
        &'a self,                                          
        socket: &TcpStream,                      
        address: *const libc::sockaddr,                  
    ) -> Completion<'a, ()> {                            
        self.with_sqe(None, false, |sqe| {         
            sqe.prep_rw(                                   
                IORING_OP_CONNECT,                         
                socket.as_raw_fd(),                  
                0,                                         
                std::mem::size_of::<libc::sockaddr>() as u64,
                Ordering::None,                   
            );                                
                                                           
            sqe.addr = address as u64;         
        })                                         
    } 

And it is called like this:

 let mut stream = net2::TcpBuilder::new_v4()?.to_tcp_stream()?;
  let c: SocketAddr = "127.0.0.1:8080".parse().unwrap();
  let (address, len) = addr2raw(&c);
  println!("connecting to address {:?} ({:?}) -> {:?}, len = {:?}", c, &c as *const SocketAddr, address, len);
  let c1 = ring.connect(&stream, address);

(I took the addr2raw function from https://github.com/rust-lang-nursery/net2-rs/blob/eda403f03033b58560d81b821f0df822f50cde4d/src/socket.rs#L87-L96 ).

I think the function should take a SocketAddr as argument instead of a libc::sockaddr (so connect() would do the conversion), I'd like to add an Ordering argument (to try and link a read or write just after connecting), and I'm ambivalent about requiring net2, but std::net:TcpStream does not allow creation of a socket outside of TcpStream::connect() or TcpListener::accept().

What do you think?

Running cargo test leads to:

thread 'test_vec_value' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 38, kind: Other, message: "Function not implemented" }', src/libcore/result.rs:1188:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:84
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1025
   5: std::io::Write::write_fmt
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/io/mod.rs:1426
   6: std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt
             at src/libstd/io/impls.rs:156
   7: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   8: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   9: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:193
  10: std::panicking::default_hook
             at src/libstd/panicking.rs:207
  11: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:471
  12: rust_begin_unwind
             at src/libstd/panicking.rs:375
  13: core::panicking::panic_fmt
             at src/libcore/panicking.rs:84
  14: core::result::unwrap_failed
             at src/libcore/result.rs:1188
  15: core::result::Result<T,E>::unwrap
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/result.rs:956
  16: vec::test_vec_value
             at tests/vec.rs:12
  17: vec::test_vec_value::{{closure}}
             at tests/vec.rs:4
  18: core::ops::function::FnOnce::call_once
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/ops/function.rs:232
  19: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/liballoc/boxed.rs:1022
  20: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
  21: std::panicking::try
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/panicking.rs:270
  22: std::panic::catch_unwind
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/panic.rs:394
  23: test::run_test_in_process
             at src/libtest/lib.rs:567
  24: test::run_test::run_test_inner::{{closure}}
             at src/libtest/lib.rs:474
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

running on:

➜  rio uname -a
Linux benchy 4.18.0-147.3.1.el8_1.x86_64 #1 SMP Fri Jan 3 23:55:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
➜  rio rustc --version
rustc 1.41.0 (5e1a79984 2020-01-27)

Hi,

I wanted to ask if it would be possible to support sendmsg and recvmsg calls through io_uring via the Rio interface? I would like to use io_uring for UDP, which is quite bleeding edge: Ubuntu 20.04 comes with Kernel 5.4, which supports the aforementioned methods, but not the plain send / recv calls that rio currently uses (see here). Hence, implementing the sendmsg/recvmsg would significantly ease adoption for me.

I would also volunteer contributing this feature if you agree.

Thanks!

Martin

I attempted to start playing around with rio and grabbed the example: https://docs.rs/rio/0.9.4/rio/#examples
and modified it to call write_at_ordered instead of write_at and specify and order of Ordering::Link.

This promptly crashes with the error

Error: Os { code: 27, kind: Other, message: "File too large" }

I thought maybe the fact that the file was 4gb might be related and changed the code to write a smaller file, but this also failed the same error.

I know little about Rust, but it seems that

blocks future connections before the current client is disconnected, which generally forbids concurrent requests serving. Only one connection is allowed at the same time.

Related project: https://github.com/haraldh/rust_echo_bench

Any thoughts?

MIT/Apache-2.0 license is available for spacejam's github sponsors.
Cargo.toml

Rio seems to have a tricky licence: GPLv3 with exceptions. But I think this fact is not documented properly.

Primary places for licence to be visible are crates.io page, Github page and docs.rs page.

• On the crates.io page you see "GPL-3.0". Longer text about exceptions only briefly blinks though until README is loaded.
• On Github, you don't see any licence info in README, but see LIZENZ file, which is for GPLv3 without mentioning the exception.
• On the docs.rs page there's nothing about licence at all.

I suggest to copy that phrase from the description inside Cargo.toml to README and to root module doccomment. Maybe a longer document about how the exception is granted, when it is terminated and other lawyery details can be provided in the repository.

Related: #3 #16

https://gist.github.com/fwsGonzo/9f4d6eb74f42fc6a95953173ba210b4e

I get these on my machine.

If the assert fails, the object value_ptr pointing to (which is on the heap) will leak.

Hi, this project looks very interesting. I've started trying it out, but I certainly do not understand io_uring completely yet. I tried to create some benchmarks in a project where I think io_uring would be useful. However, once I put it in the Bencher::iter loop I always get too many files open. Should I rather open one ring for the entire application? See test case which usually reproduces the error with ulimit -n 1024:

tests/many.rs:

use std::io::prelude::*;
use std::io::SeekFrom;

#[test]
fn test_loop() {
    let mut file = std::fs::OpenOptions::new()
        .create(true)
        .read(true)
        .write(true)
        .open("/tmp/rio-test-loop-data")
        .unwrap();

    file.write(b"hello hello again").unwrap();
    file.seek(SeekFrom::Start(0)).unwrap(); // unnecessary? 

    for _ in 0..1024 {
        let ring = rio::new().unwrap();
        let mut buf = vec![0_u8; 17];
        ring.read_at(&file, &mut buf, 0).wait().unwrap();
        assert_eq!(buf, b"hello hello again");
    }

    std::fs::remove_file("/tmp/rio-test-loop-data").unwrap(); // does not (usually) get called
}

Cool library, but I would be wary of the way the license is structured. From the description:

GPL-3.0 nice bindings for io_uring. MIT/Apache-2.0 license is available for spacejam's github sponsors.

You cannot re-license other peoples' work, so this precludes accepting any pull requests made under the GPL.

Will rio eventually support all the things including Unix sockets (with their SCM_RIGHTS and other tricks), madvice and so on (including not yet appeared things)? Or it would focus on just file and TCP API?

Shall each IORING_OP_ inaccessible with rio be considered a missing feature to be eventually implemented?

This is not necessary as the type contains no field that isn't Send and is therefore Send already. See this playground. If the point is to ensure that property for the future, rather add an explicit test asserting the property.

I am currently transforming KuiBaDB into a fully asynchronous database, and want to use RIO as the underlying implementation of asynchronous IO.

But it seems that IORING_SETUP_SQPOLL can not work in RIO. RIO should also update Sq::ktail when IORING_SETUP_SQPOLL is enabled, but it doesn't. This causes the kernel to always think that there is no new SQE, which causes the Completion::wait() block forever.

fn main() {
    let mut cfg = rio::Config::default();
    cfg.sq_poll = true;
    let ring = cfg.start().expect("create uring");
    let file = std::fs::File::open("file").expect("openat");
    let data: &mut [u8] = &mut [0; 66];
    let completion = ring.read_at(&file, &data, 0);

    // if using threads
    completion.wait().unwrap();
}

[zhanyi@X rustplay]# /home/zhanyi/project/org/hidva.com/rustplay/target/debug/rustplay
^C  # block forever...

Hey Tyler, been eagerly following your progress with rio and sled on Twitter. Been tracking io_uring for a while, decided to finally spend some time programming in it today.

I've been working on moving my I/O benchmarks in napkin-math to use io_uring on Linux. I started with the sequential test-case, as this seemed easiest to start with (I skimmed through Axboe's io-uring-cp(1) too, since it's

I had some trouble allocating the buffers for all these reads, FWIW--maybe something rio could help with. After poking around the API docs, I found a way to get a mutable slice without upsetting the borrow-checker with unsafe (🙈)

To my surprise, io_uring kept performing worse than simple read(2), even though I'm re-using the buffers. I tried tuning the size of the buffers and the depth of the queue, but to no avail (the settings in the script below are the ones Axboe uses in his cp example, so it seems sensible). Do note that I'm fairly inexperienced with Rust, so I may be missing things 👀

I've tried to make this very easy to reproduce. If you throw this gist into examples/sequential_io.rb and run it with cargo run --release --example sequential_io, you should see something along the lines of:

curl https://gist.githubusercontent.com/sirupsen/dca5e3483fec7cd06f2f98e1934f5b85/raw/1150c7a3d7a98127ee1111a810a63fb2459a7913/sequantial_io.rs > examples/sequential_io.rs
cargo run --release --example sequential_io
# 512mb, sequential, summing
read(2) done in 250ms, checksum: 536870912
io_uring(2) done in 295ms, checksum: 536870912

Edit: Some concerns with the summing overhead, see my comment below, but it shows the same pattern when eliminated.

I'm on 5.3.0-29-generic, I know I could be more recent, but that's the most recent Ubuntu kernel I could dust up that doesn't require re-compiling perf etc

Edit: I upgraded to 5.5.1, and results are the same.

Attempting to use sq_poll fails with the following error:

thread '<unnamed>' panicked at 'error in cqe reaper: Os { code: 6, kind: Other, message: "No such device or address" }'

I believe this is because io_uring_register is never called on the fd. I see that register is mapped out in https://github.com/spacejam/rio/blob/master/src/io_uring/syscall.rs#L81 but never appears to be called anywhere.

Am I missing something, or does this still need to be implemented? Thanks!

Added dependency on rio on MacOS 10.14.6.

[dependencies]
rio = "0.9.4"

With cargo 1.47.0, cargo check failed with the error message below:

   Compiling libc v0.2.80
    Checking rio v0.9.4
error[E0432]: unresolved imports `super::io_uring`, `super::Uring`
  --> /Users/juqing/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.4/src/completion.rs:11:5
   |
11 |     io_uring::io_uring_cqe, FromCqe, Measure, Uring, M,
   |     ^^^^^^^^                                  ^^^^^ no `Uring` in the root
   |     |
   |     maybe a missing crate `io_uring`?

error[E0433]: failed to resolve: use of undeclared type or module `Config`
   --> /Users/juqing/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.4/src/lib.rs:219:5
    |
219 |     Config::default().start()
    |     ^^^^^^ use of undeclared type or module `Config`

error[E0433]: failed to resolve: use of undeclared type or module `io_uring`
   --> /Users/juqing/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.4/src/lib.rs:298:22
    |
298 |     fn from_cqe(cqe: io_uring::io_uring_cqe) -> Self;
    |                      ^^^^^^^^ use of undeclared type or module `io_uring`

error[E0433]: failed to resolve: use of undeclared type or module `io_uring`
   --> /Users/juqing/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.4/src/lib.rs:302:22
    |
302 |     fn from_cqe(cqe: io_uring::io_uring_cqe) -> usize {
    |                      ^^^^^^^^ use of undeclared type or module `io_uring`

error[E0433]: failed to resolve: use of undeclared type or module `io_uring`
   --> /Users/juqing/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.4/src/lib.rs:309:20
    |
309 |     fn from_cqe(_: io_uring::io_uring_cqe) {}
    |                    ^^^^^^^^ use of undeclared type or module `io_uring`

error[E0412]: cannot find type `Rio` in this scope
   --> /Users/juqing/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.4/src/lib.rs:218:28
    |
218 | pub fn new() -> io::Result<Rio> {
    |           -                ^^^ not found in this scope
    |           |
    |           help: you might be missing a type parameter: `<Rio>`

error[E0282]: type annotations needed
  --> /Users/juqing/.cargo/registry/src/github.com-1ecc6299db9ec823/rio-0.9.4/src/completion.rs:25:19
   |
25 |             item: None,
   |                   ^^^^ cannot infer type for type parameter `T` declared on the enum `Option`

error: aborting due to 7 previous errors

I came across the following note in https://docs.rs/rio/0.9.3/rio/struct.Completion.html, which I found surprising and possibly disingenuous:

Safety

Never call std::mem::forget on this value. It can lead to a use-after-free bug. The fact that std::mem::forget is not marked unsafe is a bug in the Rust standard library.

In trying to understand the backstory of your position on this, I found this sequence of tweets from which I think I see the misunderstanding on your part:

• @Lucretiel(https://twitter.com/Lucretiel/status/1256430755808509952)—
  Wait, does this mean that forgetting something can lead to unsoundness in rio? That would imply I can create unsoundness by, say, accidentally creating a reference cycle

• @spacejam(https://twitter.com/sadisticsystems/status/1256519569830670336)—
  A reference cycle is fine and not the same as mem::forget because the compiler still enforces that any buffer passed to rio outlives the kernel's write into the buffer

• @spacejam(https://twitter.com/sadisticsystems/status/1256519873401745413)—
  A reference cycle is immortal. mem::forget allows desired lifetime constraints related to Drop to be violated

• @spacejam(https://twitter.com/sadisticsystems/status/1256520244505411586)—
  Immortality is safe. Violating drop-based lifetime constraints via forget can lead to use after free though

In case it helps see why these tweets are wrong, I wrote up some example code that demonstrates stack corruption (or heap corruption / use after free / however you want it) without any involvement of forget.

The same misunderstanding in rio was called out recently by someone else in https://www.reddit.com/r/rust/comments/hk8lab/ringbahn_ii_the_central_state_machine/fwt0xmt/.

Hopefully the example code will help identify a path to making a sound API for rio; otherwise it would be good to clarify the documentation to make it clear that the safety proposition is "hope that nothing else in your code interacts poorly with rio's assumptions about Rust semantics", rather than anything specific about forget.

// [dependencies]
// extreme = "666.666.666666"
// futures-util = "0.3"
// rio = { git = "https://github.com/spacejam/rio" }

#[derive(Debug)]
pub enum Buffer {
    Inline([u8; 32]),
    OutOfLine(Vec<u8>),
}

impl Buffer {
    fn as_mut(&mut self) -> &mut [u8] {
        match self {
            Buffer::Inline(buffer) => buffer,
            Buffer::OutOfLine(buffer) => buffer,
        }
    }
}

fn main() {
    let rio = rio::new().unwrap();
    let stdin = std::io::stdin();

    let mut buf = Buffer::Inline(Default::default());
    let mut slice = buf.as_mut();
    let mut completion = rio.read_at(&stdin, &mut slice, 0);
    let _ = extreme::run(async { futures_util::poll!(&mut completion) });

    struct Oops<'a>(
        rio::Completion<'a, usize>,
        std::cell::Cell<Option<std::sync::Arc<Oops<'a>>>>,
    );
    let oops = std::sync::Arc::new(Oops(completion, Default::default()));
    oops.1.set(Some(oops.clone()));
    drop(oops);

    buf = Buffer::OutOfLine(vec![0; 100]);
    std::thread::sleep(std::time::Duration::from_secs(2));
    println!("{:?}", buf);
}

$ (sleep 1; yes) | cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/repro`
Segmentation fault (core dumped)

I found weird situation: if I constantly use fsync application works steadily, but if I change it on fsyncdata, it sometimes returns error: Bad file descriptor (os error 9). In linux man I found only one thing, which MAY cause an error (but does it?): (read the paragraph, which start from fdatasync() is similar to fsync()) https://man7.org/linux/man-pages/man2/fdatasync.2.html.
So do I understand it right: if some data sensitive metadata changed (size, for example) fdatasync returns error? I am a little bit confused, because bad file descriptor doesn't point on need to fsync metadata either.
Sorry in advance if this question is silly, I am not experienced enough:C

It appears that this crate has multiple soundness issues as evidenced by several issues, yet the fact that it is currently broken is not mentioned in the README, creating a false impression.

Hi, I stumbled upon this crate, and it seems have some soundness issues.

use std::{ fs, io };


fn main() -> io::Result<()> {
    let mut ring = rio::new().expect("create uring");
    let file = fs::File::open("Cargo.toml").expect("openat");

    let completion = {
        let mut data = vec![0; 12];
        let mut in_io_slice = io::IoSliceMut::new(&mut data);
        ring.read(&file, &mut in_io_slice, 0)?
    };

    let data = vec![0x42; 12];

    ring.submit_all()?;
    completion.wait()?;

    println!("{:?}", data);

    Ok(())
}

This is a simple use-after-free poc. the kernel will write data to freed memory, which will cause a memory error.

Hey there, first of all I have to say great work with this project. It's probably one of the most intuitive uring libraries out there. I've been working on a project using Rio along with Tokio. I'm using Rio both for file and network IO, and I noticed that most functions such as write_at take any type which implements AsRawFd, however some functions such as fsync explicitly take the std::fs::File type. I'm currently using tokio's File type(tokio::fs::File), since there are some file operations like retrieving metadata that I'd like to do asynchronously which isn't supported by Rio. I'm assuming you only accept a std::fs::File since you'd only ever expect to fsync a file rather than a socket, however this means I have to do some ugliness to use a tokio File type. Currently my work around for this is to grab the raw file descriptor from the tokio file and create a std file out of that descriptor, I don't believe this is any less safe than what Rio is doing, but it's kind of a pain. I'm wondering if you'd accept a PR that changed the couple of functions that expect Files to allow any type that implements AsRawFd, or if you find it concerning that this is potentially less safe or prone to misuse since then you can technically pass in something like a socket.

I'm trying to talk TCP using io_uring with rio. Now I got this working fine on my development machine, but it somehow fails to run on a remote server.

I did try some basic debugging but am failing to see what the problem is. I'm looking for any guidance on what I might try to debug this, or on things I should check.

Here's a simplified snippet of the TCP sender:

use std::{env, io, net::TcpStream};

// Set up io_uring
let config = rio::Config::default();
let ring = config.start()?;

// Open TCP stream
let stream = TcpStream::connect("127.0.0.1:6666")?;
stream.set_nonblocking(true)?;

// Keep sending buffer repeatedly
let buf = vec![0; 1024 * 8];
loop {
    let _written = ring.send(stream, &buf).wait()?;
}

On my development machine this successfully sends gigabytes per seconds. On the remote server this fails, it looks like it sends about 2.5MB, after which it fails to send anything more. I'm not seeing any errors or panics. These tests are done on the machine itself, with a sending and receiving process, LAN isn't involved here.

• Development machine (succeeds):

  # lsb_release -a
  Distributor ID:	Ubuntu
  Description:	Ubuntu 20.04 LTS
  Release:	20.04
  Codename:	focal
  # uname -a
  Linux axiom 5.6.16-050616-generic #202006030730 SMP Wed Jun 3 07:35:14 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Remote server (fails):

  # lsb-release -a
  Distributor ID: CentOS
  Description:    CentOS Linux release 7.8.2003 (Core)
  Release:        7.8.2003
  Codename:       Core
  # uname -a
  Linux localhost.localdomain 5.7.6-1.el7.elrepo.x86_64 #1 SMP Fri Jun 26 02:56:56 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

Judging by the output of /proc/kallsyms support for io_uring is available in both kernels, as syscalls such as io_uring_setup are listed.

This is about sending data, however, receiving data (read_at) doesn't work either, nothing is received. I did test both the sending and receiving io_uring process by connecting it to a sending/receiving std TcpStream without using io_uring.

What could be happening here? I'm not sure where to look.

Posting this here since I'm using rio. Not sure if there's a better place to be posting this.