matthias247 / futures-intrusive Goto Github PK

The #[derive(Clone)] macro thinks MutexType needs to be Clone as well in this code:

This isn't intended, is it?

If not, I'd suggest implementing a hand-rolled Clone instead.

This seems related to #3. Was Sync for this type just an oversight, or might it be safely added? I guess the question is, if concurrent access to a *mut ListNode<WaitQueueEntry? could be safe?

Current compile error:

the trait `std::marker::Sync` is not implemented for `*mut futures_intrusive::intrusive_singly_linked_list::ListNode<futures_intrusive::sync::semaphore::WaitQueueEntry>`
    = note: required because it appears within the type `futures_intrusive::intrusive_singly_linked_list::ListNode<futures_intrusive::sync::semaphore::WaitQueueEntry>`
    = note: required because it appears within the type `futures_intrusive::sync::semaphore::GenericSemaphoreAcquireFuture<'static, parking_lot::raw_mutex::RawMutex>`

A week or so ago parking_lot updated to 0.11, bringing along with it lock_api 0.4.X. I don't know what would be required to implement this, and due to the increment of lock_api would unfortunately have to be a breaking change.

As of right now, as people are starting to upgrade to newer versions of parking_lot, it's also bringing with it the new version of lock_api, causing duplicate deps and possible trait mismatches.

If you're willing, I can help make progress on this.

The published version of futures-intrusive is still using out of date dependencies that were fixed in #50. Would it be possible to soon, or sometime soon, cut a new release with these up-to-date dependencies?

I have a use case where I would like to know the previous state of the channel after I called channel::close() on it. I'll look at the code but if i recall correctly, this could be done for free.

So essentially this signature:

/// Closes the channel, returning whether the channel was already closed.
fn close(&self) -> bool;

// Or alternatively:

/// Closes the channel, returning the previous state of the channel.
fn close(&self) -> PrevState;

enum PrevState {
  Open,
  Close,
}

I think this is technically considered a breaking change, but its relatively insignificant.

Would you all be open to moving the intrusive linked list and pairing heap into another crate? I'd like to write some specialized channel data structures, and having these primitives available would help a lot.

If so, I'm happy to do it!

Hi. Thanks for such a great crate! I'm wondering, why LocalChannel is explicitly stuck to ArrayBuf?

I believe this is because they contain &dyn ChannelSendAccess and &dyn ChannelReceiveAccess trait objects internally.

Minimally, being Send would enable them to be used on the multi-threaded tokio runtime.

Hi there, and thanks for your work on this library.

I don't feel very confident on how unwind safety works, but I noticed that
std::cell::UnsafeCell<futures_intrusive::channel::mpmc::ChannelState<ControlMessage<A>, futures_intrusive::buffer::ring_buffer::if_std::HeapRingBuf<ControlMessage<A>>>> may contain interior mutability and a reference may not be safely transferrable across a catch_unwind boundary

Is this in fact true, or are we missing a marker trait?

Currently timers are just stored in a simple double linked list. This list needs to be searched on each timer insertion, which makes the operation O(n) in the already cache-inefficient linked-list. It would be nice to improve the way timers are stored in order to make them better scale to a higher number of timers.

Some ideas:

• use a skiplist for timer. a timer node could also point n (10/100?) entries forward, and if the to-be-inserted timer is still further ahead we directly jump there.
• use another two dimensional list for storing timers. E.g. only store timers in an outer linked list (bucket) if they expire in the same ms. Link buckets (different ms intervals) via another linked list.

The downside of such a change is that it would increase the memory utilization of timers by up to 2 extra pointers per node (which is about +50%).

NoopLock implements RawMutex, which is an unsafe trait with the requirement

Implementations of this trait must ensure that the mutex is actually exclusive: a lock can't be acquired while the mutex is already locked.

However, NoopLock does not provide any such guarantees.

That means that a lock_api::Mutex<NoopLock, T> won't prevent multiple lock guards from being acquired simultaneously, which is unsound.

Currently the channels implemented by crate must allocate the capacity it will use ahead of time because the RingBuf trait does not allow capacity growth. This can be a problem in situations where you want to limit the capacity but don't want to allocate all the capacity ahead of time, or when you don't want to limit the capacity.This really only applies to shared types since they can grow in size.

One approach would be to distinguish between the limit and the capacity. RingBuf::can_push would use the limit to determine whether the channel is full. The initial capacity of the buffer would still be using capacity. The current behavior would correspond to a channel where limit == capacity. A unbounded channel would be more like limit = usize::max_value() and capacity = 0. In such a case the ring buffer would grow as needed.

I ran into an issue using the switchyard crate, see BVE-Reborn/switchyard#20. It depends on both futures-intrusive and parking_lot but only using their minor versions, 0.4 and 0.11 respectively. This works fine with futures-intrusive = "0.4.0" because it uses the same minor version of parking_lot but running cargo update getting futures-intrusive = "0.4.1" breaks it. Is this because the parking_lot::RawMutex is in the public API of futures-intrusive? If that is the case, then I think "0.4.1" would need to be yanked and replaced with "0.5.0".

I think the GenericMutexGuard object should me marked explicitly as !Sync, it looks like it was automatically given the Sync trait because it only consists of a GenericMutex reference which itself is Sync. However, the lock-guard object itself shouldn't be usable across threads because it assumes it has the lock acquired. (This issue was found by the Rust group at @sslab-gatech).

Here's a demonstration of how this can cause Cell, a non-Sync but Sendable type to be used across threads to create a data race:

#![forbid(unsafe_code)]

use futures_intrusive::sync::{GenericMutexGuard, Mutex};

use crossbeam_utils::thread;
use std::cell::Cell;

static SOME_INT: u64 = 123;

fn main() {
    #[derive(Debug, Clone, Copy)]
    enum RefOrInt<'a> {
        Ref(&'a u64),
        Int(u64),
    }
    let cell = Cell::new(RefOrInt::Ref(&SOME_INT));

    let futures_mutex: Mutex<Cell<_>> = Mutex::new(cell, false);
    let mutex_guard: GenericMutexGuard<_, Cell<_>> = futures_mutex.try_lock().unwrap();

    thread::scope(|s| {
        let guard_ref = &mutex_guard;
        let child = s.spawn(move |_| {
            let smuggled = &(**guard_ref);

            println!("In the thread: {:p} {:?}", smuggled, *smuggled);
            loop {
                // Repeatedly write Ref(&addr) and Int(0xdeadbeef) into the cell.
                smuggled.set(RefOrInt::Ref(&SOME_INT));
                smuggled.set(RefOrInt::Int(0xdeadbeef));
            }
        });

        println!("In main: {:p} {:?}", &(*mutex_guard), *mutex_guard);
        loop {
            if let RefOrInt::Ref(addr) = mutex_guard.get() {
                // Hope that between the time we pattern match the object as a
                // `Ref`, it gets written to by the other thread.
                if addr as *const u64 == &SOME_INT as *const u64 {
                    continue;
                }

                // Due to the data race, obtaining Ref(0xdeadbeef) is possible
                println!("Pointer is now: {:p}", addr);
                println!("Dereferencing addr will now segfault: {}", *addr);
            }
        }
    });
}

Output:

In main: 0x7ffc45082668 Cell { value: Ref(123) }
In the thread: 0x7ffc45082668 Cell { value: Ref(123) }
Pointer is now: 0xdeadbeef
Return Code: -11 (SIGSEGV)

The next_expiration API for the GenericTimerService doesn't work well in a multithreaded context. If an eventloop works like:

loop {
    let expiration = timer.next_expiration();
    let events = wait_for_events(expiration);
    handle_events(events);
}

it might miss the expiry of a timer if that one gets registered while the loop is already blocked in wait_for_events with a longer timeout.

Therefore the only reliable way to drive a timer in a multithreaded context is currently to poll the timer in regular intervals independent of the number of registered timers.

To fix this issue GenericTimerService could provide a hook that allows to wakeup the eventloop if a new timer gets registered whose expiry data is earlier than the previously registered timer.

A Semaphore with a SemaphoreReleaser without a lifetime parameter would be quite useful. I imagine that using a design similar to channel::shared would do the trick.

I'm pretty sure this crate ends up aliasing mutable references, as each future, when polled, holds a mutable reference to the data at the same time as the linked list does.

As an example: suppose there is a ManualResetEvent, and this happens:

• .wait() is called, and the future is polled once. This will insert a raw pointer to its WaitQueueEntry inside the ManualResetEvent's linked list.
• At this point, we have a mutable reference to the WaitQueueEntry transitively through the future.
• .set() is called. This then calls LinkedList::reverse_drain, which creates a &mut ListNode<WaitQueueEntry> that aliases with the one held by the future. This is UB.

However I couldn't get Miri to trigger on this, so I'm not sure.

ArrayRingBuf's capacity implementation creates a shared reference to a possibly-uninitialized array. Even though the call to .len will not read from any of the array's memory, I believe the current unsafe code guidelines indicate that the mere creation of the reference is UB, even if it's never physically read through.

One approach to fixing this would be to move from MaybeUninit<A> where A: RealArray<T> to A where A: RealArray<MaybeUninit<T>>. This both solves the length issue, and IMO represents the underlying semantics better where the array itself is always initialized, but some of its values may not be.

Alternatively, you could add a RealArray::LEN associated constant to avoid having to create a reference in the first place.

Its poll implementation only saves off its waker on the first poll, but not subsequent ones: https://github.com/Matthias247/futures-intrusive/blob/master/src/sync/manual_reset_event.rs#L107-L113. If the future's task is migrated from one executor to another, its waker will change but won't be updated internally and so won't be awoken appropriately when the event is triggered.

It just needs this in the PollState::Waiting case:

if wait_node.task.as_ref().map_or(true, |w| !w.will_wake(cx.waker())) {
    wait_node.task = Some(cx.waker().clone());
}

I'm trying to limit concurrent requests using Semaphore. But the Semaphore itself cost tons of time when there are quite a lot of futures trying to acquire it.

For example, this code spawns 10^4 futures and should run in 10000 / 100 = 100ms. But it costs 3.06s in fact. For 2*10^4 futures, it costs 14.40s (expect to be 200ms) !

extern crate async_std; // features = ["attributes"]
extern crate futures_intrusive;
use async_std::task::{sleep, spawn};
use futures_intrusive::sync::Semaphore;
use std::{sync::Arc, time::Duration};

async fn send_request() {
    // Send requests...
    sleep(Duration::from_millis(1)).await;
}

#[async_std::main]
async fn main() {
    let sem = Arc::new(Semaphore::new(false, 100));
    let hs = (0..10000)
        .map(|_| {
            let sem = sem.clone();
            spawn(async move {
                let _guard = sem.acquire(1).await;
                send_request().await;
            })
        })
        .collect::<Vec<_>>();
    for h in hs {
        h.await;
    }
}

I use cargo-flamegraph to do profiling, which shows that it cost a lot on peeking the last element from single linked list. Related code.

It seems to be designed to wake the last waiter. But why it is still required even when !is_fair?

Hi! Big fan of this library :) I wanted to note a gotcha I ran into today: dropping a OneshotReceiver causes the channel to be closed, even if there is an active ChannelReceiveFuture reading from it. This is a bit annoying since often I'm passing the receiver to something that just wants a future, but then I have to bundle the OneshotReceiver together with it. So for now I've switched to futures::channel::oneshot for oneshots.

In general, I'm not sure why the OneshotReceiver can't implement Future directly and avoid the entire .receive() dance. At least, this behaviour is something that should be documented :)

Hi @Matthias247 I just wonder if this crate should be used ? Last release was a year ago and there is few unsound issues I am not sure what the status is re: fix - if any.

Would it be helpful to nudge the people that it might not be best to use this in production anymore e.g. informational advisory around that the crate is not maintained anymore ?

Thanks a lot

Arc is supported in alloc, not just std.