google / moveit

moveit

A library for safe, in-place construction of Rust (and C++!) objects.

---

This is not an officially supported Google product.

moveit's Issues

A trait to patch pointers after moving

This isn't currently useful on stable, and likely won't be until rust-lang/rust#32838 (more specifically Allocator::grow_zeroed) and rust-lang/rust#31844 (at least #![feature(min_specialization)]) land, but an unsafe trait with a function like

/// Repairs an instance of `Self` located at `moved` back into a valid state.
///
/// # Safety
///
/// The data referenced by `moved` must have been
///
/// - previously located at `old_address`,
/// - must have been initialized as valid instance of `Self` there and
/// - must have been moved to its new location without modification since then.
unsafe fn repair_moved(old_address: NonNull<Self>, moved: Pin<&mut MaybeUninit<Self>>);

would be potentially helpful as optimisation for reallocating pinning collections.

As I said before, I don't expect this to be currently useful, and also would not currently have a use for it myself.
I just came across it while writing a blog post about pinning and wanted to jot down the idea somewhere since it doesn't fit that text.

The traits in this crate will most likely see a huge boost in helpfulness once specialisation allows them to be default-implemented for T: Unpin. Feel free to ping me when that happens and I should have time to gladly turn this feature into a pull-request (with hopefully better naming-sense, as I'm not sure repair_moved checks out and is consistent with the other traits in the crate).

[Feature Request] Emplacing into allocator aware containers

Currently Emplace is implemented for Box<T>, Rc<T>, Arc<T>. It would be great if Box<T, A>/Rc<T, A>/Arc<T, A> were supported too. This functionality could be feature gated, since allocator_api is currently nightly-only.

Would a PR for such a feature be (possibly) accepted? I don't have the capacity right now, but I may take a stab at it in the future.

Implementations of `DerefMove` for `Pin<_>` types are unsound

DerefMove allows to get a MoveRef from self, and that's not pinned anymore. If self is a Pin type then this is unsound

use std::marker::PhantomPinned;
use std::pin::Pin;

use moveit::move_ref::DerefMove;
use moveit::slot;

struct Unmovable {
    _marker: PhantomPinned,
}

impl Unmovable {
    fn assert_pinned(self: Pin<&mut Self>) {
        println!("Pinned")
    }
    fn assert_unpinned(&mut self) {
        println!("Uninned")
    }
}

fn main() {
    // This also works for creating `unmovable`:
    // 
    // moveit::moveit!(let mut unmovable = moveit::new::of(Unmovable {
    //     _marker: PhantomPinned,
    // }));
    let mut unmovable = Box::pin(Unmovable {
        _marker: PhantomPinned,
    });
    unmovable.as_mut().assert_pinned();

    slot!(
        #[dropping]
        drop_slot: _
    );
    unmovable.deref_move(drop_slot).assert_unpinned();
}

Safer constructor writing for self-referential types

Currently to write a constructor/move-constructor/copy-constructor, you have to deal with a lot of unsafe code and boilerplate steps. Each constructor has to do:

1. un-pin a MaybeUninitialized reference
2. create the being-constructed type T
3. initialize the MaybeUninitialized reference with the object from (2)
4. get a pointer, and then a reference, to the MaybeUninitialized memory
5. use the reference from (4) to set up self-referential pointers in the object from (2)

The only steps that the type T actually cares about and wants to define are (2) and (5), but today it has to write the whole algorithm in each constructor/move-constructor/copy-constructor. Like so:

fn myctor(dest: Pin<&mut MaybeUninit<<T as SafeCtor>::Output>>) {
  unsafe {
    let maybe_uninit_ref = Pin::into_inner_unchecked(dest);  // 1
    let (out, init) = T { ... };  // 2 (using the copy- or moved-from object in copy- or move-constructors)
    *maybe_uninit_ref = MaybeUninit::new(out);  // 3
    let t_ref = &mut *maybe_uninit_ref.as_mut_ptr();  // 4
    setup_self_references(t_ref, init); // 5
  }
}

I have implemented a set of "SafeCtor" safe traits that a type can implement in order to have minimal unsafe code in their implementations (even none if they do not require moving/offsetting pointers).

These traits expose the step (2) above as construct() -> (Self, Data) or copy_construct(from: &Self) -> (Self, Data) or move_construct(from: Pin<&mut Self>) -> (Self, Data). These trait functions provide the "constructed-from" object as a reference that can be directly used from safe rust. And they output Data that can be collected from the "constructed-from" object and represented in a useful way for initialization of self-references.

And they expose step (5) above as an initialize(this: &mut Self, d: Data) that consumes the constructor's side-channel Data to set up any self-referential state in the constructed object.

The other steps (1), (3) and (4) are performed in unsafe implementations in the unsafe CopyCtor and unsafe MoveCtor traits, or in a ctor::construct() function that replaces the direct use of ctor::from_placement_fn() from the being-constructed type.

@mcy has pointed out initial self-referential object construction could be done by exposing this 2-stage construction paradigm (i.e. steps (2) and (5)) through the constructing function such as ctor::new_with<T>(init: T, setup: FnOnce(Pin<&mut T>)), instead of using traits and requiring T : SafeCtor<Output = T>.

It presents a nice advantage: It would force the caller to construct the type T, which is reasonable for initial construction, and it even allows calling different constructors.

But it has a challenge as well, as copy- and move-constructors can not be implemented in the same way:

• The copy and move requires boilerplate to execute before constructing the object T, such that a reference to the copied-from or moved-from object can be given to it. So the caller can not just construct a T themselves.
• A data channel between the copied-from/moved-from object and the self-referential initialization step is required, which is provided by the Data passed between the traits' copy/move constructors and initialize.

Here is an example of a self-referential type that points into a buffer. When moving, the move-constructed type must be able to set up a new self-reference at the same offset (or in this case, the next offset).

use moveit::*;
use std::marker::PhantomPinned;
use std::pin::Pin;
use std::ptr::null_mut;

struct Cycle {
    num: [i32; 10],
    ptr: *mut i32,
    _pin: PhantomPinned,
}
impl Cycle {}

struct CycleInit {
    offset: usize,
}

impl ctor::SafeCtor for Cycle {
    type Output = Self;
    type InitializeData = CycleInit;
    fn construct() -> (Self, CycleInit) {
        (
            Cycle {
                num: [9, 8, 7, 6, 5, 4, 3, 2, 1, 0],
                ptr: null_mut(), // Set by initialize().
                _pin: PhantomPinned,
            },
            CycleInit { offset: 0 },
        )
    }
    fn initialize(this: &mut Self, init: CycleInit) {
        this.ptr = unsafe { (&mut this.num).as_mut_ptr().add(init.offset) }
    }
}
impl ctor::SafeMoveCtor for Cycle {
    fn move_construct(other: Pin<&mut Self>) -> (Self, CycleInit) {
        (
            Cycle {
                num: other.num,
                ptr: null_mut(), // Set by initialize().
                _pin: PhantomPinned,
            },
            CycleInit {
                // Move-constructing will move the `ptr` to the right one
                // so we can observe it.
                offset: 1 + unsafe { other.ptr.offset_from(&other.num[0]) as usize },
            },
        )
    }
}

The only unsafe code here is that which deals with the pointer into num. The same self-referential initialization is shared between each of construction, copy-construction, and move-construction. It's not obvious if forcing that behaviour to be shared is undesirable.

Here are the traits, which would appear in ctor.rs, as they provide generic implementations of CopyCtor and MoveCtor for any type implementing SafeCopyCtor or SafeMoveCtor respectively. The SafeCtor trait provides the self-referential setup step (2nd stage of construction) used in all construction paths, but only provides a single type-construction function without any parameters. A way to provide multiple constructors is needed.

pub trait SafeCtor {
  type Output;
  type InitializeData;
  fn construct() -> (Self::Output, Self::InitializeData);
  fn initialize(o: &mut Self::Output, init: Self::InitializeData);
}
pub trait SafeCopyCtor: CopyCtor + SafeCtor {
  fn copy_construct(other: &Self::Output) -> (Self::Output, Self::InitializeData);
}
pub trait SafeMoveCtor: MoveCtor + SafeCtor {
  fn move_construct(other: Pin<&mut Self::Output>) -> (Self::Output, Self::InitializeData);
}

unsafe impl<T: SafeCopyCtor + SafeCtor<Output = T>> CopyCtor for T {
  unsafe fn copy_ctor(src: &<T as SafeCtor>::Output, dest: Pin<&mut MaybeUninit<<T as SafeCtor>::Output>>) {
    let dest_uninit = Pin::into_inner_unchecked(dest);
    let (out, init) = <T as SafeCopyCtor>::copy_construct(src);
    *dest_uninit = MaybeUninit::new(out);
    <T as SafeCtor>::initialize(&mut *dest_uninit.as_mut_ptr(), init);
  }
}

unsafe impl<T: SafeMoveCtor + SafeCtor<Output = T>> MoveCtor for T {
unsafe fn move_ctor(
  mut src: Pin<MoveRef<<T as SafeCtor>::Output>>,
  dest: Pin<&mut MaybeUninit<<T as SafeCtor>::Output>>,
) {
  let dest_uninit = Pin::into_inner_unchecked(dest);
  let (out, init) = <T as SafeMoveCtor>::move_construct(src.as_mut());
  *dest_uninit = MaybeUninit::new(out);
  <T as SafeCtor>::initialize(&mut *dest_uninit.as_mut_ptr(), init);
}
}

pub fn construct<T: SafeCtor<Output = T>>() -> impl Ctor<Output = T> {
  unsafe {
      from_placement_fn(|dest| {
          let dest_uninit = Pin::into_inner_unchecked(dest);
          let (out, init) = <T as SafeCtor>::construct();
          *dest_uninit = MaybeUninit::new(out);
          <T as SafeCtor>::initialize(&mut *dest_uninit.as_mut_ptr(), init);
      })
  }
}

Using the safe traits looks much like using the existing ones, except that T::new() (or similar) is not called directly.

fn main() {
    moveit!(let x = new ctor::construct::<Cycle>());
    moveit!(let y = new ctor::mov(x));
    println!("{}", unsafe { *y.ptr });
}

Perhaps SafeCtor could be renamed to SelfReferenceSetup, the contruct() trait functions removed, and the ctor::construct() function modified to receive a T by value instead. That would handle both multiple construction paths, while still sharing the setup of self-references.

CONTRIBUTING.md is weird

https://github.com/google/moveit/blob/d0f1157fc99206bc04643724c57cc3b2c550b806/CONTRIBUTING.md

Project name may infringe copyright of direct compatitor of Google

Hi,

I'm not affiliated with moovit in any way.
I just wanted to give you a heads up in case there is a problem.

moveit is very close to moovit which could potentially present a problem for your employer, FYI.

move_ref::PinExt::as_move doc test leaks

from cargo miri test

---- src/move_ref.rs - move_ref::PinExt::as_move (line 278) stdout ----
Test executable failed (exit code 1).

stderr:
The following memory was leaked: alloc1499 (Rust heap, size: 4, align: 4) {
    05 00 00 00                                     │ ....
}

[Question] Is it reccomended/intended or not to extract an Unpin type from a Pin<MoveRef<T>>?

Recently, I found myself writing autocxx bindings to "plain old data" types. With the desire to not have to care about pinning/boxing/whatever these types, I figured out that I could extract the value like so, without writing any of my own unsafe code:

fn unwrap_new<T>(obj: impl New<Output = T>) -> T
where
    T: Unpin,
{
    MoveRef::into_inner(Pin::into_inner(moveit!(obj)))
}

Finding the correct sequence of things to do was not clear from documentation, nor does there appear to be a similar helper inside of moveit itself (or autocxx for that matter), leading me to think that I'm touching a part of the library that is either not well exercised, or I'm doing something that is not recommended. Therefore, I wanted to ask before I push such code into production.

Run miri as part of the CI

It might be worthwhile to run the tests with miri to check for undefined behavior. Adding the following job (adapted from Rand) to the GitHub actions should work:

  test-miri:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install toolchain
        run: |
          MIRI_NIGHTLY=nightly-$(curl -s https://rust-lang.github.io/rustup-components-history/x86_64-unknown-linux-gnu/miri)
          rustup default "$MIRI_NIGHTLY"
          rustup component add miri
      - name: Test rand
        run: |
          cargo miri test

[no_alloc]: Extend support for embedded systems by removing the need for alloc

TL;DR: An option to compile without feature = "alloc" would be great :)

---

There is a feature flag for alloc already but it's not completely implemented.

A modified basic example (using a *mut self reference) compiles if I #[cfg(feature = "alloc")] out the use core::alloc::{...} and impl for Box, Rc, ... in mod.rs. But there are other parts which might need to be adjusted.

Reduced feature set would be fine IMO as the most basic is already nice enough to just wrap self-referencing types from C FFI on the stack.

I'm not well versed all these macro-magic though :)

Stack overflow

moveit/src/alloc_support.rs

Line 81 in 5f9f8c7

let mut uninit = Box::new(MaybeUninit::<T>::uninit());

rust-lang/rust#63291

Assigning MaybeUninit::::uninit() to a local variable is usually free, even when size_of::() is large. However, passing it for example to Arc::new causes at least one copy (from the stack to the newly allocated heap memory) even though there is no meaningful data.

AFAIK, there is no way to construct Arc<MaybeUninit<T>> and Rc<MaybeUninit<T>> efficiently until feature(new_uninit) is stable.

Unsoundness in moveit::ctor::[try_]assign

>> use std::pin::Pin;
>> let mut pinned = Box::pin(Box::new(123));
>> moveit::ctor::try_assign(pinned.as_mut(), moveit::ctor::from_try_fn(|| Err(())))
Err(())
>> let b: Box<i32> = *Pin::into_inner(pinned);
>> b
-1594092836
>> b
-1594092772
>> b
-1594092836

Essentially, [try_]assign doesn't guarantee that lvalue won't be accessible if the ctor panics or (in TryCtor's case) returns an Err, so we can access lvalue.deref_mut() even after ptr::drop_in_place was called on it.