reloaded-project / reloaded.memory.buffers

[C# & Rust] An implementation of efficient, shared, concurrent and permanent storage of objects in unmanaged memory in static, non-changing locations that last the lifetime of a given process.

The Reloaded Buffers Library

Allocate Memory, & Knuckles

About

Reloaded.Memory.Buffers is a library for allocating memory between a given minimum and maximum memory address, for C# and Rust.

With the following properties:

• Memory Efficient: No wasted memory.
• Shared: Can be found and read/written to by multiple users.
• Static: Allocated data never moves, or is overwritten.
• Permanent: Allocated data lasts the lifetime of the process.
• Concurrent: Multiple users can access at the same time.
• Large Address Aware: On Windows, the library can correctly leverage all 4GB in 32-bit processes.
• Cross Platform: Supports Windows, OSX and Linux.

Note: Rust/C port also work with FreeBSD (untested), and has partial (limited) Android support.

Wiki & Documentation

For full documentation, please see the Wiki.

Example Use Cases

These are just examples:

• Hooks: Hooking libraries like Reloaded.Hooks can reduce amount of bytes stolen from functions.
• Libraries: Libraries like Reloaded.Assembler require memory be allocated in first 2GB for x64 FASM.

Usage

!!! info "The library provides a simple high level API to use."

!!! info "See Wiki for Rust usage"

Get A Buffer

Gets a buffer where you can allocate 4096 bytes in first 2GiB of address space.

var settings = new BufferSearchSettings()
{
    MinAddress = 0,
    MaxAddress = int.MaxValue,
    Size = 4096
};

// Make sure to dispose, so lock gets released.
using var item = Buffers.GetBuffer(settings);

// Write some data, get pointer back.
var ptr = item->Append(data); 

Get A Buffer (With Proximity)

Gets a buffer where 4096 bytes written will be within 2GiB of 0x140000000.

var settings = BufferSearchSettings.FromProximity(int.MaxValue, (nuint)0x140000000, 4096);

// Make sure to dispose, so lock gets released.
using var item = Buffers.GetBuffer(settings);

// Write some data, get pointer back.
var ptr = item->Append(data); 

Allocate Memory

Allows you to temporarily allocate memory within a specific address range and size constraints.

// Arrange
var settings = new BufferAllocatorSettings()
{
    MinAddress = 0,
    MaxAddress = int.MaxValue,
    Size = 4096
};

using var item = Buffers.AllocatePrivateMemory(settings);

// You have allocated memory in first 2GiB of address space.
// Disposing this memory (via `using` statement) will free it.
item.BaseAddress.Should().NotBeNull();
item.Size.Should().BeGreaterOrEqualTo(settings.Size);

You can specify another process with TargetProcess = someProcess in BufferAllocatorSettings, but this is only supported on Windows.

Crate Features (Rust)

• std: [Enabled by Default] Enables use of standard library.
• external_processes: Support external processes (windows only).
• no_format: Disables formatting code in errors, saving ~8kB of space.
• size_opt: Makes cold paths optimized for size instead of optimized for speed. [Requires 'nightly' Rust]
• c_exports Provides C exports for the library.

Community Feedback

If you have questions/bug reports/etc. feel free to Open an Issue.

Contributions are welcome and encouraged. Feel free to implement new features, make bug fixes or suggestions so long as they meet the quality standards set by the existing code in the repository.

For an idea as to how things are set up, see Reloaded Project Configurations.

Happy Hacking 💜

reloaded.memory.buffers's Issues

Executable Memory-Mapped Files on Android are Unsupported in non-Root Scenarios

On Android, it appears that creating a memory-mapped file with 'execute' permissions in a location accessible by multiple processes is not possible (without root). The Android platform seems to enforce strict W^X (Write XOR Execute) policies, which prevent memory from being simultaneously writable and executable in shared memory locations.

There's also another issue. To utilize common shared storage in Android, the requisite permissions must be set in AndroidManifest.xml.

However, despite setting the appropriate permissions, creating a RWX memory mapped file on Android does not seem to work regardless. It may be possible if you memory map the file twice (non-writeable for executing, and writeable for writing, but I have not yet tested).

Current Behaviour

Currently on Android, library will not be able to detect existing LocatorHeader(s) created by other instances of the library in the same process. A new LocatorHeader will always be created.

[Rust] Tests Should Delete Physical Memory Mapped File between Runs when Calling `LocatorHeaderFinder::reset`

This sometimes causes test failures in macOS, when tests are ran in different than usual order.
This should probably be fixed.

On macOS M1, pages enforce strict W^X and cannot be write and execute at the same time.

Related to: Reloaded-Project/Reloaded.Hooks-rs#1

Currently we toggle between R^X and R^W, because this is forced on us by the mach_ APIs. This is not ideal, in a situation like hooks, it's possible we might be toggling code in process of being executed, in which case, we're smoked.

Unfortunately the JIT APIs pthread_jit_write_protect_np only work on memory allocated with mmap, not the mach APIs, and allocating at specific address with mmap is broken on macOS. Well played.

It would be nice to experiment with mapping each custom page twice over, but that's unfortunately not a possibility right now. Without access to one of these machines, I can't test it very well, short of paying some cloud company for access (lol) or hassling someone too much for my liking.

[Android] On AArch64, Build fails when Running `cross test --target aarch64-linux-android` because it doesn't recognise `__clear_cache` intrinsic.

Tried to use this intrinsic from C (__builtin__)clear_cache).
Tried to use the LLVM export (named __clear_cache).

Both GCC and Clang should recognise this intrinsic, yet build fails (on GCC); but only on ARM based platforms. I have no idea why. Android source itself uses this intrinsic fine. No idea what is going on.