microsoft / cheriot-rtos

The RTOS components for the CHERIoT research platform

CHERIoT RTOS

This repository contains the core RTOS for the CHERIoT platform. This is currently a research project that has been open sourced to enable wider collaboration. It is not yet in a state where it should be used in production: in particular, security issues will currently be fixed in the main branch of the repo with no coordinated disclosure.

To use this, you will also some dependencies. The getting started guide describes in detail how to build these:

• A version of LLVM with CHERIoT support
• An implementation of the ISA (e.g. CHERIoT-Ibex or the emulator generated from the formal model))

These dependencies are pre-installed in the dev container that will be automatically downloaded if you open this repository in Visual Studio Code or by hitting . to open it in GitHub Code Spaces.

To clone this repository, make sure that you use git clone --recurse so that you get submodules. This repository contains symbolic links. IMPORTANT: If you wish to clone this repository on Windows, make sure that you have enabled Developer Mode and run git config --global core.symlinks true. You must do this before cloning the repository.

The getting started guide describes how to install these and how to build the test suite and examples in this repository.

The RTOS is privilege separated into a small number of core components as described in the architecture document. The C/C++ extensions used by the compartmentalisation model are described in the language extensions document.

If you have questions, please see the frequently asked questions document or raise an issue.

Building firmware images

This repo contains the infrastructure for building CHERIoT firmware images.

NOTE: The build system is currently based on xmake, but we have encountered a number of issues with our use of xmake and may switch to an alternative build system at some point.

Clone this repo into your project and create an xmake.lua referring to it. The file should start with this line:

includes("{path to this repo}/sdk")

This will enable debug and release configuration (specified with -m {release,debug}). Both are compiled with -Oz (optimise for size, even at the expense of performance).

Next you need to specify that you want to use the compiler provided by this SDK:

set_toolchains("cheriot-clang")

Now you can add targets. We provide helpers for creating library, compartment, and firmware targets. These work just like normal xmake targets:

library("lib")
    add_files("shared_c_file.c", "shared_cxx_file.cc")

compartment("example")
    add_files("example/example.c")

firmware("example-firmware")
    add_deps("lib", "example")
    on_load(function(target)
        target:values_set("threads", {
            {
                compartment = "example",
                priority = 1,
                entry_point = "entry_point",
                stack_size = 0x400,
                trusted_stack_frames = 2
            }
        })

The firmware description specifies the compartments and libraries that this system depends on and specifies the threads. Threads are listed as a Lua array of objects, each of which has the following keys:

• compartment specifies the name of the compartment in which this thread starts.
• entry_point specifies the name of the exported function from that compartment that the thread will start executing. This function must take no arguments and return void.
• stack_size specifies the size, in bytes, of the stack for this thread.
• trusted_stack_frames specifies the number of trusted stack frames (the maximum depth of cross-compartment calls possible on this thread). Note that any call that may yield is likely to require at least one additional trusted stack frame to call the scheduler so, for example, a blocking call to malloc requires three stack frames (the caller, the allocator, and the scheduler).

$ xmake config --sdk={path to CHERIoT LLVM tools}
$ xmake

This will create the output in build/cheriot/cheriot/{release,debug}/{name of firmware target}. It will also create a .dump` file in the same location giving the objdump output of the same target.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

cheriot-rtos's Issues

Loader does not check interrupt status for thread entry points

If you export an entry point function with interrupts disabled and use it as a thread entry point, it will run with interrupts enabled. This is annoying for realtime control loops and is fixable: we look up the entry point by its export table entry and so can set the relevant bit in mstatus if necessary. Doing this for the highest-priority thread means that it runs, uninterrupted, until it explicitly yields (and then resumes with interrupts disabled).

Return registers not cleared on cross-compartment return

The compiler / switcher are not currently clearing unused return registers, which can leak capabilities from called compartments. There are two possible fixes for this:

• The compiler should emit explicit moves to zero ca0 and ca1 if they are not used.
• The compiler should emit flags in the export table entry telling the switcher which to export.

Most cross-compartment calls are likely to return one value and so the first would typically add one (16-bit) instruction to each exported function. This is probably the best idea for now, because bits in the export table entry are scarce.

Separate permissions in quotas

It would be nice to have allocation capabilities that allow claiming but not allocation. This provides some defence in depth against compromised compartments. Once #140 is created, we should also separate allocation and deallocation rights so that you can hand out a sub-quota that allows allocation but not deallocation, or one that allows cleanup and returning memory to the original quota, but not allocation.

exercises/01 fails when run in docker against ibex

As noted at the Sonata Hackathon exercise/01.compartmentalisation doesn't run in the dev docker, although it does work if compiled and run on the Sonata board. I tested against the latest commit (771f2bb - Jun 3rd) in case any of the recent changes to the dev contained had fixed it, but it still fails for me.

Digging in a little the line that fails in Ibex is generating the secret in secret.cc

    secret = static_cast<int32_t>(rdcycle64());

Replacing this with, for example, makes everything run fine

    //secret = static_cast<int32_t>(rdcycle64());
    secret=666;	

Commenting out the code in secret.cc which waits for the uart (so the call to rdcycle64() is in effect the first call in the therad) also makes no difference.

Adding a compartment_error_handler() gives the error as:

JavaScript compartment: Detected PermitAccessSystemRegistersViolation(0x18).  Register PCC(0x20) contained invalid value: 0x2004ba08 (v:0 0x2004a7c0-0x2004cb40 l:0x2380 o:0x0 p: G R-cgm- X- ---)

Calling rdcycle64() in any of the examples with IBEX works fine, so I'm assuming there is some difference in the build that I'm missing, but the only thing I spotted is that exercise xmake.luna has:

option("board")
    set_default("ibex-safe-simulator")

whereas the examples use

option("board")
    set_default("sail")

I'm happy to dig further if someone can point me in the right direction.

Benchmarks are bitrotted

I tried to build the stack-usage benchmark to test #193 and found that it had bitrotted due to the recent changes to debug infrastructure. It looks like all the benchmarks need fixing except the compartment call benchmark, which was fixed by @hlef . We should:

1. Fix the benchmarks
2. Build them in CI so this doesn't happen again.

Destruction lock

We currently have a nice priority inheriting mutex, but for a lot of use cases we want a lock that can be upgraded to a destruction mode. This needs the following properties:

• It sets an extra bit in the futex word that causes other attempts to acquire it to fail.
• It wakes any pending waiters and forces them to fail to acquire the lock immediately.

This makes it easy to have an object containing a lock and the lock used to protect the object during destruction. Normal operations acquire the lock, operate over the object, and release the lock. Destruction operations acquire the lock, upgrade it to destruction mode, destroy the object. As soon as the object is in destruction mode, all threads waiting on the lock are woken and fail, all future attempts to lock fail either by trapping (the object is gone, don't try to acquire it) or by detecting that the state is 'going away'.

Large stacks can break the software revoker

A large stack region can cause the software revoker to get an untagged capability for the stacks area. The loader should round the bounds up a bit to ensure that it has access to the entire range, even if it means having slightly wider access.

Bug in zero_stack in switcher may lead to unexpected stack pointer value in callee

The comment to the zero_stack macro in the switcher claims that the \base register is left pointing at \top. This is not true if \top is not 32-byte aligned -- it is left pointing 16 bytes below \top.

For the use on the compartment call path this means that the stack pointer passed to the callee may be 16-bytes less than expected. This is probably harmless because, although it wastes 2 bytes of stack, on return the correct stack pointer is restored from the trusted stack in pop_trusted_stack.

The use in pop_trusted_stack uses ct2 as the \base and nothing relies on its value afterwards.

The fix is probably to increment the \base by 16 at the end of zero_stack, although this means one unnecessary instruction in the pop_trusted_stack use.

Compiler / switcher does not validate non-software-visible pointers

There are two important pointers held in registers that may introduce attack vectors:

• If a function takes more arguments than fit in 6 registers, ct0 contains a pointer to stack space.
• If a function returns a structure that does not fit in registers, ca0 contains caller stack space to store the result.

Both of these should have RWlgm permissions and be sufficiently large that they can hold the values that the compiler expects. None of the code in the RTOS currently uses this part of the ABI, but before we can the compiler should be fixed so that it checks these on entry and returns -1 if the checks fail. The checks should be:

• The address equals the base.
• The bounds are at least the compiler expected size.
• The base is greater than the address of csp[1] on entry.
• The permissions are at least the set listed above. This includes a check for store-local, which guarantees that this is part of our stack.

The sequence is likely to look something like:

  # Check that address equals base
  cgetaddr t1, ca0
  cgetbase t2, ca0
  bne      t1, t2, fail
  # Check that base is greater than or equal to stack pointer on entry
  blt      t2, sp, fail
  cgetlen  t1, ca0
  li       t2, {expected size}
  blt      t1, t2, fail
  cgetperm t1, ca0
  li       t2, 0x7a
  bne      t1, t2, fail
  # Rest of prolog
  ...


fail:
  li a0, -1
  li a1, -1
  cret

This is quite a long sequence (and will be effectively double this length for functions that both return structures and take on-stack arguments), so we might consider outlining it if it proves to increase code size.

[1] This is a bit fun because compartment entry points may be called from within the compartment and so we don't want to check that it's out of range of CSP.

Constraints on sealed sizes can be reduced

In the RISC-V standardisation call, @jrtc27 pointed out that we can avoid complexity for the sealing arbitrarily large objects by placing the padding before the header such that the layout is:

• Padding
• Otype header
• Unsealed object
• Padding (within the unsealed object)

The bounds of the sealed object would then be the full allocation. The address would be the start of the object header. The allocate would ensure that the range from the header to the end is representable. The unseal would do a csetbounds with the size computed from address to top. This should keep the token_unseal simple and cost only a small amount of complexity in token_seal.

[help-wanted] Import table invalid export

While following the getting started guide, I get the following errors after the xmake command:

[ 98%]: linking firmware build/cheriot/cheriot/release/test-suite
error: ld.lld: error: Import table for compartment 'scheduler' refers to address 0x8001C890, which is not a valid export
ld.lld: error: Import table for compartment 'scheduler' refers to address 0x8001C934, which is not a valid export
ld.lld: error: Import table for compartment 'scheduler' refers to address 0x8001C8E0, which is not a valid export
ld.lld: error: Import table for compartment 'scheduler' refers to address 0x8001C8DC, which is not a valid export
ld.lld: error: Import table for compartment 'scheduler' refers to address 0x8001C978, which is not a valid export
ld.lld: error: Import table for compartment 'allocator' refers to address 0x8001C85C, which is not a valid export
ld.lld: error: Import table for compartment 'allocator' refers to address 0x8001C8D8, which is not a valid export
ld.lld: error: Import table for compartment 'allocator' refers to address 0x8001C8D4, which is not a valid export
ld.lld: error: Import table for compartment 'allocator' refers to address 0x8001C934, which is not a valid export
ld.lld: error: Import table for compartment 'allocator' refers to address 0x8001C938, which is not a valid export
ld.lld: error: Import table for compartment 'allocator' refers to address 0x8001C8E0, which is not a valid export
ld.lld: error: Import table for compartment 'allocator' refers to address 0x8001C8DC, which is not a valid export
ld.lld: error: Import table for compartment 'string' refers to address 0x8001C9A4, which is not a valid export
ld.lld: error: Import table for compartment 'string' refers to address 0x8001C9A0, which is not a valid export
ld.lld: error: Import table for compartment 'allocator_test' refers to address 0x8001C88C, which is not a valid export
ld.lld: error: Import table for compartment 'allocator_test' refers to address 0x8001C954, which is not a valid export
ld.lld: error: Import table for compartment 'allocator_test' refers to address 0x8001C958, which is not a valid export
ld.lld: error: Import table for compartment 'allocator_test' refers to address 0x8001C8D8, which is not a valid export
ld.lld: error: Import table for compartment 'allocator_test' refers to address 0x8001C980, which is not a valid export
ld.lld: error: Import table for compartment 'allocator_test' refers to address 0x8001C978, which is not a valid export
ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)

I've build the ld.lld using the building dependencies instructions. And here's the version I'm running:

$ ld.lld --version
LLD 13.0.0 (https://github.com/CTSRD-CHERI/llvm-project.git a6c9dc0a4232ed4a64acc6ba4fc7f2754de1ed45) (compatible with GNU linkers)

Do you have any tips on how to proceed?

The compartment switcher holds interrupts disabled for too long

For large stacks, the compartment switcher can spend over 1,000 cycles with interrupts disabled zeroing the stack. It should do this only if both the caller and callee have interrupts disabled. In all other cases, the majority of the switcher should run with interrupts disabled.

Miscompilation of callback reference

The following test file produces invalid assembly with the current compiler:

__attribute__((cheri_compartment("other"))) void cross_compartment_call(
  __attribute__((cheri_ccallback)) void (*fn)());

__attribute__((cheri_ccallback))  void callback() {}

void test()
{
	cross_compartment_call(&callback);
	cross_compartment_call(&callback);
}

Specifically when compiled with the following command:

/cheriot-tools/bin/clang++ -c -std=c++20 -Qunused-arguments -target riscv32-unknown-unknown -mcpu=cheriot -mabi=cheriot -mxcheri-rvc -mrelax -fshort-wchar -nostdinc -Oz -g -fomit-frame-pointer -fno-builtin -fno-exceptions -fno-asynchronous-unwind-tables -fno-rtti -Werror -cheri-compartment=me -o test.o test.cc

We get the following objdump:

...
00000004 <_Z4testv>:
       4: 7d 71        	cincoffset	csp, csp, -16
       6: 06 e4        	csc	cra, 8(csp)
       8: 22 e0        	csc	cs0, 0(csp)

0000000a <.LBB1_1>:
       a: 17 04 00 00  	auipcc	cs0, 0
       e: 5b 14 04 00  	cincoffset	cs0, cs0, 0
      12: 03 33 04 00  	clc	ct1, 0(cs0)

00000016 <.LBB1_2>:
      16: 17 05 00 00  	auipcc	ca0, 0
      1a: 5b 15 05 00  	cincoffset	ca0, ca0, 0

0000001e <.LBB1_3>:
      1e: 97 03 00 00  	auipcc	ct2, 0
      22: 83 b3 03 00  	clc	ct2, 0(ct2)
      26: 82 93        	cjalr	ct2

00000028 <.LBB1_4>:
      28: 17 05 00 00  	auipcc	ca0, 0
      2c: 5b 15 05 00  	cincoffset	ca0, ca0, 0
      30: 08 61        	clc	ca0, 0(ca0)
      32: 03 33 04 00  	clc	ct1, 0(cs0)

00000036 <.LBB1_5>:
      36: 97 03 00 00  	auipcc	ct2, 0
      3a: 83 b3 03 00  	clc	ct2, 0(ct2)
      3e: 82 93        	cjalr	ct2
      40: 02 64        	clc	cs0, 0(csp)
      42: a2 60        	clc	cra, 8(csp)
      44: 41 61        	cincoffset	csp, csp, 16
      46: 82 80        	cret

Note that initialising ca0 for the first cross compartment call (0x16 .. 0x1a) uses a different sequence than for the second call (0x28 .. 0x30).
The first one is broken as it is passing an unsealed capability to the callback instead of an import table entry as the second one does.

Document and enforce stack size limits

Stacks larger than about 8 KiB are problematic for compartment switches because the switcher cannot guarantee that the remainder stack has a representable length.
In particular this csetboundsexact might trap.
Given that stack bounds must be 16-byte aligned and the capability precision is 9 bits stacks up to 16 * 511 = 8176 bytes should be fine.

We should document this limit and possibly enforce it in the xmake and / or linker.

Recursive mutex needs tests

I forgot to write them when I wrote the code (and the code was wrong). Should be easy to do if someone wants a good first issue!

No firmware works with Ibex on Arty A7 100T

Hello,
I've build the Ibex safe version for the Arty A7 100T board at 33MHz.

I've a correct result on /dev/ttyUSB1:

Ready to load firmware, hold BTN0 to ignore UART input.

I've build with the devcontainer the test-suite and the examples and no one works.

For the 01.hello_world, I've configured it :

xmake config --sdk=/cheriot-tools/ --board=ibex-arty-a7-100

Then, I've converted the result to a firmware with ibex-build-firmware.sh and send it to the serial port :

Ready to load firmware, hold BTN0 to ignore UART input.
Starting loading.  First word was: 200406B7
.......................................
Finished loading.  Last word was: 020001F4
Number of words loaded to IRAM: 000026A1
Loaded firmware, jumping to IRAM.

The cpu0_iram.vhx file contains :

200406B7
31068693
40000593
....
0300002C
0300015E
03000176
020001F4

which is correct but I get no output.

I've tried the different examples with no result.

Perhaps the memory map has changed ?

OpenOCD and gdb are not available in the devcontainer, so I can not try to trace the execution from the CPU point of view.

Any help would be appreciated.

Large globals sections are incorrectly set up

CHERIoT-Platform/llvm-project#26 reports that very large objects are not working.

This doesn't appear to be a compiler bug. Extending the test case from that example, we get a capability with the correct bounds, but cgp for the compartment appears to be being created with a zero length and so deriving a capability from that gives an untagged capability. This is presumably therefore a loader bug.

Question about std::string

This probably shows the depth of my ignorance about the compiler system, and happy to move it to a discussion group instead if that's more appropriate.

I was starting work on an example of how we might do dynamic configuration of compartments, and hit what I'm sure is noob issue with std:;string

If I change the examples/01.hello_world/hello.cc to try an use std::string

// Copyright Microsoft and CHERIoT Contributors.
// SPDX-License-Identifier: MIT

#include <compartment.h>
#include <debug.hh>
#include <fail-simulator-on-error.h>
#include <string>

/// Expose debugging features unconditionally for this compartment.
using Debug = ConditionalDebug<true, "Hello world compartment">;

/// Thread entry point.
void __cheri_compartment("hello") say_hello()
{
	// Print hello world, along with the compartment's name to the default UART.
	std::string msg = "Hello world";
	Debug::log("{}", msg);
}

xmake gives:

error: /workspaces/cheriot-rtos/sdk/include/debug.hh:388:4: error: implicit instantiation of undefined template 'DebugFormatArgumentAdaptor<std::string>'
          DebugFormatArgumentAdaptor<
...

If I remove the call to Debug so I just have the definition

    std::string msg = "Hello world";

xmake fails with:

error: ld.lld: error: undefined symbol: strlen
>>> referenced by hello.cc:14
>>>               build/cheriot/cheriot/release/hello.compartment:()

compile_commands.json includes the following - that I assume should be picking up strlen ?

{
  "directory": "/workspaces/cheriot-rtos/examples/01.hello_world",
  "arguments": ["/cheriot-tools/bin/clang", "-c", "-std=c2x", "-Qunused-arguments", "-target", "riscv32-unknown-unknown", "-mcpu=cheriot", "-mabi=cheriot", "-mxcheri-rvc", "-mrelax", "-fshort-wchar", "-nostdinc", "-Oz", "-g", "-ffunction-sections", "-fdata-sections", "-fomit-frame-pointer", "-fno-builtin", "-fno-exceptions", "-fno-asynchronous-unwind-tables", "-fno-c++-static-destructors", "-fno-rtti", "-Werror", "-I/workspaces/cheriot-rtos/sdk/include/c++-config", "-I/workspaces/cheriot-rtos/sdk/include/libc++", "-I/workspaces/cheriot-rtos/sdk/include", "-fvisibility=hidden", "-DNDEBUG", "-o", "build/.objs/string/cheriot/cheriot/release/__/__/sdk/lib/string/strlen.c.o", "../../sdk/lib/string/strlen.c"],
  "file": "../../sdk/lib/string/strlen.c"
},

Feels like I'm missing something obvious about using std::string ?

A few typos in the examples READMEs

Spotted a few issues while working through the examples - will be happy to crate a PR for them but assumed you want an issue first:

1. /three/the/
   

   cheriot-rtos/examples/04.temporal_safety/README.md

   Line 18 in 5b6d221

   The only field that changes after three call to `free` is the *tag* or *valid* bit.

2. A few bits of duplicted text
   https://github.com/microsoft/cheriot-rtos/blob/5b6d2217586aced18b56a71186c98178be1843f9/examples/08.memory_safety/README.md?plain=1#L12C1-L14C1

Add std:string overload to debug

Would be nice to support hazard pointer like claims on heap objects

We have a notion of claims of heap objects, but they're fairly heavy, requiring a full compartment call to either acquire or drop. We'd like a lighter-weight option for a small number of objects per thread: hazard pointers understood by the heap.

Hazard pointers originate in the world of lock/wait-free data structures, where pointers have a four-stage lifecycle: unpublished, public, withdrawn, and freed. Hazard pointers mediate the transition from withdrawn to freed, deferring it until there are no more hazardous, local copies of once-public pointers. Because everyone involved is presumed cooperative, hazard lists need only be scanned once: a request to free memory must follow that pointer being withdrawn from the shared state, and so it cannot become newly hazardous.

The story for potentially uncooperative threads is slightly different, as it may be impossible to ensure that a pointer is completely withdrawn from the shared state (and is not cached somewhere without being declared hazardous) when it comes time to free() it. This is, after all, why we have sweeping revocation! As such, our story needs to be a little different. In particular, we need a ratchet mechanism to advance an object through these states:

1. allocated and hazard-able (and possibly hazardous)
2. allocated but no longer hazard-able, with N hazards registered
3. ...
4. allocated but no longer hazard-able, with 1 hazard registered
5. No longer hazardous, and so free (well, quarantined)

Additionally, hazards are intended to be ephemeral. As such, we do not expect them to survive cross-compartment calls. (And anyway, a callee cannot assume that its caller has declared any hazards appropriately, and so must do so itself; if we ensured that hazards persisted across calls, there would be redundant registrations.)

Here is a design sketch. This is going to need to be a fair bit of carefully crafted assembler but I believe it's entirely possible.

There is a Hazard Registry maintained on behalf of the allocator by a privileged library. A Hazard Registry Entry (or just "Hazard") is an optional pair of a capability to a (sub)object and a taint flag. The None branch of the option is indicated by a null cap. Hazard entries are, we imagine, stored in static sealed objects, with handle type SealedHeapHazard, each of which has room for some number of hazards. To avoid linked-list traversal, we imagine that these are collected in a dedicated section, forming a "linker set".

The privileged library exposes a single call to clients (and more to the allocator itself):
void *heap_hazard_register(SealedHeapHazards *hh, size_t ix, void *p), with interrupts deferred,

• unseals hh,
• releases the existing hazard in the ix-th entry in hh, if any, and
• scans all existing hazards, looking for a tainted entry that precludes registering p (see below), and
• if p is tagged, installs p into that entry and and returns p.
  If p is untagged or has a tainted hazard, this call returns null. The registry entries are write-only from the clients' perspective: it is not possible to retrieve a hazard pointer.
   
  Releasing an untainted hazard is as simple as replacing it with null. Releasing a tainted hazard triggers a full, interrupt-enabled cross-compartment call to the allocator to advance the effort to free the associated object. Importantly, the hazard remains tainted while this call is in progress so that the taint is seen by other calls to heap_hazard_register. Specifically, this advance involves scanning the set of hazards, and either
• finding no relevant hazards; proceed to quarantine the object
• propagating taint: taint some other, relevant hazard, upgrading the pointer in that entry to be the full object
  before replacing the hazard with null.

Upon discovering that there are no more claims to an object, the allocator will effectively register this object as a tainted hazard and then immediately release that hazard (that is, scan the hazard table looking for another entry to taint). One could imagine a dedicated SealedHeapHazard chunk of the registry specifically for free()'s use, but it might make more sense to special case this primordial tainting.

Tainted hazards are always carrying pointers full heap objects, while untainted ones may have sub-object pointers; this simplifies the scan above without unduly challenging the initial sweep for hazards.

Commentary welcome.

Create sub-quotas for allocation capabilities

The allocator currently deals with static heap quotas. Sometimes it's useful to allow another compartment to allocate memory on your behalf, but not to consume your entire quota. We should allow a mechanism for creating a new quota object that contains part of an existing quota. This should have the same ID as the original, so that objects can be freed in both.

The main complication in this is that, currently, quotas are checked without the allocator lock being held (I think). We need to make sure that the mechanism is robust in the presence of a quota being deallocated during an allocation or deallocation operation.

Static sealing types parameterized by template parameters?

It might be nice to replace

cheriot-rtos/sdk/include/thread_pool.h

Lines 70 to 79 in b4b6204

	/**
	* Helper that generates a different sealing key per type using the
	* allocator's token mechanism.
	*/
	template<typename T>
	inline SKey sealing_key_for_type()
	{
	static SKey key = token_key_new();
	return key;
	}

with something that didn't incur the dynamic costs. The static mechanism is tantalizingly close, and yet.

Using the existing macros is a non-starter, due to phasing: the preprocessor would need to have the (stringified) type being substituted for T, but it's operating before any of that nonsense has happened.

I've tried, to no success, slight tweaks on

template<typename T>
static constexpr loader::ExportEntry sealing_key_for_type_ee
  __attribute__((section("compartment_exports"))) = {
    .functionStart    = 0,
    .minimumStackSize = 0,
    .flags            = loader::ExportEntry::SealingTypeEntry};

template<typename T>
static constexpr loader::ImportEntry sealing_key_for_type_ie
  __attribute__((section("compartment_imports"))) = {
    .boot = {.address = &sealing_key_for_type_ee<T>, .size = 0}};

/**
 * Helper that generates a different sealing key per type using the
 * allocator's token mechanism.
 */
template<typename T>
inline SKey sealing_key_for_type()
{
	return static_cast<SKey>(sealing_key_for_type_ie<T>.pointer);
}

but, of course, that doesn't quite work either: I can't initialize the ptraddr_t .boot.address (having given a name to the anonymous struct in union ImportEntry) with a pointer type, and I can't reinterpret_cast<ptraddr_t>(&sealing_key_for_type_ee<T>) either, because that's not allowed as a constexpr.

Better protect against stack-overflow attacks

Threads have a single stack that gets subdivided as they cross compartment boundaries.

This means that a compartment controls how much stack space another compartment has before calling it.

If this stack space is not checked at the compartment entry point, this creates an exploitable attack vector.

Example: if the total stack size if 4KB, a compartment A can maliciously consume exactly 3.96KB of stack to trigger a stack overflow at a very precise point in compartment B to corrupt its state and mount subsequent attacks.

This is a real problem for the TCB, e.g., the allocator or the scheduler.

Currently, we have a mechanism to thwart such attacks: the export table entry contains the stack required for the first function that's called. This number is easy to compute and makes sense as a bare minimum.

However, this number will remain vastly insufficient for most real-world cases, because it does not account for any other functions that first entry point may call.

Looking forward, we should:

1. Make it possible to specify arbitrary minimum stack requirements at an entry point granularity through a function attribute. Computing the worst case would be tricky, and potentially undesirable as it is effectively a policy that we do not want to hardcode in the compiler.

2. Go through TCB entry points (scheduler, allocator), and experimentally determine a safe value for this new attribute.

3. Discuss this issue in the book (and how to solve it with the attribute!)

MMIO regions are artificially constrained to 2^24 bytes

Specify a large mmio region, e.g. in the board spec:

        "plic": {
            "start"  : 0x60000000,
            "length" :  0x8000000
        },

The result generates an import table entry for the builtin plic support:

        {
          "kind": "MMIO",
          "length": 134217728,
          "permits_load": true,
          "permits_load_mutable": false,
          "permits_load_store_capabilities": false,
          "permits_store": true,
          "start": 1610612736
        },

But the loader reports:

loader: Import table: 0x80001e70, 0x88 bytes
loader: Building mmio capability 0x10000000 + 0x100 (permissions: G RW---- -- ---)
loader: Building mmio capability 0x2000000 + 0x10000 (permissions: G RW---- -- ---)
loader: Skipping sealing type import
loader: Building mmio capability 0x60000000 + 0x0 (permissions: G RW---- -- ---)

which results in an invalid cap/ptr being constructed for plicPrios & co in StandardPlic::StandardPlic.

Looks like this is due to:

      /**
       * The size and permissions.  The size is currently used for static
       * sealed objects and MMIO imports.  The permissions part is used
       * only for MMIO regions.
       *
       * The size is stored in the low 24 bits.  We cannot provide
       * fine-grained capabilities beyond that size, leaving the top 8
       * bits free.  These encode a subset of permissions.  MMIO regions
       * cannot be local or executable.
       */
      size_t sizeAndPermissions;

but I don't see where the data are written to identify why the length/size reads back as zero (I assume they are read directly on startup).

Tag violation when creating a std::string from string literal over 14 characters in length

Hit this when working on #255

If in test_runner.cc lines 111-112

        const std::string S = "I am a walrus"s;
	debug_log("Trying to print std::string: {}", S);

are changed to

        const std::string S = "I am the walrus"s;
	debug_log("Trying to print std::string: {}", S);

it results in

Test runner: Trying to print std::string: Test runner: mcause: 0x1c, pcc: 0x80006e30 (v:0 0x80006d90-0x800075e8 l:0x858 o:0x0 p: G R-cgm- X- ---)
Test runner: Error TagViolation(0x2) in register CA1(0xb)
Test runner: Current test crashed

I tried to create a simple reproducer based on examples/01.hello_world, but it looks like I'm hitting another error in the error handler as

#include <compartment.h>
#include <debug.hh>
#include <string>

/// Expose debugging features unconditionally for this compartment.
using Debug = ConditionalDebug<true, "Hello world compartment">;
using namespace std::string_literals;

extern "C" ErrorRecoveryBehaviour
compartment_error_handler(ErrorState *frame, size_t mcause, size_t mtval)
{
	Debug::log("--- Opps ---");	
	//Debug::log("mcause: {}, pcc: {}", mcause, frame->pcc);
	auto [reg, cause] = CHERI::extract_cheri_mtval(mtval);
	//Debug::log("Error {} in register {}", reg, cause);
	return ErrorRecoveryBehaviour::ForceUnwind;
}

/// Thread entry point.
void __cheri_compartment("hello") say_hello()
{
	Debug::log("Hello world");
	const std::string S1 = "I am an walrus"s;
	Debug::log("Hello again");
	const std::string S2 = "I am the walrus"s;
	Debug::log("Hello once more");
}

gives

Hello world compartment: Hello world
Hello world compartment: Hello again
Hello world compartment: --- Opps ---
SUCCESS

but uncommenting the line to print mcause gives

Hello world compartment: Hello world
Hello world compartment: Hello again
Hello world compartment: SUCCESS

Should interrupts be enabled when the scheduler yields?

As of the merge of #44 e24f37f disabled interrupts when yielding in the scheduler.
It's not clear whether the comment on yield_interrupt_enabled which describes the need for enabling interrupts still applies.
A simple test of a thread calling thread_sleep with interrupts disabled worked as expected: the thread was suspended, yielded and the idle thread was scheduled allowing timer interrupts to be received.

We should think about this and, at the very least, fix the comment.

Move message queues out of the scheduler

Message queues should be implemented with a ring buffer that uses producer / consumer counters and a buffer. Each endpoint can be represented as

• Write end: A write-only capability to the buffer, a read-only capability to the consumer counter, a read-write capability to the producer counter.
• Read end: A read-only capability to the buffer, a read-only capability to the producer counter, a read-write capability to the consumer counter.

We can then implement the create, push, and pop operations as library routines. Inserting into the buffer happens in the sender's context and so we don't have to care about any attacks from invalid source capabilities. Similarly, popping happens in the receiver's context. The producer and consumer counters can be used as futexes and so there's no scheduler state needed.

This removes a load of code from the scheduler (and therefore from the TCB for availability). It will require an API (likely a static inline) for adding waiters to multiwaiter objects.

heap_free_all frees sealed objects

The heap_free_all function frees all objects allocated with an allocation capability. This means that it can be used to attack compartments that have allocated memory for a given caller.

It shouldn't, and we should provide a heap_free_all_sealed that frees everything sealed with a particular type that can be used for cleanup.

This is somewhat complicated by the fact that we do want to drop claims on sealed objects, we just don't want to allow them to actually be freed. @nwf, what do you think the right behaviour is?

Heaps are insufficiently aligned on systems with > 256 KiB of RAM

The heap start is 512-byte aligned. This is fine for heaps up to 256 KiB, but above that causes problems. The loader should treat the heap import region specially and align the start correctly for the available size.