ctsrd-cheri / cheribsd

FreeBSD adapted for CHERI-RISC-V and Arm Morello.

CheriBSD

CheriBSD extends FreeBSD to implement memory protection and software compartmentalization features supported by the CHERI ISA. CheriBSD includes support for CHERI extensions to the Armv8-A (via Morello) and RISC-V architectures. To build and run CheriBSD we recommend using the cheribuild script.

For information on our branching model, updates, and flag days, please read CHERI-UPDATING.md.

The CheriBSD web page can be found here: http://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheribsd.html

The Qemu-CHERI web page may also be useful: http://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri-qemu.html

More information about CHERI can be found on http://cheri-cpu.org and in the following Technical Reports:

An Introduction to CHERI https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-941.pdf

Capability Hardware Enhanced RISC Instructions: CHERI Instruction-Set Architecture https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-927.pdf

CHERI C/C++ Programming Guide https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-947.pdf

FreeBSD Source:

This is the top level of the FreeBSD source directory.

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms. A large community has continually developed it for more than thirty years. Its advanced networking, security, and storage features have made FreeBSD the platform of choice for many of the busiest web sites and most pervasive embedded networking and storage devices.

For copyright information, please see the file COPYRIGHT in this directory. Additional copyright information also exists for some sources in this tree - please see the specific source directories for more information.

The Makefile in this directory supports a number of targets for building components (or all) of the FreeBSD source tree. See build(7), config(8), FreeBSD handbook on building userland, and Handbook for kernels for more information, including setting make(1) variables.

For information on the CPU architectures and platforms supported by FreeBSD, see the FreeBSD website's Platforms page.

Source Roadmap:

Directory	Description
bin	System/user commands.
cddl	Various commands and libraries under the Common Development and Distribution License.
contrib	Packages contributed by 3rd parties.
crypto	Cryptography stuff (see crypto/README).
etc	Template files for /etc.
gnu	Commands and libraries under the GNU General Public License (GPL) or Lesser General Public License (LGPL). Please see gnu/COPYING and gnu/COPYING.LIB for more information.
include	System include files.
kerberos5	Kerberos5 (Heimdal) package.
lib	System libraries.
libexec	System daemons.
release	Release building Makefile & associated tools.
rescue	Build system for statically linked /rescue utilities.
sbin	System commands.
secure	Cryptographic libraries and commands.
share	Shared resources.
stand	Boot loader sources.
sys	Kernel sources (see sys/README.md).
targets	Support for experimental DIRDEPS_BUILD
tests	Regression tests which can be run by Kyua. See tests/README for additional information.
tools	Utilities for regression testing and miscellaneous tasks.
usr.bin	User commands.
usr.sbin	System administration commands.

For information on synchronizing your source tree with one or more of the FreeBSD Project's development branches, please see FreeBSD Handbook.

cheribsd's Issues

split malloc out of libc

The malloc implementation is currently strongly tied to libc and the malloc implementation assumes access to memmap (and sbrk). Within a sandbox this doesn't make sense. We either want the current model of a simple malloc with a fixed heap, a one-shot malloc (also with a fixed heap), or a malloc accessed via the system class to support garbage collection. Decoupling malloc from libc will allow the standard libc to be used and an appropriate malloc to be selected at compile time.

CheriABI - userspace support - static binaries

This issue is with regards to being able to compile statically linked binaries against CheriABI -- a capability-aware system-call ABI -- such that entire userspace applications can be capability-based, rather than just sandboxed enclaves.

It will depend on #15, kernel support for CheriABI.

strcmp crashes in cheriabi

char str[] = "123456789";
char str2[] = "123456789";
strcmp(str, str2);

This crashes. My initial thought was that it was caused by the can-always-read-last-aligned-word assumption, but the fault is a read fault at 0x3, so not actually a capability address. This may be a compiler bug...

Off-by-one error in sandbox ABI strlen

char str[] = "123456789";
printf("%d\n", strlen(str));

Prints 10 in the sandbox ABI, but should print 9 (and does normally).

/bin/df experiences alignment error with CheriABI

When running a CheriABI version of /bin/df, I experience bus errors:

> ./df
Filesystem          1K-blocks    Used    Avail Capacity  Mounted on
Bus error

Compiled with the following modifications:

--- a/bin/df/Makefile
+++ b/bin/df/Makefile
@@ -4,6 +4,9 @@
 MOUNT= ${.CURDIR}/../../sbin/mount
 .PATH: ${MOUNT}

+WANT_CHERI=pure
+WANT_DUMP=yes
+
 PROG=  df
 SRCS=  df.c vfslist.c

Console output reports an alignment error:

FreeBSD/mips (beri1) (ttyj0)

login: cpuid = 0
C00: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
C01: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:12009eca0 t:0
C02: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
C03: v:1 s:0 p:7fff81fd b:000000012009ede0 l:0000000000000900 o:48 t:0
C04: v:1 s:0 p:7fff81fd b:0000007fffffb7a0 l:0000000000000260 o:20 t:0
C05: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120085bc0 t:0
C06: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:12006bda0 t:0
C07: v:1 s:0 p:7fff81fd b:0000007fffffbbdc l:0000000000000004 o:0 t:0
C08: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7fffffa9c8 t:0
C09: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7fffffa9f8 t:0
C10: v:1 s:0 p:7fff81fd b:0000007fffffc2e4 l:0000000000000004 o:0 t:0
C11: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
C12: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120052d78 t:0
C13: v:1 s:0 p:000081d5 b:0000007fffffd320 l:0000000000000008 o:0 t:0
C14: v:1 s:0 p:7fff81fd b:0000007fffffc380 l:0000000000000020 o:0 t:0
C15: v:1 s:0 p:7fff81fd b:0000007fffffc360 l:0000000000000020 o:0 t:0
C16: v:1 s:0 p:7fff81fd b:0000007fffffc1e0 l:0000000000000020 o:0 t:0
C17: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:12004aec8 t:0
C18: v:1 s:0 p:7fff81fd b:000000012009ede0 l:0000000000000900 o:48 t:0
C19: v:1 s:0 p:7fff81fd b:0000007fffffbba0 l:0000000000000020 o:0 t:0
C20: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120099870 t:0
C21: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120085bc0 t:0
C22: v:1 s:0 p:7fff81ff b:00000001600ef000 l:0000000000004000 o:25 t:0
C23: v:1 s:0 p:7fff81ff b:00000001600ef000 l:0000000000004000 o:25 t:0
C24: v:1 s:0 p:7fff81fd b:0000007fffffbb90 l:0000000000000004 o:0 t:0
C26: v:1 s:0 p:7fff81ff b:0000000000000000 l:0000010000000000 o:0 t:0
C31: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120052ea0 t:0
cpuid = 0
C00: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
C01: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:12009eca0 t:0
C02: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
C03: v:1 s:0 p:7fff81fd b:000000012009ede0 l:0000000000000900 o:48 t:0
C04: v:1 s:0 p:7fff81fd b:0000007fffffb7a0 l:0000000000000260 o:20 t:0
C05: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120085bc0 t:0
C06: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:12006bda0 t:0
C07: v:1 s:0 p:7fff81fd b:0000007fffffbbdc l:0000000000000004 o:0 t:0
C08: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7fffffa9c8 t:0
C09: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7fffffa9f8 t:0
C10: v:1 s:0 p:7fff81fd b:0000007fffffc2e4 l:0000000000000004 o:0 t:0
C11: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
C12: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120052d78 t:0
C13: v:1 s:0 p:000081d5 b:0000007fffffd320 l:0000000000000008 o:0 t:0
C14: v:1 s:0 p:7fff81fd b:0000007fffffc380 l:0000000000000020 o:0 t:0
C15: v:1 s:0 p:7fff81fd b:0000007fffffc360 l:0000000000000020 o:0 t:0
C16: v:1 s:0 p:7fff81fd b:0000007fffffc1e0 l:0000000000000020 o:0 t:0
C17: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:12004aec8 t:0
C18: v:1 s:0 p:7fff81fd b:000000012009ede0 l:0000000000000900 o:48 t:0
C19: v:1 s:0 p:7fff81fd b:0000007fffffbba0 l:0000000000000020 o:0 t:0
C20: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120099870 t:0
C21: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120085bc0 t:0
C22: v:1 s:0 p:7fff81ff b:00000001600ef000 l:0000000000004000 o:25 t:0
C23: v:1 s:0 p:7fff81ff b:00000001600ef000 l:0000000000004000 o:25 t:0
C24: v:1 s:0 p:7fff81fd b:0000007fffffbb90 l:0000000000000004 o:0 t:0
C26: v:1 s:0 p:7fff81ff b:0000000000000000 l:0000010000000000 o:0 t:0
C31: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120052ea0 t:0
Mar 14 17:55:45 beri1 kernel: ALIGNMENT_FIX_ERR: pid 967 tid 100045 (df), uid 999: pc 0x120052ea0 got a write fault (type 0x5) at 0x12009ee28
Mar 14 17:55:45 beri1 kernel: Trapframe Register Dump:
Mar 14 17:55:45 beri1 kernel: zero: 0   at: 0x12009eca0 v0: 0   v1: 0x1200a2d78
Mar 14 17:55:45 beri1 kernel: a0: 0x7fffffaaf0  a1: 0x1 a2: 0x32        a3: 0x7fffffad1c
Mar 14 17:55:45 beri1 kernel: a4: 0x8   a5: 0   a6: 0x7fffffa980        a6: 0x7fffffa8d0
Mar 14 17:55:45 beri1 kernel: t0: 0x13  t1: 0xc t2: 0x7fffffb640        t3: 0x7fffffb5c0
Mar 14 17:55:45 beri1 kernel: t8: 0x1   t9: 0x120052d78 s0: 0x12009fa90 s1: 0
Mar 14 17:55:45 beri1 kernel: s2: 0x1   s3: 0x120070000 s4: 0x5 s5: 0x7fffffb4e0
Mar 14 17:55:45 beri1 kernel: s6: 0x120040000   s7: 0x8 k0: 0   k1: 0
Mar 14 17:55:45 beri1 kernel: gp: 0x12009fa90   sp: 0x7fffffaa20        s8: 0x7fffffaa20       ra: 0x7fffffb4c0
Mar 14 17:55:45 beri1 kernel: sr: 0x4000fcf3    mullo: 0x2af714a1c0000000       mulhi: 0       badvaddr: 0x12009ee28
Mar 14 17:55:45 beri1 kernel: cause: 0x114      pc: 0x120052ea0
Mar 14 17:55:45 beri1 kernel: Page table info for bad address 0x12009ee28: pde = 0x98000000044a4000, pte = 0x8000000014959e

The code at $pcc = 0x120052ea0 is as follows:

0000000120052d78 <__Bfree_D2A>:
   120052d78:   67bdff20        daddiu  sp,sp,-224
   120052d7c:   67a400d8        daddiu  a0,sp,216
   120052d80:   ebcb2003        csd     s8,a0,0(c11)
   120052d84:   67a400d0        daddiu  a0,sp,208
   120052d88:   ea0b2003        csd     s0,a0,0(c11)
   120052d8c:   faabe8a0        csc     c21,sp,160(c11)
   120052d90:   fa8be880        csc     c20,sp,128(c11)
   120052d94:   fa6be860        csc     c19,sp,96(c11)
   120052d98:   fa4be840        csc     c18,sp,64(c11)
   120052d9c:   fa2be820        csc     c17,sp,32(c11)
   120052da0:   03a0f02d        move    s8,sp
   120052da4:   49b21800        cmove   c18,c3
   120052da8:   49b96002        cgetoffset      t9,c12
   120052dac:   3c010005        lui     at,0x5
   120052db0:   0039182d        daddu   v1,at,t9
   120052db4:   49819000        ctoptr  at,c18,c0
   120052db8:   1020003b        beqz    at,120052ea8 <__Bfree_D2A+0x130>
   120052dbc:   00000000        nop
   120052dc0:   c8520106        clw     v0,zero,32(c18)
   120052dc4:   2841000a        slti    at,v0,10
   120052dc8:   1420000a        bnez    at,120052df4 <__Bfree_D2A+0x7c>
   120052dcc:   6470cd18        daddiu  s0,v1,-13032
   120052dd0:   660180f0        daddiu  at,s0,-32528
   120052dd4:   48810047        cfromptr        c1,c0,at
   120052dd8:   c8210003        cld     at,zero,0(c1)
   120052ddc:   48000807        cgetpcc c1
   120052de0:   49ac0841        csetoffset      c12,c1,at
   120052de4:   48f16000        cjalr   c17,c12
   120052de8:   49a39000        cmove   c3,c18
   120052dec:   08014baa        j       120052ea8 <__Bfree_D2A+0x130>
   120052df0:   00000000        nop
   120052df4:   66018938        daddiu  at,s0,-30408
   120052df8:   48810047        cfromptr        c1,c0,at
   120052dfc:   c8210003        cld     at,zero,0(c1)
   120052e00:   48940047        cfromptr        c20,c0,at
   120052e04:   c8340006        clw     at,zero,0(c20)
   120052e08:   1020001f        beqz    at,120052e88 <__Bfree_D2A+0x110>
   120052e0c:   00000000        nop
   120052e10:   66019638        daddiu  at,s0,-27080
   120052e14:   48810047        cfromptr        c1,c0,at
   120052e18:   c8210003        cld     at,zero,0(c1)
   120052e1c:   66029658        daddiu  v0,s0,-27048
   120052e20:   48810087        cfromptr        c1,c0,v0
   120052e24:   c8410003        cld     v0,zero,0(c1)
   120052e28:   4800a807        cgetpcc c21
   120052e2c:   49aca841        csetoffset      c12,c21,at
   120052e30:   48810087        cfromptr        c1,c0,v0
   120052e34:   49b30800        cmove   c19,c1
   120052e38:   48f16000        cjalr   c17,c12
   120052e3c:   49a39800        cmove   c3,c19
   120052e40:   c8320102        clwu    at,zero,32(c18)
   120052e44:   de028030        ld      v0,-32720(s0)
   120052e48:   00010978        dsll    at,at,0x5
   120052e4c:   6442eca0        daddiu  v0,v0,-4960
   120052e50:   48810087        cfromptr        c1,c0,v0
   120052e54:   d8410800        clc     c2,at,0(c1)
   120052e58:   c8540006        clw     v0,zero,0(c20)
   120052e5c:   f8520000        csc     c2,zero,0(c18)
   120052e60:   10400011        beqz    v0,120052ea8 <__Bfree_D2A+0x130>
   120052e64:   fa410800        csc     c18,at,0(c1)
   120052e68:   660191b0        daddiu  at,s0,-28240
   120052e6c:   48810047        cfromptr        c1,c0,at
   120052e70:   c8210003        cld     at,zero,0(c1)
   120052e74:   49aca841        csetoffset      c12,c21,at
   120052e78:   48f16000        cjalr   c17,c12
   120052e7c:   49a39800        cmove   c3,c19
   120052e80:   08014baa        j       120052ea8 <__Bfree_D2A+0x130>
   120052e84:   00000000        nop
   120052e88:   de018030        ld      at,-32720(s0)
   120052e8c:   00021000        sll     v0,v0,0x0
   120052e90:   00021178        dsll    v0,v0,0x5
   120052e94:   6421eca0        daddiu  at,at,-4960
   120052e98:   48810047        cfromptr        c1,c0,at
   120052e9c:   d8411000        clc     c2,v0,0(c1)
   120052ea0:   f8520000        csc     c2,zero,0(c18)                   <--- Crash here
   120052ea4:   fa411000        csc     c18,v0,0(c1)
   120052ea8:   03c0e82d        move    sp,s8
   120052eac:   da2be820        clc     c17,sp,32(c11)
   120052eb0:   da4be840        clc     c18,sp,64(c11)
   120052eb4:   da6be860        clc     c19,sp,96(c11)
   120052eb8:   da8be880        clc     c20,sp,128(c11)
   120052ebc:   daabe8a0        clc     c21,sp,160(c11)
   120052ec0:   67a100d0        daddiu  at,sp,208
   120052ec4:   ca0b0803        cld     s0,at,0(c11)
   120052ec8:   67a100d8        daddiu  at,sp,216
   120052ecc:   cbcb0803        cld     s8,at,0(c11)
   120052ed0:   49008800        cjr     c17
   120052ed4:   67bd00e0        daddiu  sp,sp,224

The value in $c18 is indeed inadequately aligned for a CSC instruction with 0 offset:

C18: v:1 s:0 p:7fff81fd b:000000012009ede0 l:0000000000000900 o:48 t:0

Possibly a compiler bug, but another set of eyes would be appreciated.

Surprising EACCES returned by open(2) on CheriABI

Having compiled nvi to use CheriABI:

--- a/usr.bin/vi/Makefile
+++ b/usr.bin/vi/Makefile
@@ -4,6 +4,9 @@

 .include <src.opts.mk>

+WANT_CHERI=pure
+WANT_DUMP=yes
+
 SRCDIR=                ${.CURDIR}/../../contrib/nvi
 SUBDIR+=       catalog

I get a mysterious and unexpected EACCES back from its call to open a temporary file:

> ktrace ./nvi
ex/vi: Input encoding conversion not supported
ex/vi: Error: /tmp/vi.fnX81ZifUk: Permission denied
Service unavailable> 

When I inspect trace output, I see:

  1010 nvi      CALL  stat(0x16027a420,0x7fffffcd88)
  1010 nvi      NAMI  "/tmp"
  1010 nvi      STRU  struct stat {dev=106, ino=2, mode=041777, nlink=7, uid=0, gid=0, rdev=312, atime=1426476132, stime=1426481931.003911000, ctime=1426481931.003911000, birthtime=1426476132, size=1536, blksize=32768, blocks=8, flags=0x0 }
  1010 nvi      RET   stat 0
  1010 nvi      CALL  open(0x16027a420,0xa02<O_RDWR|O_CREAT|O_EXCL>,0xa<S_IXGRP|S_IWOTH>)
  1010 nvi      NAMI  "/tmp/vi.ElkbxBzqK6"
  1010 nvi      RET   open 3
  1010 nvi      CALL  fstat(0x3,0x7fffffd1a8)
  1010 nvi      STRU  struct stat {dev=106, ino=78, mode=0100010, nlink=1, uid=999, gid=0, rdev=0, atime=1426481931.008197000, stime=1426481931.008197000, ctime=1426481931.008197000, birthtime=1426481931.008167000, size=0, blksize=32768, blocks=0, flags=0x0 }
  1010 nvi      RET   fstat 0
  1010 nvi      CALL  close(0x3)
...
  1010 nvi      CALL  open(0x16027a420,0x4<O_NONBLOCK>,<unused>0x1b6)
  1010 nvi      NAMI  "/tmp/vi.ElkbxBzqK6"
  1010 nvi      RET   open -1 errno 13 Permission denied
  1010 nvi      CALL  stat(0x7fffffbdd8,0x7fffffc348)

Non-CheriABI nvi seems to work better:

  1032 nvi      CALL  stat(0x160884100,0x7fffffd0c0)
  1032 nvi      NAMI  "/tmp"
  1032 nvi      STRU  struct stat {dev=106, ino=2, mode=041777, nlink=7, uid=0, gid=0, rdev=312, atime=1426476132, stime=1426482151.223722000, ctime=1426482151.223722000, birthtime=1426476132, size=1536, blksize=32768, blocks=8, flags=0x0 }
  1032 nvi      RET   stat 0
  1032 nvi      CALL  open(0x160884100,0xa02<O_RDWR|O_CREAT|O_EXCL>,0x180<S_IRUSR|S_IWUSR>)
  1032 nvi      NAMI  "/tmp/vi.mFhBHhFxyQ"
  1032 nvi      RET   open 3
  1032 nvi      CALL  fstat(0x3,0x7fffffd6d0)
  1032 nvi      STRU  struct stat {dev=106, ino=79, mode=0100600, nlink=1, uid=999, gid=0, rdev=0, atime=1426482151.229216000, stime=1426482151.229216000, ctime=1426482151.229216000, birthtime=1426482151.229187000, size=0, blksize=32768, blocks=0, flags=0x0 }
  1032 nvi      RET   fstat 0
  1032 nvi      CALL  close(0x3)
...
  1032 nvi      CALL  open(0x160884100,0x4<O_NONBLOCK>,<unused>0x1b6)
  1032 nvi      NAMI  "/tmp/vi.mFhBHhFxyQ"
  1032 nvi      RET   open 3

This look pretty similar, but maybe there's a bug in the ABI wrapper for open(2)...?

CheriABI /usr/bin/host experiences tag violation

Compiling /usr/bin/host for CheriABI:

--- a/usr.bin/host/Makefile
+++ b/usr.bin/host/Makefile
@@ -1,5 +1,8 @@
 # $FreeBSD$

+WANT_CHERI=pure
+WANT_DUMP=yes
+
 LDNSDIR=       ${.CURDIR}/../../contrib/ldns
 LDNSHOSTDIR=   ${.CURDIR}/../../contrib/ldns-host

I experience a tag violation when I run host .:

> ./host .
Signal 34

Console output:

cpuid = 0
CHERI cause: ExcCode: 0x02 RegNum: $c03 (tag violation)
$c00: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
$c01: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
$c02: v:1 s:0 p:7fff81fd b:0000007fffffc870 l:0000000000000008 o:0 t:0
$c03: v:0 s:0 p:7fff81ff b:0000000160377060 l:0000000000000080 o:20 t:0
$c04: v:1 s:0 p:7fff81ff b:000000016037a100 l:0000000000000100 o:a0 t:0
$c05: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7fffffcd90 t:0
$c06: v:1 s:0 p:7fff81fd b:0000007fffffbd48 l:0000000000000008 o:0 t:0
$c07: v:1 s:0 p:7fff81fd b:0000007fffffbd40 l:0000000000000008 o:0 t:0
$c08: v:1 s:0 p:7fff81fd b:0000007fffffbcb0 l:0000000000000008 o:0 t:0
$c09: v:1 s:0 p:7fff81fd b:0000007fffffbca8 l:0000000000000008 o:0 t:0
$c10: v:1 s:0 p:7fff81fd b:0000007fffffbd28 l:0000000000000008 o:0 t:0
$c11: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
$c12: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:12002c508 t:0
$c13: v:1 s:0 p:7fff81ff b:000000016037a100 l:0000000000000100 o:7c t:0
$c14: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120363d50 t:0
$c15: v:1 s:0 p:7fff81fd b:0000007fffffbd08 l:0000000000000008 o:0 t:0
$c16: v:1 s:0 p:7fff81fd b:0000007fffffbd00 l:0000000000000008 o:0 t:0
$c17: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120029de4 t:0
$c18: v:1 s:0 p:7fff81fd b:0000007fffffce78 l:0000000000000008 o:0 t:0
$c19: v:1 s:0 p:7fff81ff b:000000016037a100 l:0000000000000100 o:20 t:0
$c20: v:0 s:0 p:7fff81ff b:0000000160377060 l:0000000000000080 o:20 t:0
$c21: v:1 s:0 p:7fff81fd b:0000007fffffc8e0 l:0000000000000020 o:0 t:0
$c22: v:1 s:0 p:7fff81ff b:0000000160370fe0 l:0000000000000400 o:20 t:0
$c23: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120029d08 t:0
$c24: v:1 s:0 p:7fff81fd b:0000007fffffcdd8 l:0000000000000001 o:0 t:0
$c26: v:1 s:0 p:7fff81ff b:0000000000000000 l:0000010000000000 o:0 t:0
$c31: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:12002c534 t:0
Mar 17 06:06:31 beri1 kernel: USER_CHERI_EXCEPTION: pid 4359 tid 100067 (host), uid 999: CP2 fault (type 0x32)
Mar 17 06:06:31 beri1 kernel: Trapframe Register Dump:
Mar 17 06:06:31 beri1 kernel: zero: 0   at: 0x1 v0: 0x1203ac508 v1: 0
Mar 17 06:06:31 beri1 kernel: a0: 0     a1: 0x80        a2: 0x4 a3: 0x7fffffc110
Mar 17 06:06:31 beri1 kernel: a4: 0x3800e000    a5: 0x3751c00   a6: 0xffffffffe8f2e3c0  a6: 0xe0e000
Mar 17 06:06:31 beri1 kernel: t0: 0x90ac62187d869c42    t1: 0   t2: 0xee0000    t3: 0
Mar 17 06:06:31 beri1 kernel: t8: 0x70e0000     t9: 0x12002c508 s0: 0x35        s1: 0x20
Mar 17 06:06:31 beri1 kernel: s2: 0x1   s3: 0x1203b1990 s4: 0   s5: 0x2
Mar 17 06:06:31 beri1 kernel: s6: 0x1c  s7: 0   k0: 0   k1: 0
Mar 17 06:06:31 beri1 kernel: gp: 0x1203b1990   sp: 0x7fffffc820        s8: 0x7fffffc820        ra: 0
Mar 17 06:06:31 beri1 kernel: sr: 0x4000fcf3    mullo: 0xf3a9   mulhi: 0        badvaddr: 0x1200a82f0
Mar 17 06:06:31 beri1 kernel: cause: 0x148      pc: 0x12002c534

$c03 is indeed untagged, although appears to have been a valid pointer previously:

$c03: v:0 s:0 p:7fff81ff b:0000000160377060 l:0000000000000080 o:20 t:0

Code as follows:

000000012002c508 <ldns_rdf_get_type>:
   12002c508:   67bdffa0        daddiu  sp,sp,-96
   12002c50c:   ebcbeac3        csd     s8,sp,88(c11)
   12002c510:   fa2be820        csc     c17,sp,32(c11)
   12002c514:   03a0f02d        move    s8,sp
   12002c518:   49b96002        cgetoffset      t9,c12
   12002c51c:   3c010038        lui     at,0x38
   12002c520:   0039102d        daddu   v0,at,t9
   12002c524:   48810007        cfromptr        c1,c0,zero
   12002c528:   49c11841        cne     at,c3,c1
   12002c52c:   10200007        beqz    at,12002c54c <ldns_rdf_get_type+0x44>
   12002c530:   00000000        nop
   12002c534:   c8430046        clw     v0,zero,8(c3)                                    <--- boom here
   12002c538:   03c0e82d        move    sp,s8
   12002c53c:   da2be820        clc     c17,sp,32(c11)
   12002c540:   cbcbeac3        cld     s8,sp,88(c11)
   12002c544:   49008800        cjr     c17
   12002c548:   67bd0060        daddiu  sp,sp,96
   12002c54c:   64415488        daddiu  at,v0,21640
   12002c550:   6422e8d8        daddiu  v0,at,-5928
   12002c554:   48810087        cfromptr        c1,c0,v0
   12002c558:   c8410003        cld     v0,zero,0(c1)
   12002c55c:   dc238038        ld      v1,-32712(at)
   12002c560:   48000807        cgetpcc c1
   12002c564:   49ac0881        csetoffset      c12,c1,v0
   12002c568:   dc228038        ld      v0,-32712(at)
   12002c56c:   6463b4c9        daddiu  v1,v1,-19255
   12002c570:   dc218038        ld      at,-32712(at)
   12002c574:   488300c7        cfromptr        c3,c0,v1
   12002c578:   6442b47e        daddiu  v0,v0,-19330
   12002c57c:   48840087        cfromptr        c4,c0,v0
   12002c580:   6421b4be        daddiu  at,at,-19266
   12002c584:   48850047        cfromptr        c5,c0,at
   12002c588:   48f16000        cjalr   c17,c12
   12002c58c:   64040021        daddiu  a0,zero,33

Possibly we have an errant non-memcpy version of memcpy somewhere? (or a memory overwrite bug involving wherever $c3 was loaded from?).

Loaded capabilities for CJALR targets appear (sometimes) $c0-derived rather than $pcc-derived?

When I compile /bin/echo with CheriABI:

--- a/bin/echo/Makefile
+++ b/bin/echo/Makefile
@@ -1,6 +1,9 @@
 #      @(#)Makefile    8.1 (Berkeley) 5/31/93
 # $FreeBSD$

+WANT_CHERI=pure
+WANT_DUMP=yes
+
 PROG=  echo

 .include <bsd.prog.mk>

(and also with other programs) I frequently see crashes involving trying to CJALR to a $c12 that doesn't have the execute bit set:

login: cpuid = 0
cpuid = 0
CHERI cause: ExcCode: 0x11 RegNum: $c12 (permit execute violation)
$c00: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
$c01: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120069060 t:0
$c02: v:1 s:0 p:7fff81ff b:0000000160069fe0 l:0000000000000100 o:a0 t:0
$c03: v:1 s:0 p:7fff81ff b:0000000160069fe0 l:0000000000000100 o:20 t:0
$c04: v:1 s:0 p:7fff81ff b:000000016006aee0 l:0000000000000100 o:0 t:0
$c05: v:1 s:0 p:7fff81ff b:000000016006aee0 l:0000000000000100 o:0 t:0
$c06: v:1 s:0 p:7fff81fd b:0000007fffffdf40 l:0000000000000020 o:0 t:0
$c07: v:1 s:0 p:7fff81fd b:0000007fffffdf20 l:0000000000000020 o:0 t:0
$c08: v:1 s:0 p:7fff81fd b:0000007fffffdf00 l:0000000000000020 o:0 t:0
$c09: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120058e10 t:0
$c10: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:12004b660 t:0
$c11: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
$c12: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120011330 t:0
$c13: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
$c14: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
$c15: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
$c16: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
$c17: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120001dc4 t:0
$c18: v:1 s:0 p:7fff81ff b:0000000160069fe0 l:0000000000000100 o:20 t:0
$c19: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120002040 t:0
$c20: v:1 s:0 p:7fff81fd b:0000007fffffddf4 l:0000000000000004 o:0 t:0
$c21: v:1 s:0 p:7fff81fd b:0000007fffffde60 l:0000000000000020 o:0 t:0
$c22: v:1 s:0 p:7fff81fd b:0000007fffffde98 l:0000000000000004 o:0 t:0
$c23: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7fffffe240 t:0
$c24: v:1 s:0 p:7fff81fd b:0000007fffffde40 l:0000000000000020 o:0 t:0
$c26: v:1 s:0 p:7fff81ff b:0000000000000000 l:0000010000000000 o:0 t:0
$c31: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120002078 t:0
Mar 14 21:44:56 beri1 kernel: USER_CHERI_EXCEPTION: pid 788 tid 100064 (echo), uid 999: CP2 fault (type 0x32)
Mar 14 21:44:56 beri1 kernel: Trapframe Register Dump:
Mar 14 21:44:56 beri1 kernel: zero: 0   at: 0x400       v0: 0x120002040 v1: 0
Mar 14 21:44:56 beri1 kernel: a0: 0x1   a1: 0x2 a2: 0x1002      a3: 0
Mar 14 21:44:56 beri1 kernel: a4: 0     a5: 0   a6: 0   a6: 0
Mar 14 21:44:56 beri1 kernel: t0: 0     t1: 0   t2: 0   t3: 0
Mar 14 21:44:56 beri1 kernel: t8: 0     t9: 0x120002040 s0: 0x2 s1: 0x120081110
Mar 14 21:44:56 beri1 kernel: s2: 0x400 s3: 0xffffffffffffffff  s4: 0x2 s5: 0x2
Mar 14 21:44:56 beri1 kernel: s6: 0x2   s7: 0x63        k0: 0   k1: 0
Mar 14 21:44:56 beri1 kernel: gp: 0x40  sp: 0x7fffffdca0        s8: 0x7fffffdca0
        ra: 0x80
Mar 14 21:44:56 beri1 kernel: sr: 0x4000fcf3    mullo: 0x8000   mulhi: 0       badvaddr: 0x16006a0e0
Mar 14 21:44:56 beri1 kernel: cause: 0x148      pc: 0x120002078
Mar 14 21:45:36 beri1 sshd[791]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key
Mar 14 21:47:55 beri1 sshd[793]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key

The code disassembles as follows:

0000000120002040 <writev>:
   120002040:   67bdffa0        daddiu  sp,sp,-96
   120002044:   ebcbeac3        csd     s8,sp,88(c11)  
   120002048:   fa2be820        csc     c17,sp,32(c11)
   12000204c:   03a0f02d        move    s8,sp
   120002050:   49b96002        cgetoffset      t9,c12
   120002054:   3c010008        lui     at,0x8
   120002058:   0039082d        daddu   at,at,t9
   12000205c:   6421f0d0        daddiu  at,at,-3888
   120002060:   64218068        daddiu  at,at,-32664
   120002064:   48810047        cfromptr        c1,c0,at
   120002068:   c8210003        cld     at,zero,0(c1)
   12000206c:   48810047        cfromptr        c1,c0,at
   120002070:   64010400        daddiu  at,zero,1024
   120002074:   d9810800        clc     c12,at,0(c1)
   120002078:   48f16000        cjalr   c17,c12                         <--- Boom here
   12000207c:   00000000        nop
   120002080:   03c0e82d        move    sp,s8
   120002084:   da2be820        clc     c17,sp,32(c11)
   120002088:   cbcbeac3        cld     s8,sp,88(c11)
   12000208c:   49008800        cjr     c17
   120002090:   67bd0060        daddiu  sp,sp,96

It looks like $c12 is loaded from a vector of initialised capabilities. $c12 points where one might expect, the __sys_writev system call stub:

0000000120011330 <__sys_writev>:
   120011330:   34020079        li      v0,0x79
   120011334:   0000000c        syscall
   120011338:   14e00003        bnez    a3,120011348 <err>
   12001133c:   00000000        nop
   120011340:   49008800        cjr     c17  
   120011344:   00000000        nop

This suggests that the pointer value for this capability is correct, but the permissions are wrong. I'm guessing that (perhaps) the linker isn't setting up capabilities to __sys_writev() and friends with suitable permissions? Is the C startup/runtime code always deriving statically initialised capabilities from $c0, even though sometimes they should be $pcc-derived? Or is the linker not passing on the right information to select the right originating capability?

(@davidchisnall may also take a view on this?)

printf %p prints 32-bit value

Printing a capability with base 0x7fffffe274 and offset 0 prints 0xffff03fe with %p.

units(1) crashes with a length violation on CheriABI - possibly varargs related?

When compiling units(1) for CheriABI:

--- a/usr.bin/units/Makefile
+++ b/usr.bin/units/Makefile
@@ -2,6 +2,9 @@

 .include <src.opts.mk>

+WANT_CHERI=pure
+WANT_DUMP=yes
+
 PROG=  units
 FILES= units.lib
 FILESDIR=      ${SHAREDIR}/misc

I encounter a tag violation on running it:

> ./units
Signal 34

With the following console output:

cpuid = 0
CHERI cause: ExcCode: 0x01 RegNum: $c02 (length violation)
$c00: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
$c01: v:1 s:0 p:7fff81fd b:0000007ffffee000 l:0000000000000020 o:0 t:0
$c02: v:1 s:0 p:000081d5 b:0000007ffffee140 l:0000000000000008 o:0 t:0
$c03: v:1 s:0 p:000081d5 b:0000007ffffee140 l:0000000000000008 o:20 t:0
$c04: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
$c05: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120244c60 t:0
$c06: v:1 s:0 p:7fff81ff b:00000001601fa000 l:0000000000000400 o:360 t:0
$c07: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7ffffed980 t:0
$c08: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7ffffed918 t:0
$c09: v:1 s:0 p:7fff81ff b:0000007fffffed04 l:0000000000000014 o:13 t:0
$c10: v:1 s:0 p:7fff81ff b:0000000160214000 l:0000000000000400 o:2e t:0
$c11: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
$c12: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120018840 t:0
$c13: v:1 s:0 p:000081d5 b:0000007ffffee140 l:0000000000000008 o:0 t:0
$c14: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201b7e1f t:0
$c15: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201b7ea7 t:0
$c16: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201abd1c t:0
$c17: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120003050 t:0
$c18: v:1 s:0 p:7fff81ff b:00000001601f1000 l:0000000000001000 o:20 t:0
$c19: v:1 s:0 p:7fff81fd b:0000007ffffee3bc l:0000000000000004 o:0 t:0
$c20: v:1 s:0 p:7fff81ff b:00000001601ef000 l:0000000000000200 o:20 t:0
$c21: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
$c22: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201a4330 t:0
$c23: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120002a00 t:0
$c24: v:1 s:0 p:7fff81fd b:0000007ffffedffc l:0000000000000004 o:0 t:0
$c26: v:1 s:0 p:7fff81ff b:0000000000000000 l:0000010000000000 o:0 t:0
$c31: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120018b08 t:0
Mar 16 20:56:04 beri1 kernel: USER_CHERI_EXCEPTION: pid 2840 tid 100068 (units), uid 999: CP2 fault (type 0x32)
Mar 16 20:56:04 beri1 kernel: Trapframe Register Dump:
Mar 16 20:56:04 beri1 kernel: zero: 0   at: 0x120010000 v0: 0xffffffffffffffe0  v1: 0x4
Mar 16 20:56:04 beri1 kernel: a0: 0x120018ae0   a1: 0x1201a0000 a2: 0x4 a3: 0
Mar 16 20:56:04 beri1 kernel: a4: 0     a5: 0x8 a6: 0x7ffffebed8        a6: 0x7ffffebed0
Mar 16 20:56:04 beri1 kernel: t0: 0x10  t1: 0xc t2: 0x7ffffebac0        t3: 0x7ffffeba40
Mar 16 20:56:04 beri1 kernel: t8: 0xc   t9: 0x120018840 s0: 0   s1: 0x8
Mar 16 20:56:04 beri1 kernel: s2: 0x1201f49d0   s3: 0x8 s4: 0x1 s5: 0x1201f0000
Mar 16 20:56:04 beri1 kernel: s6: 0x120000000   s7: 0xffffffffffffffff  k0: 0   k1: 0
Mar 16 20:56:04 beri1 kernel: gp: 0x7ffffee380  sp: 0x7ffffedac0        s8: 0x7ffffedac0        ra: 0x7ffffee380
Mar 16 20:56:04 beri1 kernel: sr: 0x4000fcf3    mullo: 0        mulhi: 0        badvaddr: 0x120003000
Mar 16 20:56:04 beri1 kernel: cause: 0x148      pc: 0x120018b08

LLVM has helpfully provided an uninformative block name $BB99_13, for which there isn't an obvious jump source:

0000000120018ae0 <$BB99_13>:
   120018ae0:   d8410000        clc     c2,zero,0(c1)  
   120018ae4:   49a11002        cgetoffset      at,c2
   120018ae8:   6421001f        daddiu  at,at,31
   120018aec:   6402ffe0        daddiu  v0,zero,-32
   120018af0:   00220824        and     at,at,v0
   120018af4:   49a21041        csetoffset      c2,c2,at
   120018af8:   64010020        daddiu  at,zero,32
   120018afc:   49a31040        cincoffset      c3,c2,at
   120018b00:   de4180e8        ld      at,-32536(s2)
   120018b04:   f8610000        csc     c3,zero,0(c1)
   120018b08:   d8820000        clc     c4,zero,0(c2)                   <-- Boom here
   120018b0c:   48000807        cgetpcc c1
   120018b10:   6421d720        daddiu  at,at,-10464
   120018b14:   49ac0841        csetoffset      c12,c1,at
   120018b18:   00102800        sll     a1,s0,0x0
   120018b1c:   64040000        daddiu  a0,zero,0
   120018b20:   64060000        daddiu  a2,zero,0
   120018b24:   48f16000        cjalr   c17,c12
   120018b28:   49a39000        cmove   c3,c18
   120018b2c:   08006288        j       120018a20 <$BB99_9+0x8>
   120018b30:   e8580002        csw     v0,zero,0(c24)

It indeed looks like $c02 is too short for a capability load:

$c02: v:1 s:0 p:000081d5 b:0000007ffffee140 l:0000000000000008 o:0 t:0

Unfortunately, the disassembly does not make it clear how we got here, but given the return jump to $BB99_9, perhaps it originated somewhere near there (you never know):

0000000120018a18 <$BB99_9>:
   120018a18:   2401ffff        li      at,-1
   120018a1c:   e8380002        csw     at,zero,0(c24)
   120018a20:   c8580006        clw     v0,zero,0(c24)               <-- Jumps to here rather than 120018a18?
   120018a24:   03c0e82d        move    sp,s8
   120018a28:   67a10560        daddiu  at,sp,1376
   120018a2c:   da2b0800        clc     c17,at,0(c11)
   120018a30:   67a10580        daddiu  at,sp,1408
   120018a34:   da4b0800        clc     c18,at,0(c11)   
   120018a38:   67a105a0        daddiu  at,sp,1440
   120018a3c:   da6b0800        clc     c19,at,0(c11)
   120018a40:   67a105c0        daddiu  at,sp,1472
   120018a44:   da8b0800        clc     c20,at,0(c11)
   120018a48:   67a105e0        daddiu  at,sp,1504
   120018a4c:   daab0800        clc     c21,at,0(c11)
   120018a50:   67a10600        daddiu  at,sp,1536
   120018a54:   dacb0800        clc     c22,at,0(c11)
   120018a58:   67a10620        daddiu  at,sp,1568
   120018a5c:   daeb0800        clc     c23,at,0(c11)
   120018a60:   67a10640        daddiu  at,sp,1600  
   120018a64:   db0b0800        clc     c24,at,0(c11)
   120018a68:   67a10660        daddiu  at,sp,1632    
   120018a6c:   ca0b0803        cld     s0,at,0(c11)
   120018a70:   67a10668        daddiu  at,sp,1640
   120018a74:   ca2b0803        cld     s1,at,0(c11)
   120018a78:   67a10670        daddiu  at,sp,1648
   120018a7c:   ca4b0803        cld     s2,at,0(c11)
   120018a80:   67a10678        daddiu  at,sp,1656
   120018a84:   cbcb0803        cld     s8,at,0(c11)
   120018a88:   49008800        cjr     c17
   120018a8c:   67bd0680        daddiu  sp,sp,1664

This code is close to, and might be part of:

0000000120018840 <el_set>:
   120018840:   67bdf980        daddiu  sp,sp,-1664
   120018844:   67a10678        daddiu  at,sp,1656
   120018848:   ebcb0803        csd     s8,at,0(c11)
   12001884c:   67a10670        daddiu  at,sp,1648
   120018850:   ea4b0803        csd     s2,at,0(c11)
   120018854:   67a10668        daddiu  at,sp,1640
   120018858:   ea2b0803        csd     s1,at,0(c11)
   12001885c:   67a10660        daddiu  at,sp,1632
   120018860:   ea0b0803        csd     s0,at,0(c11)
   120018864:   67a10640        daddiu  at,sp,1600
   120018868:   fb0b0800        csc     c24,at,0(c11)
   12001886c:   67a10620        daddiu  at,sp,1568
   120018870:   faeb0800        csc     c23,at,0(c11)
   120018874:   67a10600        daddiu  at,sp,1536
   120018878:   facb0800        csc     c22,at,0(c11)
   12001887c:   67a105e0        daddiu  at,sp,1504
   120018880:   faab0800        csc     c21,at,0(c11)
   120018884:   67a505c0        daddiu  a1,sp,1472
   120018888:   fa8b2800        csc     c20,a1,0(c11)
   12001888c:   67a505a0        daddiu  a1,sp,1440
   120018890:   fa6b2800        csc     c19,a1,0(c11)
   120018894:   67a50580        daddiu  a1,sp,1408
   120018898:   fa4b2800        csc     c18,a1,0(c11)
   12001889c:   67a50560        daddiu  a1,sp,1376
   1200188a0:   fa2b2800        csc     c17,a1,0(c11)
   1200188a4:   03a0f02d        move    s8,sp
   1200188a8:   0080802d        move    s0,a0
   1200188ac:   49b21800        cmove   c18,c3
   1200188b0:   49b96002        cgetoffset      t9,c12  
   1200188b4:   3c01001e        lui     at,0x1e
   1200188b8:   0039202d        daddu   a0,at,t9
   1200188bc:   48950007        cfromptr        c21,c0,zero
   1200188c0:   49c19540        ceq     at,c18,c21
   1200188c4:   14200057        bnez    at,120018a24 <$BB99_9+0xc>
   1200188c8:   2402ffff        li      v0,-1
   1200188cc:   67c10540        daddiu  at,s8,1344
   1200188d0:   49a15841        csetoffset      c1,c11,at
   1200188d4:   64020020        daddiu  v0,zero,32
   1200188d8:   48210880        csetbounds      c1,c1,v0
   1200188dc:   67c1053c        daddiu  at,s8,1340
   1200188e0:   49a25841        csetoffset      c2,c11,at
   1200188e4:   64030004        daddiu  v1,zero,4
   1200188e8:   483810c0        csetbounds      c24,c2,v1
   1200188ec:   49810800        ctoptr  at,c1,c0
   1200188f0:   f9ab0800        csc     c13,at,0(c11)
   1200188f4:   2e010019        sltiu   at,s0,25
   1200188f8:   0010283c        dsll32  a1,s0,0x0
   1200188fc:   10200046        beqz    at,120018a18 <$BB99_9>    
   120018900:   00000000        nop
   120018904:   6492c190        daddiu  s2,a0,-15984
   120018908:   0005203e        dsrl32  a0,a1,0x0
   12001890c:   de458040        ld      a1,-32704(s2) 
   120018910:   64010008        daddiu  at,zero,8
   120018914:   0081001c        dmult   a0,at
   120018918:   00002012        mflo    a0
   12001891c:   0085202d        daddu   a0,a0,a1
   120018920:   dc844b50        ld      a0,19280(a0)
   120018924:   0092202d        daddu   a0,a0,s2
   120018928:   00800008        jr      a0
   12001892c:   00000000        nop

This in turn seems plausibly called from main(), which might be the function the basic block references by $c17 is in ($BB0_15). Which seems compatible to with the units(1) source code, where main() calls el_set):

                el = el_init(argv[0], stdin, stdout, stderr);
                el_set(el, EL_PROMPT, &prompt);
                el_set(el, EL_EDITOR, "emacs");
                el_set(el, EL_SIGNAL, 1);
                el_set(el, EL_HIST, history, inhistory);
                el_source(el, NULL);
                history(inhistory, &ev, H_SETSIZE, 800);

It seems a good bet we're looking at an issue with varargs and hence possibly a toolchain issue. I can't easily tell from the available debugging information to hand which it is, but it's a good bet that it's the first, which passes a pointer to a function into libeditline:

static const char * prompt(EditLine *e __unused) {
        return promptstr;
}

A close second best guess would be the passing in of inhistory:

        History *inhistory;

Perhaps @davidchisnall could opine on this use of varargs by units(1)?

units(1) fails with tag violation in ncurses

# ./units
Signal 34

Register dump:

CHERI cause: ExcCode: 0x02 RegNum: C03 (tag violation)
C00: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
C01: v:1 s:0 p:7fff81ff b:00000001601ed180 l:0000000000000080 o:20 t:0
C02: v:1 s:0 p:7fff81ff b:0000000160205000 l:0000000000002000 o:247 t:0
C03: v:0 s:0 p:7fff81ff b:0000000160205000 l:0000000000002000 o:4d t:0
C04: v:1 s:0 p:7fff81ff b:0000000160205000 l:0000000000002000 o:20 t:0
C05: v:0 s:0 p:39b9ba3a b:79797a7a7b7b7c7c l:7d7d7e7e2e2e2d2d o:fbfbfbfbfbfbfbfc
 t:717272
C06: v:1 s:0 p:7fff81fd b:0000007ffffe9fd4 l:0000000000000004 o:0 t:0
C07: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7ffffea5b8 t:0
C08: v:1 s:0 p:7fff81ff b:00000001601fb800 l:0000000000000800 o:427 t:0
C09: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201a5ed3 t:0
C10: v:1 s:0 p:7fff81ff b:00000001601fb800 l:0000000000000800 o:427 t:0
C11: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
C12: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:1200f02d0 t:0
C13: v:1 s:0 p:000081d5 b:0000007ffffea880 l:0000000000000008 o:0 t:0
C14: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201b7fc7 t:0
C15: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201b804f t:0
C16: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201abec4 t:0
C17: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:1200483c4 t:0
C18: v:1 s:0 p:7fff81fd b:0000007ffffea860 l:0000000000000fa0 o:0 t:0
C19: v:1 s:0 p:7fff81fd b:0000007ffffe7f38 l:0000000000000008 o:0 t:0
C20: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:1200f02d0 t:0
C21: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:12020c1c0 t:0
C22: v:1 s:0 p:7fff81fd b:0000007ffffea860 l:0000000000000fa0 o:0 t:0
C23: v:1 s:0 p:7fff81fd b:0000007ffffe8000 l:0000000000002000 o:0 t:0
C24: v:1 s:0 p:7fff81fd b:0000007ffffe7f80 l:0000000000000080 o:0 t:0
C26: v:1 s:0 p:7fff81ff b:0000000000000000 l:0000010000000000 o:0 t:0
C31: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:1200f02d4 t:0
Oct 14 18:19:52 beri1 kernel: USER_CHERI_EXCEPTION: pid 11551 tid 100067 (units), uid 0: CP2 fault (type 0x32)
Oct 14 18:19:52 beri1 kernel: Trapframe Register Dump:
Oct 14 18:19:52 beri1 kernel: zero: 0   at: 0x1 v0: 0   v1: 0x247
Oct 14 18:19:52 beri1 kernel: a0: 0x20  a1: 0xfffffffffffffffe  a2: 0x80       a3: 0xffffffffffffffff
Oct 14 18:19:52 beri1 kernel: a4: 0xffffffffffffffff    a5: 0xffffffffffffffff a6: 0xfffffffffffffffe   a6: 0xffffffffffffffff
Oct 14 18:19:52 beri1 kernel: t0: 0     t1: 0   t2: 0x7ffffea020        t3: 0x7ffffe9fa0
Oct 14 18:19:52 beri1 kernel: t8: 0x10  t9: 0x1201a34e8 s0: 0x3 s1: 0
Oct 14 18:19:52 beri1 kernel: s2: 0x120210000   s3: 0x2 s4: 0x1 s5: 0x1
Oct 14 18:19:52 beri1 kernel: s6: 0xc0  s7: 0x20        k0: 0   k1: 0
Oct 14 18:19:52 beri1 kernel: gp: 0x1201f4b70   sp: 0x7ffffe7e60        s8: 0x7ffffe7e60        ra: 0
Oct 14 18:19:52 beri1 kernel: sr: 0x4000fcf3    mullo: 0x6997f3a348a0df44      mulhi: 0xffffffffffffcf9f        badvaddr: 0x160214000
Oct 14 18:19:52 beri1 kernel: cause: 0x148      pc: 0x1200f02d4

The crash occurs in strlen at:

00000001200f02d0 <strlen>:
   1200f02d0:   34020000        li      v0,0x0
   1200f02d4:   c8631004        clb     v1,v0,0(c3)

$c17 tells us that we're in _nc_wrap_entry in contrib/ncurses/ncurses/tinfo/alloc_entry.c.

The obvious garbage in $c5 may be interesting.

/bin/sh experiences a tag violation in CheriABI - NULL pointer to setjmp()?

Using /bin/sh as compiled for CheriABI:

--- a/bin/sh/Makefile
+++ b/bin/sh/Makefile
@@ -3,6 +3,9 @@

 .include <src.opts.mk>

+WANT_CHERI=pure
+WANT_DUMP=yes
+
 PROG=  sh
 INSTALLFLAGS= -S
 SHSRCS=        alias.c arith_yacc.c arith_yylex.c cd.c echo.c error.c eval.c \

I experience a tag violation:

> ./sh
Signal 34

Console debugging information:

cpuid = 0
CHERI cause: ExcCode: 0x02 RegNum: $c03 (tag violation)
$c00: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
$c01: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:12022dc98 t:0
$c02: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:12022f678 t:0
$c03: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
$c04: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201e1c40 t:0
$c05: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
$c06: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201fb3d4 t:0
$c07: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:1201c9a18 t:0
$c08: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201fac18 t:0
$c09: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201fac26 t:0
$c10: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201fac2f t:0
$c11: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
$c12: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:1200b2e10 t:0
$c13: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201f748a t:0
$c14: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201f5aab t:0
$c15: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201f5b33 t:0
$c16: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201f3898 t:0
$c17: v:0 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1202119c0 t:1
$c18: v:1 s:0 p:7fff81fd b:0000007fffffde80 l:0000000000000060 o:0 t:0
$c19: v:1 s:0 p:7fff81fd b:0000007fffffdee0 l:0000000000000060 o:0 t:0
$c20: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7fffffe280 t:0
$c21: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:12027b858 t:0
$c22: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120023bd0 t:0
$c23: v:1 s:0 p:7fff81fd b:0000007fffffde7c l:0000000000000004 o:0 t:0
$c24: v:1 s:0 p:7fff81fd b:0000007fffffe080 l:0000000000000020 o:0 t:0
$c26: v:1 s:0 p:7fff81ff b:0000000000000000 l:0000010000000000 o:0 t:0
$c31: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:1200b2e2c t:0
Mar 16 21:59:05 beri1 kernel: USER_CHERI_EXCEPTION: pid 2988 tid 100048 (sh), uid 999: CP2 fault (type 0x32)
Mar 16 21:59:05 beri1 kernel: Trapframe Register Dump:
Mar 16 21:59:05 beri1 kernel: zero: 0   at: 0x1200b2e10 v0: 0xffffffffbadfaced  v1: 0x1
Mar 16 21:59:05 beri1 kernel: a0: 0x53  a1: 0   a2: 0x50        a3: 0xe0
Mar 16 21:59:05 beri1 kernel: a4: 0x100 a5: 0x120       a6: 0x160       a6: 0x140
Mar 16 21:59:05 beri1 kernel: t0: 0     t1: 0   t2: 0x1c0       t3: 0x2
Mar 16 21:59:05 beri1 kernel: t8: 0     t9: 0x1200f0cb8 s0: 0x1 s1: 0x120235a10
Mar 16 21:59:05 beri1 kernel: s2: 0x20  s3: 0   s4: 0   s5: 0
Mar 16 21:59:05 beri1 kernel: s6: 0     s7: 0   k0: 0   k1: 0
Mar 16 21:59:05 beri1 kernel: gp: 0     sp: 0x7fffffdea0        s8: 0   ra: 0x10000000000
Mar 16 21:59:05 beri1 kernel: sr: 0x4000fcf3    mullo: 0        mulhi: 0        badvaddr: 0x120130b58
Mar 16 21:59:05 beri1 kernel: cause: 0x148      pc: 0x1200b2e2c

$c03 indeed appears to be a NULL (and hence untagged) pointer:

$c03: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0

The $pc value (1200b2e2c) is in� _setjmp:

00000001200b2e10 <_setjmp>:
   1200b2e10:   d86be860        clc     c3,sp,96(c11)
   1200b2e14:   da2be820        clc     c17,sp,32(c11) 
   1200b2e18:   cbcbea83        cld     s8,sp,80(c11)
   1200b2e1c:   cbebeac3        cld     ra,sp,88(c11)  
   1200b2e20:   67bd0080        daddiu  sp,sp,128
   1200b2e24:   3c02badf        lui     v0,0xbadf
   1200b2e28:   3442aced        ori     v0,v0,0xaced
   1200b2e2c:   e8430003        csd     v0,zero,0(c3)         <-- Boom here
   1200b2e30:   ebe30043        csd     ra,zero,8(c3)
   1200b2e34:   ea030083        csd     s0,zero,16(c3)
   1200b2e38:   ea2300c3        csd     s1,zero,24(c3)
   1200b2e3c:   ea430103        csd     s2,zero,32(c3)
   1200b2e40:   ea630143        csd     s3,zero,40(c3)
   1200b2e44:   ea830183        csd     s4,zero,48(c3)
   1200b2e48:   eaa301c3        csd     s5,zero,56(c3)
   1200b2e4c:   eac30203        csd     s6,zero,64(c3)
   1200b2e50:   eae30243        csd     s7,zero,72(c3)
   1200b2e54:   eba30283        csd     sp,zero,80(c3)
   1200b2e58:   ebc302c3        csd     s8,zero,88(c3)
   1200b2e5c:   eb830303        csd     gp,zero,96(c3)
   1200b2e60:   480c1802        cgetbase        t0,c3
   1200b2e64:   49ad1802        cgetoffset      t1,c3
   1200b2e68:   018d682d        daddu   t1,t0,t1
   1200b2e6c:   65ac00f8        daddiu  t0,t1,248
   1200b2e70:   240effe0        li      t2,-32
   1200b2e74:   01cc7024        and     t2,t2,t0
   1200b2e78:   01cd682f        dsubu   t1,t2,t1
   1200b2e7c:   f9636800        csc     c11,t1,0(c3)   
   1200b2e80:   f9836820        csc     c12,t1,32(c3)  
   1200b2e84:   f9a36840        csc     c13,t1,64(c3) 
   1200b2e88:   f9c36860        csc     c14,t1,96(c3)
   1200b2e8c:   f9e36880        csc     c15,t1,128(c3)
   1200b2e90:   fa0368a0        csc     c16,t1,160(c3)
   1200b2e94:   fa2368c0        csc     c17,t1,192(c3)
   1200b2e98:   fa4368e0        csc     c18,t1,224(c3)
   1200b2e9c:   fa636900        csc     c19,t1,256(c3)
   1200b2ea0:   fa836920        csc     c20,t1,288(c3)
   1200b2ea4:   faa36940        csc     c21,t1,320(c3)
   1200b2ea8:   fac36960        csc     c22,t1,352(c3)
   1200b2eac:   fae36980        csc     c23,t1,384(c3)
   1200b2eb0:   fb0369a0        csc     c24,t1,416(c3)
   1200b2eb4:   49a40000        cgetdefault     c4
   1200b2eb8:   f88369c0        csc     c4,t1,448(c3)
   1200b2ebc:   0000102d        move    v0,zero
   1200b2ec0:   49008800        cjr     c17

Unfortunately, it's a bit hard to work out which context we're in, as $c17 doesn't point to a useful place:

0000000120211760 <__CTOR_LIST__>:
   120211760:   ffffffff        sd      ra,-1(ra)
   120211764:   ffffffff        sd      ra,-1(ra)
...
   1202119b0:   00000001        movf    zero,zero,$fcc0
   1202119b4:   201be078        addi    k1,zero,-8072
   1202119b8:   00000001        movf    zero,zero,$fcc0
   1202119bc:   201be468        addi    k1,zero,-7064
   1202119c0:   00000001        movf    zero,zero,$fcc0        <-- $c17 points here?
   1202119c4:   201ca510        addi    gp,zero,-23280  
   1202119c8:   00000001        movf    zero,zero,$fcc0
   1202119cc:   201cc1b0        addi    gp,zero,-15952

However, it sounds like we might have an issue with the capability-aware setjmp/longjmp implementation.

Also tagging @davidchisnall in case he wants to opine

Multi-stack support for libcheri compartments

Currently, libcheri compartments are single-threaded and non-reentrant. Support should be added so that compartments can stash a set of stacks, rather than one stack, and pick the first available one. If the set is exhausted, we may need to allocate more -- or perform some garbage collection activity. It's not yet clear how we should handle compartment faults: a cleanly returning compartment can return its own stack. But a faulting compartment doesn't have the opportunity to do so. Possibly the libcheri runtime should forceably return the stack in that case as part of unwinding?

Depends on #8.

Capsicum-like syscall set for unprivileged CHERI compartments

The set of syscalls permitted for Capsicum compartments includes several things that should not be permitted for unprivileged compartments. These include (but are not limited to):

• File descriptor operations (e.g. read, write) that take a (forgeable) integer rather than a capability.
• Virtual address space modification functions (e.g. mmap, mprotect), which should not be permitted as they can be used to break CoW guarantees for sandboxes
• Signal handling functions, which could allow the sandbox to provide code to run with elevated privilege.

A modified libsyscall should allow banned syscalls to be issued as methods on the system object.

optimized CHERI assembly versions of string functions

There are MIPS assembly versions of many string functions which typically use optimizations based on word size. Many could be reimplemented with CHERI assembly with some care.

Function	Notes
bcmp	Optimized version assumes word reads are always safe, need to handle poorly aligned capability lengths.
bcopy	memmove with different argument order
bzero	Aligns the head, writes zeroed words, then cleans up the tail. Do the same with full capability writes in the middle.
memmove	bcopy with different argument order
strchr	Simple bytewise comparison loop.
strcmp	Simple bytewise comparison loop.
strrchr	Simple bytewise comparison loop.

NB: In all cases it's worth checking in the compiler generates equivalently optimal assembly before writing new code.

CheriABI - kernel support

Supporting a userspace execution environment in which all pointers are capabilities requires a system-call interface that understands capabilities, and as necessary, maps them to/from pointers for use by the kernel. In some cases, we need to retain the full capability in the ABI in order to return pointers to userspace with tags (i.e., opaque pointer-aware types in kqueue). It would also be nice (very good indeed) to respect capability bounds when using a userspace-provided capability for access to user memory.

This issue is for kernel-side support for CheriABI.

Teach FreeBSD/mips kernel to work with PIC code generation

Currently, the FreeBSD/mips kernel is designed to be compiled as non-PIC code. As non-PIC code generation isn't supported by LLVM on MIPS, we should switch the kernel to support PIC. This requires some changes to the build system, but also changes to in-kernel assembly to set up $t9 (i.e., do ABI calls), and likely a little bit of non-PIC code generation for inline assembly during the boot and exception handling to set up $gp and friends, as we can't rely on the calling context to do that. We should also check that the context-switch code and various other MIPS-specific bits of code behave well here.

No posix_memalign(2) in CheriABI

In trying to build cp(1) with CheriABI, I encountered the fact that posix_memalign(2) is missing in CheriABI. This appears to be used by the POSIX.1e ACL code in libc.

split syscall code out of libc

We would like to be able to replace the custom libc_cheri with the standard libc, but doing so requires that syscalls be proxied through the cheri_system class. Splitting the system call implementations into a libsyscalls would allow this and enable other novel consumers of libc to replace the system call implementation with their own.

Use xkphys rather than PCB fixed-TLB space for each thread's trusted stack

Currently, the trusted stack for userspace compartmentalisation is embedded in the machine-dependent portion of the PCB (thread control block). This is necessary so that the trusted stack is in a region of memory that cannot through TLB misses -- and the PCB occupies the fixed TLB entries reserved for the kernel thread stack. This overfills the fixed area, and also bounds the size of the trusted stack to a very small number of frames. This should be fixed by shifting the trusted stack to contigmalloc()'d memory pointed to by the PCB, which will be exposed via the xkphys region so that it doesn't suffer page faults.

Capsicum ioctl limiting not available in CheriABI?

I built uniq(1) with CheriABI, and ran into the following problem:

MIPS ABI uniq:

root@beri1:/tmp # yes | head -10 | uniq -c
  10 y

CheriABI uniq:

root@beri1:/tmp # yes | head -10 | ./uniq -c
uniq: unable to limit ioctls for stdout: No error: 0

ktrace seems to confirm:

  2679 ktrace   NAMI  "./uniq"
  2679 uniq     RET   execve 0
  2679 uniq     CALL  cap_rights_limit(0,0x7fffffde70)
  2679 uniq     STRU  cap_rights_t <CAP_READ,CAP_FSTAT>
  2679 uniq     RET   cap_rights_limit 0
  2679 uniq     CALL  cap_rights_limit(0x1,0x7fffffde70)
  2679 uniq     STRU  cap_rights_t <CAP_WRITE,CAP_FSTAT,CAP_IOCTL>
  2679 uniq     RET   cap_rights_limit 0
  2679 uniq     CALL  cap_ioctls_limit(0x1,0x7fffffde68,0x1)
  2679 uniq     RET   cap_ioctls_limit -1 errno 78 Function not implemented
  2679 uniq     CALL  write(0x2,0x7fffffcea0,0x6)
  2679 uniq     GIO   fd 2 wrote 6 bytes
       "uniq: "
  2679 uniq     RET   write 6
  2679 uniq     CALL  write(0x2,0x7fffffcee0,0x21)
  2679 uniq     GIO   fd 2 wrote 33 bytes
       "unable to limit ioctls for stdout"

brandelf -c can only be run once

brandelf -c with a file that has already been branded dies with a confusing error. It would be nice if it could silently succeed (if passed the same value) or warn (if passed the other CHERI flavour).

FreeBSD libc quicksort (qsort) has hand-crafted memcpy()

FreeBSD's in-libc version of QuickSort (qsort(3)) needs to swap array elements, and implements this via qsort.c:swapcode(). The code detects integer and long-sized copies and optimises those, but otherwise falls back on bytewise copies. This strips the tags from pointers sorted by quicksort, breaking (for example) NSS module initialisation (nss_module_load()), which wants to sort a vector of structures by ordering strings pointed to by the structures. A later pointer dereference will throw a hardware exception as a result. swapcode() needs to be updated to support tag preservation -- e.g., by introducing a memswap() that is tag preserving, or similar. It's not clear to me what the alignment expectations are for arguments to qsort(), and hence how careful a copying routine needs to be (the int/long code seems to assume pretty strong alignment).

Make the kernel execution environment for hybrid-ABI CHERI C code to run in

Currently, the kernel is compiled only using gcc and a small amount of CHERI-aware assembly code. This issue is to allow kernel .C files to be compiled to use CHERI capabilities in hybrid mode (ie., selected annotated pointers are compiled to be capabilities). This requires a number of changes including including:

• Modify the kernel's thread state and voluntary context-switch code to maintain a full set of callee-save capability registers, as is done for MIPS general-purpose registers. Right now, the kernel maintains a 'micro-ABI' -- two capability registers that it is safe for inline assembly to use, but we need the full ABI to be maintained by context switching.
• Update existing CHERI assembly code to respect the ABI -- the code that exists may or may not respect ABI requirements for capability pointers.
• Investigate how best to update kernel memory copies to propagate capabilities. Right now, any carefully selected memory copies (e.g., the pmap copy routine) preserve tags, in order to prevent errant tag leakage. We will now need in-kernel memory copies to preserve tags in some cases -- but need to ensure that copyin() and copyout() continue to drop tags.
• It would be useful to update DDB/KGDB/etc to better support capabilities, to make debugging of in-kernel capability use easier.

This work depends on #13, to allow Clang/LLVM to compile the kernel as position-independent code.

Possible user/group lookup / NSS problems in CheriABI?

When rm(1) tries to confirm deletion on an immutable (schg) file, the prompt does not correctly display user and group names:

root@beri1:/tmp # ./rm foo
override rw-r--r-- 0/0 schg for foo? y
rm: foo: No error: 0

This could be an issue with NSS?

(Note that error output problem above is described in #37, a separate issue.)

setjmp / longjmp missing

CHERIABI currently lacks a working setjmp (I believe longjmp also doesn't work, but setjmp crashes the program).

PCC should probably not be writeable

Currently, PCC has both the store and store-capability permissions. It probably shouldn't have either.

Test case:

$ ~/sdk/bin/clang -mabi=sandbox fail.c -lc -lmalloc_simple  
$ cat fail.c
#include <assert.h>

int main(void);
void foo(void)
{
    void *ret = __builtin_memcap_program_counter_get();
    // We shouldn't be able to write through code capabilities
    assert((__builtin_memcap_perms_get(ret) & __CHERI_CAP_PERMISSION_PERMIT_STORE_CAPABILITY__) == 0);
    assert((__builtin_memcap_perms_get(ret) & __CHERI_CAP_PERMISSION_PERMIT_STORE__) == 0);
}

int main(void)
{
        foo();
        return 0;
}
$ ~/sdk/bin/clang -mabi=sandbox fail.c -lc -lmalloc_simple

Copying this to a CHERI box and running it gives:

# ./a.out
Assertion failed: ((__builtin_memcap_perms_get(ret) & __CHERI_CAP_PERMISSION_PERMIT_STORE_CAPABILITY__) == 0), function foo, file fail.c, line 9.
pid 173 (a.out), uid 0: exited on signal 6 (core dumped)
Abort trap (core dumped)

CheriABI stty(1) prints incorrect baud

In running stty(1) compiled with CheriABI:

--- a/bin/stty/Makefile
+++ b/bin/stty/Makefile
@@ -1,6 +1,9 @@
 #      @(#)Makefile    8.1 (Berkeley) 5/31/93
 # $FreeBSD$

+WANT_CHERI=pure
+WANT_DUMP=yes
+
 PROG=  stty
 SRCS=  cchar.c gfmt.c key.c modes.c print.c stty.c util.c

I've discovered that it prints the baud information incorrectly:

> ./stty
#-64594 disc; speed 9600 baud;
lflags: echoe echoke echoctl
iflags: -icrnl inlcr -ixon -ixany -imaxbel -brkint inpck
oflags: -opost ocrnl tab3
cflags: cs8 -parenb

Compare with non-CheriABI stty(1):

> stty
speed 9600 baud;
lflags: echoe echoke echoctl
oflags: tab0
cflags: cs8 -parenb

It is not clear to me whether this is a CheriABI bug (e.g., in ioctl handling), an stty(1) bug that manifests only with CHERI-style compilation (e.g., relating to printf), or a toolchain bug.

(@davidchisnall may wish to opine.)

Make $c0 non-executable, $pcc non-writable in processes

Currently, the CheriBSD kernel initialises a number of userspace registers with complete rights for the user address space, including load, store, and execute. Ideally, we would initialise registers used for data access for load and store, and registers for code execution as load and execute. de0dd48 sets up the groundwork for a chance to differentiate code and data capabilities in the kernel, but requires a few things to be done first:

• libcheri needs to derive all code capabilities (sealed or otherwise) from $pcc. An issue has been opened in the CHERI Clang issue tracker about a compiler builtin for CFromPtr to make this easier.
• CheriABI injects new capabilities into the process while it is running, not just during setup, and those capabilities currently grant all rights. In some cases, they should possibly grant fewer -- e.g., mmap(2) pointers should perhaps have only the rights requested (and/or masked by the rights of a capability passed in as an argument). #58 relates to this, but there are a number of other comments in the CheriABI code about what appropriate permissions might be in various cases. Most of the time, it will involve preserving permissions passed into the kernel earlier.

Once these issues are resolved, we should be able to throw the switch. Userspace will still be able to construct sets of capabilities allowing execute and store to the same underlying virtual memory -- this is desirable to support features such as just-in-time compilation -- but by separating the two hierarchies, we prevent the construction (and hence leaking) of any single capability connoting both sets of rights simultaneously. This is probably better than not, but requires a bit of thinking as well about other implications.

NB: Merge request #44 attempted to accomplish something similar performed solely in the CheriBSD C-language startup code. I think it is probably better done in the kernel such that it affects both CheriABI and non-CheriABI processes, but can see arguments both ways.

errno difficulties in CheriABI: errno possibly broken, and progname not printed correctly by err(3).

Compiling 'realpath' for CheriABI, it appears to work, but error handling is wonky when a file doesn't exist:

root@beri1:/tmp # ./realpath foo
/tmp/foo
root@beri1:/tmp # ./realpath bar
: bar: No error: 0
root@beri1:/tmp # realpath bar
realpath: bar: No such file or directory

It looks likely that errno is not being set (or perhaps not being accesses) correctly, and also that the getprogname() mechanism used in err(3) is not working.

Coredump support for tagged memory

We have extended the coredump format to store CHERI capability registers -- but not tags on memory, which should be implemented.

Move to external stack for libcheri compartments

Currently, the stack used by a libcheri compartment is within the global $c0 of the compartment. It would be preferable to have an externally allocated stack for compartments so as to differentiate the main data segment from the stack, and also as a step towards multi-stack compartments.

The initial version of this change should allocate a stack for each libcheri sandbox object, and install a suitably permissioned capability to that stack in the sandbox metadata. Rather than constructing a stack pointer from $c0, entry code should load its stack capability from the sandbox metadata.

/bin/date experiences intermittent crashes with CheriABI

If I compile /bin/date with CheriABI:

--- a/bin/date/Makefile
+++ b/bin/date/Makefile
@@ -3,6 +3,9 @@

 .include <src.opts.mk>

+WANT_CHERI=pure
+WANT_DUMP=yes
+
 PROG=  date
 SRCS=  date.c netdate.c vary.c

Then I experience intermittent crashes with the resulting binary:

> ./date
Sat Mar 14 17:44:00 UTC 2015
> ./date
Signal 34

This may have to do with CFI in CheriABI:

FreeBSD/mips (beri1) (ttyj0)

login: cpuid = 0
cpuid = 0
CHERI cause: ExcCode: 0x11 RegNum: C12 (permit execute violation)
C00: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
C01: v:1 s:0 p:7fff81fd b:000000012015b680 l:0000000000000900 o:300 t:0
C02: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:12012f8e1 t:0
C03: v:1 s:0 p:7fff81fd b:000000012015b680 l:0000000000000900 o:300 t:0
C04: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:12012f8e1 t:0
C05: v:1 s:0 p:7fff81fd b:0000007fffffda80 l:0000000000000400 o:400 t:0
C06: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:12012a0e0 t:0
C07: v:1 s:0 p:7fff81fd b:0000007fffffc09c l:0000000000000004 o:0 t:0
C08: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:120159bc0 t:0
C09: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7fffffcc44 t:0
C10: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7fffffcc40 t:0
C11: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:0 t:0
C12: v:1 s:0 p:7fff81fd b:000000012015b680 l:0000000000000900 o:120052d38 t:0
C13: v:1 s:0 p:000081d5 b:0000007fffffd680 l:0000000000000020 o:0 t:0
C14: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:7fffffcc48 t:0
C15: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
C16: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
C17: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120002a2c t:0
C18: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120002060 t:0
C19: v:1 s:0 p:7fff81fd b:0000007fffffd960 l:0000000000000060 o:0 t:0
C20: v:1 s:0 p:7fff81fd b:0000007fffffda80 l:0000000000000400 o:0 t:0
C21: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120005790 t:0
C22: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:120002060 t:0
C23: v:0 s:0 p:00000000 b:0000000000000000 l:0000000000000000 o:0 t:0
C24: v:1 s:0 p:7fff81fd b:0000000000000000 l:0000010000000000 o:1201254f0 t:0
C26: v:1 s:0 p:7fff81ff b:0000000000000000 l:0000010000000000 o:0 t:0
C31: v:1 s:0 p:7fff81d7 b:0000000000000000 l:0000010000000000 o:12002643c t:0
Mar 14 17:44:05 beri1 kernel: USER_CHERI_EXCEPTION: pid 948 tid 100045 (date), uid 999: CP2 fault (type 0x32)
Mar 14 17:44:05 beri1 kernel: Trapframe Register Dump:
Mar 14 17:44:05 beri1 kernel: zero: 0   at: 0x120052d38 v0: 0x12015b560 v1: 0x1c
Mar 14 17:44:05 beri1 kernel: a0: 0x7fffffd678  a1: 0x2 a2: 0x35        a3: 0x7fffffb540
Mar 14 17:44:05 beri1 kernel: a4: 0x7fffffbf58  a5: 0x7fffffbf50        a6: 0x7fffffbf18       a6: 0x7fffffbf10
Mar 14 17:44:05 beri1 kernel: t0: 0x14  t1: 0x1e        t2: 0x7fffffbb00        t3: 0x7fffffba80
Mar 14 17:44:05 beri1 kernel: t8: 0x2   t9: 0x1200263d0 s0: 0x400       s1: 0x20
Mar 14 17:44:05 beri1 kernel: s2: 0x20  s3: 0   s4: 0x8 s5: 0
Mar 14 17:44:05 beri1 kernel: s6: 0     s7: 0xffffffffffffffff  k0: 0   k1: 0
Mar 14 17:44:05 beri1 kernel: gp: 0x12017d840   sp: 0x7fffffd5e0        s8: 0x7fffffd5e0       ra: 0x7fffffd7f8
Mar 14 17:44:05 beri1 kernel: sr: 0x4000fcf3    mullo: 0x6666666666666667       mulhi: 0       badvaddr: 0x1200263d0
Mar 14 17:44:05 beri1 kernel: cause: 0x148      pc: 0x12002643c

Code dump from printf() in /bin/date -- crash occurs in the CJALR at 0x12002643c:

00000001200263d0 <printf>:
   1200263d0:   67bdff60        daddiu  sp,sp,-160
   1200263d4:   67a40098        daddiu  a0,sp,152
   1200263d8:   ebcb2003        csd     s8,a0,0(c11)
   1200263dc:   fa2be860        csc     c17,sp,96(c11)
   1200263e0:   03a0f02d        move    s8,sp
   1200263e4:   49b96002        cgetoffset      t9,c12
   1200263e8:   3c010015        lui     at,0x15
   1200263ec:   0039082d        daddu   at,at,t9
   1200263f0:   64217470        daddiu  at,at,29808
   1200263f4:   67c20020        daddiu  v0,s8,32
   1200263f8:   49a15881        csetoffset      c1,c11,v0
   1200263fc:   64020020        daddiu  v0,zero,32
   120026400:   48210880        csetbounds      c1,c1,v0
   120026404:   f9a10000        csc     c13,zero,0(c1)
   120026408:   6422afc0        daddiu  v0,at,-20544
   12002640c:   48810087        cfromptr        c1,c0,v0
   120026410:   c8410003        cld     v0,zero,0(c1)
   120026414:   64218238        daddiu  at,at,-32200
   120026418:   48810047        cfromptr        c1,c0,at
   12002641c:   c8210003        cld     at,zero,0(c1)
   120026420:   48810087        cfromptr        c1,c0,v0
   120026424:   d8210000        clc     c1,zero,0(c1)  
   120026428:   49a21800        cmove   c2,c3
   12002642c:   48001807        cgetpcc c3
   120026430:   49ac1841        csetoffset      c12,c3,at
   120026434:   49a30800        cmove   c3,c1
   120026438:   49a41000        cmove   c4,c2
   12002643c:   48f16000        cjalr   c17,c12                <--- Crash here
   120026440:   49a56800        cmove   c5,c13
   120026444:   03c0e82d        move    sp,s8
   120026448:   da2be860        clc     c17,sp,96(c11)
   12002644c:   67a10098        daddiu  at,sp,152
   120026450:   cbcb0803        cld     s8,at,0(c11)
   120026454:   49008800        cjr     c17
   120026458:   67bd00a0        daddiu  sp,sp,160
   12002645c:   00000000        nop

The execute bit in $c12 is indeed not set:

C12: v:1 s:0 p:7fff81fd b:000000012015b680 l:0000000000000900 o:120052d38 t:0

The value in $c12 is a little surprising, given both a base and an offset that resemble virtual addresses. Presumably only one of these should resemble a virtual address. This looks a bit like it might be a processor bug, as $c3 is loaded from $pcc only a few instructions earlier, and although CSetOffset is now an instruction of ill repute, this is 256-bit CHERI and so this should in fact be fine. A second set of eyes on the code generation here would be useful; I think @davidchisnall or @michael-roe had reported seeing incorrect values come out of $pcc in recent testing?

DTrace CCall/CReturn support

We can now KTrace CCall/CReturn events per #17. We should add Dtrace probes for these events as well.

No `va_copy` coverage in cheritest

It looks as if va_copy has always been broken in the sandbox ABI, but nothing in the test suite has detected this.

intptr_t is not intcap_t in CheriABI mode

#include <stdint.h>

_Static_assert(sizeof(uintptr_t) == sizeof(void*), "uintptr_t size is wrong");
_Static_assert(sizeof(intptr_t) == sizeof(void*), "intptr_t size is wrong");

These static assertions fail and this is currently breaking some of the tests in Peter Sewell's C test suite.

KTrace CCall/CReturn support

It would be very useful if the kernel's ktrace(1) facility could be used to track CHERI object-capability invocation/return. This would require dropping out of the CCall/CReturn fast paths to insert log entries describing the transition -- e.g., object code and data pointers, method number, and perhaps other argument information. This might be done via a software-defined exception -- but it will be a bit tricky to unwind "this is a slow path" from "this is an exception, do signal delivery".

Stack-allocation bounds-check tests in capability ABI / sandbox mode

Currently, cheritest checks that stack allocations have suitable bounds outside of a compartment. It should also run these tests within a compartment.

Rename CheriBSD's cheritest to cheribsdtest

The name "cheritest" is redundant to the CHERI processor's own test suite, which frequently leads to confusion. Renaming cheritest to cheribsdtest would avoid that confusion.

Complete migration of cheritest from explicit cinvoke to compiler-provided stubs

Currently, cheritest continues to use libcheri's cinvoke functions explicitly. This is unlikely to be a much used path in the future -- and perhaps should be removed -- so we should update the test suite to use compiler-provided stubs instead.

nsswitch problems with CheriABI

I've noticed that a number of CheriABI programs are having problems looking up user and group information, suggesting a problem in the nsswitch mechanism. For example, id(1) doesn't list usernames/groups:

> ./id
uid=999 gid=999 groups=999,0
> id
uid=999(ctsrd) gid=999(ctsrd) groups=999(ctsrd),0(wheel)

And the finger(1) command also can't look up users:

> ./finger ctsrd
finger: ctsrd: no such user

Develop a CHERI-aware garbage-collector extension to jemalloc

Tagged memory can support reliable C-language garbage collection. This issue is for the development of garbage-collection extensions to jemalloc, which allows us to defer reuse of memory by the allocator when delegated to sandboxes, to prevent use-after-free attacks. This requires developing a garbage collector, kernel APIs to find pages in the virtual address space that might contain tagged capabilities, test cases, etc.

1MB malloc crashes with a tag violation

The following simple program crashes with a tag violation with CHERIABI:

#include <stdlib.h>
volatile void *a;
int main(void)
{
    a = malloc(1048576);
}

CheriABI: sucap(9) does not take a permissions argument

Currently, the kernel's sucap(9) function accepts most capability fields, but not a permissions field. As we begin to divorce executable capabilities (r+x) and data capabilities (r+w), we will need to set permissions more specifically here. But will we always know what permissions are required?

mechanism to restrict method calls in a system object

The system class provides many different methods. It was possible to restrict the methods that could be called in a given system object using sandbox_object_new_flags(), but the underlying checks used to enforce those flags were removed in 8761124 when cheri_system_enter() was removed. A new mechanism needs to be designed and implemented.

Change trusted-stack APIs and ABIs to support configurable / variable-length stacks

Currently, the sysarch() system calls used by userspace to inspect and modify kernel-maintained trusted stacks embed a notion of a fixed-size trusted stack. Once trusted stacks are allowed to grow (see #10), we will want userspace to be able to behave more flexibly -- and to be able to change the trusted stack size at runtime if needed.

Linked to #10 -- arguably, dependent on it.

no kernel support for ps_strings

The CheriABI implementation is missing support for retrieving cheri_ps_strings and converting it to a normal ps_strings structure as required. See #ifdef COMPAT_FREEBSD32 in sys/kern/kern_proc.c

Signal delivery is broken in cheriabi

sigaction takes a struct that contains a code capability. Cheriabi doesn't appear to add a cheri_ compat version to extract the code capability. We probably also need some changes to the signal delivery code to install a code capability, as well as a code pointer, though I believe that we have some of this currently to allow signals to run outside of a sandbox when raised in the sandbox.

CheriBSD needs a portable way to report CHERI exceptions

Currently, we add SIGPROT but require use of a non-portable field in the machine-dependent signal context to resolve different failure modes (e.g., untagged pointer, bounds check, and permissions error) from one another. In due course, it would be desirable to offer a portable API to query the lower-level code information. My leaning is towards portable extension to sigcontext, but I could also yet be convinced that more signals would be appropriate (although this would increase the burden of maintaining additional signals in CheriBSD).

sig_atomic_t in CheriABI

In trying to compile csh(1), I found that it depended on a definition of sig_atomic_t, which appears not to be defined in CheriABI. @davidchisnall thinks it might want to be a 64-bit integer type, as it is an integer type, whereas I take the view that we would like software that uses sig_atomic_t to hold pointers to work. Which raises the pragmatic question: how much software expects sig_atomic_t to be able to contain a pointer? If any non-trivial number, then perhaps it should be intcap_t?