UART not working when compiling in debug. about rust-raspberrypi-os-tutorials

[raw-bin]

To add to the above: I find that if a Rust function is decorated with the '#[naked]' decorator then no stack pointer references are made in that function's prologue and epilogue. This is as expected. However, this is only true in release builds. Debug builds seem to ignore the decorator. I do not know whether this is by design or whether it is a bug. I do mean to raise a ticket to clarify with the Rust lang folks. In summary - if you intend using naked functions then don't use debug builds else bad things may happen if the stack isn't accessible.

…

On Mon, Feb 10, 2020 at 8:52 PM Richard A. Healy ***@***.***> wrote: 0000000000080000 _start: 80000: sub sp, sp, #0x20 80004: str x30, [sp, #0x10] As @andre-richter <https://github.com/andre-richter> speculated looks like stack issues. So here's how it works - behind the scenes asm is added to a C or Rust function which stores and restores state information on the stack (all the stack pointer (sp) references in the assembly) before and after the function has executed. If the stack hasn't been set up in the cpu hardware bad things happen. In some cases (like above) we don't want the compiler to output the prologue/epilogue asm code for a function. This is called a naked function. In rust there's a lot of talk about naked functions and an rfc: https://github.com/rust-lang/rfcs/blob/master/text/1201-naked-fns.md but after wading through all the discussion I'm not sure how stable it is or how it's supposed to work. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#45>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFMKYQZAZCAYPW6GJOWMMDRCG5AHANCNFSM4KSJX2FA> .

[andre-richter]

@vlabo @rahealy to conclude this issue, I found that #[naked] does the needful in this case and removes the inital stack pointer subtraction. Using it, together with setting the stack pointer upfront will unblock you. Here's a quick hack:

diff --git a/12_cpu_exceptions_part1/src/arch/aarch64.rs b/12_cpu_exceptions_part1/src/arch/aarch64.rs
index 7e85676..a8ea889 100644
--- a/12_cpu_exceptions_part1/src/arch/aarch64.rs
+++ b/12_cpu_exceptions_part1/src/arch/aarch64.rs
@@ -20,9 +20,12 @@ use cortex_a::{asm, regs::*};
 ///
 /// - Linker script must ensure to place this function at `0x80_000`.
 #[no_mangle]
+#[naked]
 pub unsafe extern "C" fn _start() -> ! {
     const CORE_MASK: u64 = 0x3;
 
+    asm!("mov sp, #0x8000");
+
     // Expect the boot core to start in EL2.
     if (bsp::BOOT_CORE_ID == MPIDR_EL1.get() & CORE_MASK)
         && (CurrentEL.get() == CurrentEL::EL::EL2.value)

While this works, I am not planning on adding it to the master tutorials for now.

Best,
Andre

[andre-richter]

My guess without checking: Without release mode, some of the early boot code written in Rust, like _start(), would probably miss inlining of some helper function calls.
Since the code doesn't check if the EL2 stack pointer is sane, its maybe going wrong there.

If you want to dig yourself, use make objdump and have a look at those early functions.

[rahealy]

0000000000080000 _start:
   80000:      	sub	sp, sp, #0x20
   80004:      	str	x30, [sp, #0x10]

As @andre-richter speculated looks like stack issues.

So here's how it works - behind the scenes asm is added to a C or Rust function which stores and restores state information on the stack (all the stack pointer [sp] references in the assembly) before and after the function has executed. If the stack hasn't been set up in the cpu hardware bad things happen.

In some cases (like above) we don't want the compiler to output the prologue/epilogue asm code for a function. This is called a naked function. In rust there's a lot of talk about naked functions and an rfc:

https://github.com/rust-lang/rfcs/blob/master/text/1201-naked-fns.md

but after wading through all the discussion I'm not sure how stable it is or how it's supposed to work.

[andre-richter]

I tried with tutorial 12. This is the assembly I get viewed with objdump:

0000000000080000 _start:
   80000:       sub     sp, sp, #0x40
   80004:       str     x30, [sp, #0x30]
   80008:       adrp    x0, #0x11000
   8000c:       add     x0, x0, #0xf60
   80010:       adrp    x1, #0x5000
   80014:       add     x1, x1, #0x79c
   80018:       str     x1, [sp, #0x20]
   8001c:       bl      #0x90d8 <<cortex_a::regs::mpidr_el1::Reg as register::cpu::RegisterReadOnly<u64,()>>::get::h8aff23850bcdec24>

Running QEMU with -d in_asm shows that it hangs here:

----------------
IN:
0x00080000:  d10103ff  sub      sp, sp, #0x40
0x00080004:  f9001bfe  str      x30, [sp, #0x30]
0x00080008:  b0000080  adrp     x0, #0x91000
0x0008000c:  913d8000  add      x0, x0, #0xf60
0x00080010:  b0000021  adrp     x1, #0x85000
0x00080014:  911e7021  add      x1, x1, #0x79c
0x00080018:  f90013e1  str      x1, [sp, #0x20]
0x0008001c:  94002436  bl       #0x890f4

I never see qemu executing the next block at 0x890f4. Need to take a look why that is.

[andre-richter]

Thanks @rahealy, good to have a more fine-grained solution at hand!

Also, this issue makes me wonder what the default SP_EL2 value is on the real HW.
I need to check that with JTAG sometime.

[vlabo]

I compared the two kernels. I'm not really good at reading arm assembly but accessing the registers with the cortex_a functions is very different in debug and release.

release:

0000000000080000 _start:
   80000:      	mrs	x8, MPIDR_EL1
   80004:      	tst	x8, #0x3
   80008:      	b.eq	#0xc <_start+0x14>
   8000c:      	wfe
   80010:      	b	#-0x4 <_start+0xc>
   80014:      	mov	w8, #0x80000
   80018:      	mov	sp, x8
   8001c:      	bl	#0x34c <rust_pi_os::kernel_init::hcb7d7cff2860b865>
   80020:      	brk	#0x1

debug:

0000000000080000 _start:
   80000:      	sub	sp, sp, #0x20
   80004:      	str	x30, [sp, #0x10]
   80008:      	adrp	x0, #0x2000
   8000c:      	ldr	x0, [x0, #0xdf8]
   80010:      	bl	#0x124 <<cortex_a::regs::mpidr_el1::Reg as register::cpu::RegisterReadOnly<u64,()>>::get::h0f7f9659690def2c>
   80014:      	str	x0, [sp, #0x8]
   80018:      	mov	x8, #0x3
   8001c:      	ldr	x9, [sp, #0x8]
   80020:      	and	x8, x9, x8
   80024:      	cbz	x8, #0xc <_start+0x30>
   80028:      	wfe
   8002c:      	b	#-0x4 <_start+0x28>
   80030:      	adrp	x0, #0x2000
   80034:      	ldr	x0, [x0, #0xe00]
   80038:      	mov	x1, #0x80000
   8003c:      	bl	#0x79c <<cortex_a::regs::sp::Reg as register::cpu::RegisterReadWrite<u64,()>>::set::he1177b4360cd6a20>
   80040:      	bl	#0xc04 <rust_pi_os::kernel_init::h4878867f46b1f710>

original:

#[no_mangle]
pub unsafe extern "C" fn _start() -> ! {
    const CORE_MASK: u64 = 0x3;

    if BOOT_CORE_ID == MPIDR_EL1.get() & CORE_MASK {
        SP.set(BOOT_CORE_STACK_START);
        kernel_init()
    } else {
        // If not core0, infinitely wait for events.
        wait_forever()
    }
}

[vlabo]

Last night I was playing with the code and I managed to make the compiler inline the calls when compiling in debug. I replace #[inline] with #[inline(always)] on get and set functions in __read_raw and __write_raw macros in the cortex-a crate. I'm not sure if this helps.

debug dump:

0000000000080000 _start:
   80000:      	sub	sp, sp, #0x90
   80004:      	str	x30, [sp, #0x80]
   80008:      	adrp	x8, #0x12000
   8000c:      	add	x8, x8, #0x880
   80010:      	str	x8, [sp, #0x30]
   80014:      	mrs	x8, MPIDR_EL1
   80018:      	str	x8, [sp, #0x38]
   8001c:      	ldr	x8, [sp, #0x38]
   80020:      	str	x8, [sp, #0x20]
   80024:      	b	#0x48 <_start+0x6c>

It still hangs when running.

[rahealy]

As mentioned before, unless you're setting the stack pointer before _start the code might be accessing memory that doesn't exist and the cpu is throwing an exception.

Here's an example compiled from tutorial 12. I don't know why It looks a bit different from the compiler output you got @andre-richter but it likely applies:

0000000000080000 _start:
   80000:      	sub	sp, sp, #0x50
   80004:      	str	x30, [sp, #0x40]
   80008:      	adrp	x0, #0x13000
   8000c:      	add	x0, x0, #0xb8c
   80010:      	adrp	x8, #0x11000
   80014:      	add	x8, x8, #0x838
   80018:      	adrp	x1, #0x4000
   8001c:      	add	x1, x1, #0xbf8
   80020:      	str	x8, [sp, #0x30]
   80024:      	str	x1, [sp, #0x28]
   80028:      	bl	#0x7288 <<cortex_a::regs::mpidr_el1::Reg as register::cpu::RegisterReadOnly<u64,()>>::get::h626ce4daed827ed5>
   8002c:      	str	x0, [sp, #0x20]
   80030:      	b	#0x3c <_start+0x6c>

I use gdb-multiarch connected to qemu via tcp. This might look different from the tutorial's recommend toolchain.

Before we begin look at the stack pointer value. It is 0x0.

Breakpoint 1 at 0x80000
(gdb) c
Continuing.
Thread 1 hit Breakpoint 1, 0x0000000000080000 in ?? ()
(gdb) info reg sp
sp             0x0                 0x0

First instruction says subtract 0x50 from stack pointer:

80000: sub sp, sp, #0x50

Execute the instruction and look at stack pointer:

(gdb) si
0x0000000000080004 in ?? ()
(gdb) info reg sp
sp             0xffffffffffffffb0  0xffffffffffffffb0

RPi doesn't have 0xffffffffffffffb0 (1.84467440737e+19) bytes of memory but we'll continue anyway.

Second instruction says to take the value at register x30 and store it at the memory address pointed at by the stack pointer + offset 0x40. Remember the stack pointer is pointing at memory that doesn't exist:

80004: str x30, [sp, #0x40]

Execute the instruction. Code locks up. Ctrl+C to interrupt execution and see what happened:

(gdb) si

Thread 1 received signal SIGINT, Interrupt.
0x0000000000000200 in ?? ()

When we tried to access memory that doesn't exist the CPU raised an exception and jumped to exception handler code. Since we haven't told the CPU where to find the exception handler code yet it jumped to 0x200 which contains 0x0. 0x0 is not a valid ARM instruction and the code is locked :(

0x200 .inst 0x00000000 ; undefined

[vlabo]

Thank you for the explanation. Now it look obvious when I think about it.

I opened the issue because I spend several hours trying to find why my make file was not working and the one from the repository was before releasing that I need to compile with release flag.
I was interested to find out why and now I know.

Feel free to close the issue if you think it's not relevant to the project.

[andre-richter]

Thanks a lot @rahealy, very insightful!

Until now, I thought that the -d in_asm output of QEMU would only show the last instruction before "something" happens, therefore I suspected the first branch instruction would be the culprit.

That the EL2 stack pointer in QEMU is initially zero comes a bit surprising to me, although I never thought about what the "code" that runs before the user's binary actually does. I would have expected it to get at least SOME best-guess default stack size, so that you can operate with branches in EL2. Maybe there is a config option somewhere.

[rahealy]

@andre-richter I can't speak to the -d in_asm flag but I can show you how I set up and debug on my machine.

See the following changes to the makefile here

QEMU
I add -s -S to qemu invocation. Per qemu manpage:

-s Shorthand for -gdb tcp::1234, i.e. open a gdbserver on TCP port 1234.
-S Do not start CPU at startup (you must type 'c' in the monitor)

Docker
I add -p 1234:1234 in docker invocation to open port that gdb-multiarch can connect to.

GDB
To save typing I use a config file (gdbcommands.txt) w/gdb here

Workflow
In one console:

~/rust-raspi3-OS-tutorials/12_cpu_exceptions_part1$ make all objdump > list.lst ; make qemu
...compiler output...

In a different console:

~/rust-raspi3-OS-tutorials/12_cpu_exceptions_part1$ gdb-multiarch -x ./gdbcommands.txt

Everyone has their own way. This is just how I got things to work using what I knew at the time and have stuck with it. :)