Golang httpserver app crashes occasionally on startup about osv

[nyh]

Very nice research work, I think you got both the problem and the solution.
Indeed in the past in many different contexts (both while I was working on nest KVM and while I was working on the OSv kernel), seeing random segfaults in an already-debugged application was a telltale sign of forgotten FPU saving. In the nested KVM example, I used to run long compilations in the VM knowing that compilations used a lot of floating point operations (not for actual floating point - more like the SSE vectorized string operations you mentioned) and would often crash because of FPU bugs.

In syscall() itself we already use sched::fpu_lock to save the FPU state. You are right that it was a mistake that we call setup_large_syscall_stack() before syscall() but don't save the FPU state. Like you said, given that setup_large_syscall_stack() (and free_tiny_syscall_stack()) are only called once per thread (which actually uses syscall()), I agree that adding fpu::lock to these functions would be a good and easy solution with negligible overhead.

It's good that you noticed that if we use fpu::lock this will require a bigger "tiny stack" (maybe it's worth adding the "canary" code you added to help catch bugs in the future?). There can be a solution for that, but I'm not sure if it's worth the hassle of writing more code if just increasing the 1K "tiny stack" to 2K is enough. The solution I think we can do to keep the "tiny stack" just 1K (or even substantially less) is to use a per-CPU "temporary stack" just for running setup_large_syscall_stack() . We already have exactly such a stack allocated, the arch_cpu::percpu_exception_stack but I'm not sure we can or if it's wise to reuse it (can we have an exception while setup_large_syscall_stack() is running?) but if not we can have another one like it (you don't have to put it in arch-cpu.hh, you can use generic per-cpu support from osv/percpu.hh. Then we can switch to that temporary stack, save the FPU state on it and allocate the new stack.

[wkozaczuk]

Yes, I thought about this per-cpu stack solution at some point. But I am struggling to understand how we would use it when handling setup_large_syscall_stack(). Both this and free_tiny_syscall_stack() may sleep (because the 1st one calls malloc() and the 2nd one calls free()) so they may be preempted to another thread also in the middle of the setup of a large syscall stack on the same CPU. Two threads cannot use the same stack, can they? Disable preemption during setup_large_syscall_stack()? But is it allowed during malloc and free?