unikraft / lib-musl

musl: A C standard library

Musl

Musl is a standard C library (libc) providing POSIX support on top of the Linux system call API. These are support files for Musl support in Unikraft.

Musl is the default and recommended libc for Unikraft. newlib is another libc available for Unikraft. Musl provides thread support natively, while newlib requires pthread-embedded; Musl doesn't require (and must not be built with) pthread-embedded.

Note that lwip is required for networking support.

Using Musl with Unikraft

Musl is to be used as the standard C library in Unikraft applications, such as app-nginx. The application build system must be aware of the location of the local clone of the Musl repository. When Musl is selected the internal Unikraft nolibc library will be unselected.

Remarks

The default configuration doesn't include complex number support, i.e. the LIBMUSL_COMPLEX option is disabled (see Config.uk); see commit 45c4aa58. This is because, in case of using Clang to build Musl, the symbol _muldc3 (used for complex numbers) is undefined; in case of using Clang and requiring complex number support, compiler-rt is to be included to the build.

lib-musl's Issues

multiple definition of `timer_create'

When adding musl to kraft.yaml libraries, selecting it in menuconfig and compiling, the following error occurs:

2021-03-14 00:43:01 INFO     LD      libmuslglue.ld.o
2021-03-14 00:43:01 INFO     OBJCOPY libmuslglue.o
2021-03-14 00:43:01 INFO     LD      libmusl.ld.o
2021-03-14 00:43:02 INFO     OBJCOPY libmusl.o
2021-03-14 00:43:03 INFO     LD      app-gats_kvm-x86_64.ld.o
/usr/bin/ld: /home/jan/Desktop/gats2/apps/app-gats/build/libmusl.o: in function `timer_create':
/home/jan/Desktop/gats2/apps/app-gats/build/libmusl/origin/musl-1.1.19//src/time/timer_create.c:82: multiple definition of `timer_create'; /home/jan/Desktop/gats2/apps/app-gats/build/libuktime.o:/home/jan/.unikraft/unikraft/lib/uktime/timer.c:42: first defined here
/usr/bin/ld: /home/jan/Desktop/gats2/apps/app-gats/build/libmusl.o: in function `timer_delete':
/home/jan/Desktop/gats2/apps/app-gats/build/libmusl/origin/musl-1.1.19//src/time/timer_delete.c:6: multiple definition of `timer_delete'; /home/jan/Desktop/gats2/apps/app-gats/build/libuktime.o:/home/jan/.unikraft/unikraft/lib/uktime/timer.c:49: first defined here
/usr/bin/ld: /home/jan/Desktop/gats2/apps/app-gats/build/libmusl.o: in function `timer_gettime':
/home/jan/Desktop/gats2/apps/app-gats/build/libmusl/origin/musl-1.1.19//src/time/timer_gettime.c:6: multiple definition of `timer_gettime'; /home/jan/Desktop/gats2/apps/app-gats/build/libuktime.o:/home/jan/.unikraft/unikraft/lib/uktime/timer.c:67: first defined here
/usr/bin/ld: /home/jan/Desktop/gats2/apps/app-gats/build/libmusl.o: in function `scandir':
/home/jan/Desktop/gats2/apps/app-gats/build/libmusl/origin/musl-1.1.19//src/dirent/scandir.c:12: multiple definition of `scandir'; /home/jan/Desktop/gats2/apps/app-gats/build/libvfscore.o:/home/jan/.unikraft/unikraft/lib/vfscore/main.c:908: first defined here
/usr/bin/ld: /home/jan/Desktop/gats2/apps/app-gats/build/libmusl.o: in function `timer_getoverrun':
/home/jan/Desktop/gats2/apps/app-gats/build/libmusl/origin/musl-1.1.19//src/time/timer_getoverrun.c:6: multiple definition of `timer_getoverrun'; /home/jan/Desktop/gats2/apps/app-gats/build/libuktime.o:/home/jan/.unikraft/unikraft/lib/uktime/timer.c:74: first defined here
/usr/bin/ld: /home/jan/Desktop/gats2/apps/app-gats/build/libmusl.o: in function `timer_settime':
/home/jan/Desktop/gats2/apps/app-gats/build/libmusl/origin/musl-1.1.19//src/time/timer_settime.c:6: multiple definition of `timer_settime'; /home/jan/Desktop/gats2/apps/app-gats/build/libuktime.o:/home/jan/.unikraft/unikraft/lib/uktime/timer.c:59: first defined here
collect2: error: ld returned 1 exit status
make[2]: *** [/home/jan/.unikraft/unikraft/plat/kvm/Linker.uk:22: /home/jan/Desktop/gats2/apps/app-gats/build/app-gats_kvm-x86_64.dbg] Error 1
make[1]: *** [Makefile:984: sub-make] Error 2
make: *** [Makefile:32: _all] Error 2
2021-03-14 00:43:03 INFO     make: Leaving directory '/home/jan/.unikraft/unikraft'

Ignoring signals fails

On Linux, when the Rust runtime initializes, it resets the SIGPIPE handler to ignore. (std/src/sys/unix/mod.rs#L59-L67, std/src/sys/unix/mod.rs#L164-L195):

    // By default, some platforms will send a *signal* when an EPIPE error
    // would otherwise be delivered. This runtime doesn't install a SIGPIPE
    // handler, causing it to kill the program, which isn't exactly what we
    // want!
    //
    // Hence, we set SIGPIPE to ignore when the program starts up in order
    // to prevent this problem. Add `#[unix_sigpipe = "..."]` above `fn main()` to
    // alter this behavior.

What Rust does is equivalent to the following C program:

#include <assert.h>
#include <signal.h>

int main(int argc, char *argv[]) {
    void (*res)(int) = signal(SIGPIPE, SIG_IGN);
    assert(res != SIG_ERR);

    return 0;
}

• On Linux, this succeeds.
• On Unikraft, with uksignal, this succeeds.
• On Unikraft, with musl, this fails.

[...]
Assertion failed: res != SIG_ERR (/home/kroening/devel/unikraft/apps/app-helloworld/main.c: main: 6)
[    0.105735] CRIT: [libmusl] <abort.c @    7> abort called

Returning a success value without doing anything would resolve this problem.

Proposal: Convert all patches to mail-formatted patches

Editing or updating patches in the patch directory is unnecessarily hard, especially if a patch somewhere in the middle should to be added or updated.

It would be much easier if the patches would be mail-formatted (some of them already are).

That way, you could:

1. checkout the appropriate musl version ( currently v1.1.19)
2. apply the patches to the musl git repository (git am)
3. make the changes to the git history
4. export the new versions with git format-patch v1.1.19

Currently, a similar workflow is possible by manually applying non-mail-formatted patches with git apply and then committing each of these changes. However, converting back to the original patch format is unnecessarily hard.

Other points to consider:

• Applying the patches into a git repository is also a useful tool for reviewing the patches.
• The mail-formatted patches are already handled well by the Unikraft build system.

undefined reference to `__muldc3`

When using lib-musl, the following link-time error occurs:

 E  /usr/bin/ld: warning: /github/workspace/_helloworld/.unikraft/build/libmusl/setjmp.x86_64.o: missing .note.GNU-stack section implies executable stack
 E  /usr/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
 i    OBJCOPY libmusl.o
 i    LD      helloworld_qemu-x86_64.dbg
 E  /usr/bin/ld: warning: /github/workspace/_helloworld/.unikraft/build/libkvmplat.o: requires executable stack (because the .note.GNU-stack section is executable)
 E  /usr/bin/ld: /github/workspace/_helloworld/.unikraft/build/libmusl.o: in function `cpow':
 E  /github/workspace/_helloworld/.unikraft/build/libmusl/origin/musl-1.2.3//src/complex/cpow.c:7: undefined reference to `__muldc3'
 E  /usr/bin/ld: /github/workspace/_helloworld/.unikraft/build/libmusl.o: in function `cpowf':
 E  /github/workspace/_helloworld/.unikraft/build/libmusl/origin/musl-1.2.3//src/complex/cpowf.c:5: undefined reference to `__mulsc3'
 E  /usr/bin/ld: /github/workspace/_helloworld/.unikraft/build/libmusl.o: in function `cpowl':
 E  /github/workspace/_helloworld/.unikraft/build/libmusl/origin/musl-1.2.3//src/complex/cpowl.c:11: undefined reference to `__mulxc3'
 E  collect2: error: ld returned 1 exit status
 E  make[1]: *** [/github/workspace/plat/kvm/Linker.uk:48: /github/workspace/_helloworld/.unikraft/build/helloworld_qemu-x86_64.dbg] Error 1
 E  make: *** [Makefile:1055: sub-make] Error 2

This issue was introduced in #64.

Either LIBMUSL_COMPLEX needs to be unselected or lib-compiler-rt must be added to the build.

Makefile.uk: `-Wno-cast-function-type` GCC option only available for GCC >= 8

When building Musl with GCC <= 7, the following warning appears:

cc1: warning: unrecognized command line option ‘-Wno-cast-function-type’

It's caused by line 125 in Makefile.uk.

The line should be updated to only be available when GCC >= 8 is used.

Fix `unknown option` warnings when building with `clang`

When building musl with clang, a lot of unknown option warnings get generated:

warning: unknown warning option '-Wno-format-contains-nul' [-Wunknown-warning-option]
warning: unknown warning option '-Wno-restrict' [-Wunknown-warning-option]
warning: unknown warning option '-Wno-unused-but-set-variable'; did you mean '-Wno-unused-const-variable'? [-Wunknown-warning-option]
warning: unknown warning option '-Wno-maybe-uninitialized'; did you mean '-Wno-uninitialized'? [-Wunknown-warning-option]

Change the Makefile.uk file to add the flags only if the compiler used is gcc.

Hackathon points: 30

Missing implementation for `stdatomic`

Musl currently does not support the C11 standard for atomic variables, and as far as I know, it will never support it.

Unfortunately, plenty of applications depend on the respective header files to compile successfully.

As such, to avoid problems, one thing we could do is to implement stdatomic ourselves using gcc calls, as Unikraft is currently compiled with mostly gcc.

`poll` returns `ENOSYS`

On Linux, when the Rust runtime initializes, Rust sanitizes the standard streams: (std/src/sys/unix/mod.rs#L54-L57)

    // The standard streams might be closed on application startup. To prevent
    // std::io::{stdin, stdout,stderr} objects from using other unrelated file
    // resources opened later, we reopen standards streams when they are closed.

Rust does so, by first polling on the file descriptors. On EINVAL, EAGAIN, and ENOMEM, Rust falls back from using poll (std/src/sys/unix/mod.rs#L108-L118).

On Unikraft, with musl, this returns ENOSYS, though, resulting in an abort.

C code for reproduction:

#include <assert.h>
#include <errno.h>
#include <poll.h>
#include <stddef.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    if (poll(NULL, 0, 0) == -1) {
        printf("%i\n", errno == ENOSYS);
        assert(errno == EINVAL || errno == EAGAIN || errno == ENOMEM);
    }

    return 0;
}

prints

[...]
1
Assertion failed: errno == EINVAL || errno == EAGAIN || errno == ENOMEM (/home/kroening/devel/unikraft/apps/app-helloworld/main.c: main: 10)
[    0.106567] CRIT: [libmusl] <abort.c @    7> abort called

We could approach this issue from both Unikraft and Rust. We could either

1. Make poll on Unikraft return any of EINVAL, EAGAIN, or ENOMEM or
2. make Rust also fall back from using poll on ENOSYS.

I personally think, 2 would be the better option, though that somewhat conflicts with the idea that we should not have to adjust programs for running on Unikraft and instead fix the issue on our side.

What do you think?