nuta / resea Goto Github PK
View Code? Open in Web Editor NEWA microkernel-based hackable operating system.
License: Other
A microkernel-based hackable operating system.
License: Other
Resea lacks an advantage of modern microkernels: reliability. Improving reliability is the main goal for 2020. This issue tracks tasks related to reliability improvements.
i run the system in virtio legacy mode with export VIRTIO_LEGACY=y
command. Should we add a build option for this?
or we can run this make run VIRTIO_LEGACY=y
Most servers (except device drivers) depends only on the message passing and some system calls. I think it would be great if we could have a unit testing framework which provides:
libs/resea
emulation without running the kernel. It allows using your favorite Linux/macOS debugging tools!This is the last piece for the policy-free kernel: allow userspace programs to decide how tasks are scheduled. The following design is based on MINIX3.
reset
. If reset
is -1, it sends an exception message to its pager task.sched
system call.
error_t sys_sched(task_t task, int priority, int quantum, int reset, unsigned affinity);
sys_spawn
: Add TASK_SCHED
flag not to start the new task until sys_sched
is invoked.
gpu_device
interfacenamespace gpu_device {
rpc set_mode() -> (num_buffers: size);
rpc num_buffers() -> (num_buffers: size);
rpc get_buffer(index: size) -> (shm: handle);
rpc show_buffer(index: size) -> ();
}
#include <ui.h>
ui_text_t button_text;
void button_clicked(ui_event_t ev, ui_button_t button) {
ui_button_set_text(button_text, "Clicked!");
}
void render_ui(void) {
ui_window_t win = ui_window();
ui_window_set_size(200, 300);
ui_window_set_title("My first application");
ui_canvas_t canvas = ui_window_get_canvas(win);
ui_text_t text = ui_text();
ui_text_set_body(text, "Hello World");
ui_text_set_size(text, UI_SIZE_H1);
ui_text_set_rgba(text, 255, 0, 0, 255);
ui_draw_text(canvas, text, 10, 10);
ui_button_t button = ui_button();
button_text = ui_text();
ui_text_set_body(button_text, "Click here");
ui_button_set_text(button_text);
ui_on_click(button_text , button_clicked);
ui_draw_button();
}
Window {
title: "My first application",
size: Size {
width: 200,
height: 300,
},
canvas: Canvas {
items: [
Text {
body: "Hello",
color: Rgba(0, 0, 0, 0),
handlers: { ... }
},
Button {
text: Text {
body: "Hello",
color: Rgba(0, 0, 0, 0),
handlers: { ... }
},
handlers: {
click: button_clicked,
]
}
]
}
}
Kernel Address Sanitizer (KASAN) is a runtime memory error (e.g. use-after-free) checker. While it is "kernel" address sanitizer, we can use it in the userspace.
Briefly speaking, when KASAN is enabled, the compiler inserts code to call hook functions (__asan_store8_noabort
) before each memory access (e.g. *ptr = 1;
). KASAN runtime (what we need to implement) is responsible for tracking how each memory bytes are valid.
You don't need to implement as described in the paper. Just use it as memory access hooks.
Add the following compiler options to $CFLAGS
:
--target=x86_64-pc-linux-elf
-fsanitize=undefined,kernel-address
-mllvm -asan-instrumentation-with-call-threshold=0
-mllvm -asan-globals=false
-mllvm -asan-stack=false
-mllvm -asan-use-after-return=false
-mllvm -asan-use-after-scope=false
Implement the KASAN runtime in libs/common
.
// Stores the current state of the each memory bytes.
uint8_t shadow[NUM_BYTES /* .bss size + .data size + heap size */];
#ifdef KERNEL
// We don't support KASan in kernel space for now.
void __asan_load8_noabort(vaddr_t addr) {
}
#else
void __asan_load8_noabort(vaddr_t addr) {
if (!shadow[addr]) {
PANIC("ASan: detected an invalid access to %p", addr);
}
}
#endif
Furthermore, you need to update shadow
in malloc
, free
, and functions written in assembly like memcpy
.
See Porting Guide
genidl.py
generates the HTML document for IPC messages on Resea. As you can see, we need further improvements on webpage design.
We'd like to run unmodified NetBSD device drivers and applications on Resea.
2019/12/18 23時ごろ、http://resea-web-server.seiya.me でデモを見せていただいたのですが、その際ルータの方で TCP FIN and no ACK の検知が走りました。
原因などは確認できていないのでこちらの環境の問題の可能性もありますが、ご報告だけさせていただきます。
Hello, I know this is not probably the best place to ask, but I started to learn some system programming and I have a kernel that actually just start and set the IDT.
I'm using gcc as cross compiler but I want to switch to clang/llvm I know that ofr its nature clang is a cross compiler so i dont need to recompile it.
The question is: what are the options to make clang to cross compile to x86(_64) bare metal?
Something like -target=i386-none ?
This is needed because obviously when you start to buld a kernel from scratch you are going to run a binary on the bare metal and you don't have any environment yet.
Thank you for the help.
There are too many allocs and frees for simple network operations. This could become a problem for performance.
[e1000] received 60 bytes
[tcpip] mbuf_alloc()
[tcpip] tcp: port=80, seq=00186a01, ack=00000000, len=0 [ SYN ]
[tcpip] tcp: new client (port=80)
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[e1000] sent 42 bytes
[e1000] received 64 bytes
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[e1000] sent 54 bytes
[e1000] received 60 bytes
[tcpip] mbuf_alloc()
[tcpip] tcp: port=80, seq=00186a02, ack=00000001, len=0 [ [e1000] received 195 bytes
ACK ]
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_alloc()
[tcpip] tcp: port=80, seq=00186a02, ack=00000001, len=141 [ ACK ]
[tcpip] tcp: received 141 bytes (seq=186a02)
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_alloc()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[tcpip] mbuf_delete_one()
[e1000] sent 54 bytes
[webapi] new data
While achieving a truly realtime operating system will be a tough job, we might be able to implement some improvements on realtime-ness. This issue tracks ideas and the progress on the topic.
memcpy_from_user/memcpy_to_user
).
Shared Memory API allows tasks to communicate each others without copying data. Combining the existing IPC APIs, it is a promising approach to copy large data (e.g. file contents) efficiently.
In Resea, we can implement shared memory in vm server (I believe we don't need any modifications to the kernel!). The vm should provide the following interface shm
:
# interface.idl
namespace shm {
rpc create(size: size) -> (shm_id: int);
rpc map(shm_id: int) -> (vaddr: vaddr);
}
Where create
allocates some physical memory pages for a new shared memory, and map
maps (by map_page()
) the specified shared memory (shm_id
) into an unused virtual address space (by alloc_virt_pages()
) and returns its address.
We'll need a new data structure to track existing shared memory entries:
struct shm {
bool in_use;
int shm_id;
paddr_t paddr;
size_t len;
};
#define NUM_SHARED_MEMS_MAX 32
struct shared_mems[NUM_SHARED_MEMS_MAX];
Add a test in resea/servers/apps/test
. The test would look like:
void shm_test(void) {
struct message m;
m.type = SHM_CREATE_MSG;
m.size = 256;
ASSERT_OK(ipc_call(INIT_TASK, &m));
int shm_id = m.shm_create_reply.shm_id;
struct message m;
m.type = SHM_MAP_MSG;
ASSERT_OK(ipc_call(INIT_TASK, &m));
// The mapped page should be writable, i.e. this `memset` should not cause a segfault...
memset((void *) m.shm_map_reply.vaddr, 0x5a, 256);
}
rand_bytes() call in servers/tcpip/main.c at line 186 and at line 214 crashes tcpip
Currently, the handle management library (libs/resea/handle.c) use the pair of a client task ID and task-local ID (handle_t
) as a handle, e.g., file handle in fs servers and TCP sockets in tcpip server. The problem is that, since it checks the client task ID to deny accessing other tasks' handles, user programs can't transfer their own handles to others.
This is the tracking issue for supporting handle type in IPC to allow handle transferring like SCM_RIGHTS
in Linux.
Slab Allocator is a dynamic memory allocation mechanism dedicated for commonly used objects.
Unlike Linux kernel, since we don't (and will never) have memory allocator in Resea Kernel, you'll implement it in the userspace based on the current malloc
.
dm
server (Device driver Management server)<resea/io.h>
)bus_space(9)
.Is it possible to run unmodified Linux device drivers on Resea?
I tested with qemu, approximately it uses %25 of CPU. i played with counts_per_tick
value but didn't have any stable state.
ARCH_X64:=y
#
# Build options
#
BUILD_DIR:=build
BUILD_DEBUG:=y
# CONFIG_BUILD_RELEASE is not set
LLVM_PREFIX:=
LLVM_SUFFIX:=
GRUB_PREFIX:=
# end of Build options
#
# Servers
#
BOOTSTRAP:=bootstrap
# CONFIG_BENCHMARK_SERVER is not set
DISPLAY_SERVER:=y
E1000_SERVER:=y
HELLO_SERVER:=y
# CONFIG_MINLIN_SERVER is not set
PS2KBD_SERVER:=y
RAMDISK_SERVER:=y
SHELL_SERVER:=y
TARFS_SERVER:=y
TCPIP_SERVER:=y
WEBAPI_SERVER:=y
# end of Servers
ARCH:=x64
SERVERS := tarfs display ps2kbd e1000 webapi shell ramdisk hello tcpip
with this config it goes forever(i thnk) when i ctrl+c it gives
root [ /opt/resea ]# LC_ALL=C make build
CC kernel/arch/x64/task.c
CC kernel/arch/x64/vm.c
CC kernel/arch/x64/serial.c
CC servers/tarfs/main.c
^Cmake[345]: *** [servers/tarfs/build.mk:5: build/user/servers/tarfs/tarball.o] Interrupt
make[344]: *** [servers/tarfs/build.mk:5: build/user/servers/tarfs/tarball.o] Interrupt
make[343]: *** [servers/tarfs/build.mk:5: build/user/servers/tarfs/tarball.o] Interrupt
make[342]: *** [servers/tarfs/build.mk:5: build/user/servers/tarfs/tarball.o] Interrupt
make[341]: *** [servers/tarfs/build.mk:5: build/user/servers/tarfs/tarball.o] Interrupt
Just crashes.
[ramdisk] ready
[ps2kbd] starting...
[shell] starting...
[appmgr] starting...
[webapi] starting...
[fatfs] starting...
[display] starting...
[e1000] starting...
[e1000] found a e1000 device (bus=0, slot=3, bar0=feb80000, irq=11)
[kernel] enabled IRQ: task=e1000, vector=11
[kernel] enabled IRQ: task=ps2kbd, vector=1
[ps2kbd] ready
[appmgr] ready
[display] ready
[fatfs] servers/fatfs/main.c:39 PANIC: failed to locate a FAT file system
[fatfs] WARN: Backtrace:
[fatfs] WARN: #0: 000000000100076e main()+0x58e
[e1000] initialized the device
[fatfs] WARN: #1: 000000000100942a start()+0x2e
[kernel] WARN: Exception #13[kernel] WARN: RIP = 000000000100942f CS = 000000000000002b RFL = 0000000000000206
[kernel] WARN: SS = 0000000000000023 RSP = 0000000003004048 RBP = 0000000003004120
[kernel] WARN: RAX = 000000000100942f RBX = 00000000fffffff5 RCX = 00000000010034e8
[kernel] WARN: RDX = 0000000000000000 RSI = 0000000000000000 RDI = 0000000000000000
[kernel] WARN: R8 = 0000000000000000 R9 = 0000000000000000 R10 = 0000000000000000
[kernel] WARN: R11 = 0000000000000246 R12 = 0000000000000000 R13 = 0000000000000000
[kernel] WARN: R14 = 000000000100be60 R15 = 0000000001004270 ERR = 0000000000000032
[e1000] MAC address = 52:54:00:12:34:56
[tcpip] starting...
[init] WARN: fatfs: exception occurred, killing the task...
[kernel] destroying fatfs...
[display] WARN: unknown message (type=1)
[appmgr] WARN: unknown message type (type=1)
[init] WARN: unknown message type (type=1)
[ramdisk] unknown message 1
The make command says
CC kernel/main.c
clang: error: unknown argument: '-fstack-size-section'
Makefile:116: recipe for target 'build/kernel/kernel/main.o' failed
make: *** [build/kernel/kernel/main.o] Error 1
After commenting out CFLAGS += -fstack-size-section
I get a lot of
warning: unknown warning option '-Werror=pointer-integer-compare'; did you mean '-Werror=string-compare'? [-Wunknown-warning-option]
1 warning generated.
The compilation stops with
LD build/user/kvs.debug.elf
SYMBOLS build/user/kvs.debug.elf
STRIP build/user/kvs.elf
make: llvm-objcopy: Command not found
Makefile:187: recipe for target 'build/user/kvs.elf' failed
make: *** [build/user/kvs.elf] Error 127
Fixed it by adding "-6.0" here
OBJCOPY := $(LLVM_PREFIX)llvm-objcopy-6.0$(LLVM_SUFFIX)
Then I get
LD build/user/init.debug.elf
SYMBOLS build/user/init.debug.elf
STRIP build/user/init.elf
OBJCOPY build/init.bin
llvm-objcopy-6.0: Unknown command line argument '-j.initfs'. Try: 'llvm-objcopy-6.0 -help'
llvm-objcopy-6.0: Did you mean '-stats'?
llvm-objcopy-6.0: Unknown command line argument '-j.text'. Try: 'llvm-objcopy-6.0 -help'
llvm-objcopy-6.0: Did you mean '-j'?
llvm-objcopy-6.0: Unknown command line argument '-j.data'. Try: 'llvm-objcopy-6.0 -help'
llvm-objcopy-6.0: Did you mean '-stats'?
llvm-objcopy-6.0: Unknown command line argument '-j.rodata'. Try: 'llvm-objcopy-6.0 -help'
llvm-objcopy-6.0: Did you mean '-mxgot'?
llvm-objcopy-6.0: Unknown command line argument '-j.bss'. Try: 'llvm-objcopy-6.0 -help'
llvm-objcopy-6.0: Did you mean '-j'?
llvm-objcopy-6.0: Unknown command line argument '-Obinary'. Try: 'llvm-objcopy-6.0 -help'
llvm-objcopy-6.0: Did you mean '-O'?
Makefile:107: recipe for target 'build/init.bin' failed
make: *** [build/init.bin] Error 1
After adding a space between all "-j" and "-O" make says
LD build/user/init.debug.elf
SYMBOLS build/user/init.debug.elf
STRIP build/user/init.elf
OBJCOPY build/init.bin
llvm-objcopy-6.0: 'build/user/init.elf': The file was not recognized as a valid object file
Makefile:107: recipe for target 'build/init.bin' failed
make: *** [build/init.bin] Error 1
I don't know what to do. I'm on Linux Mint x64. clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
make menuconfig
fails with
CONFIG .config.mk
Traceback (most recent call last):
File "./tools/config.py", line 3, in
import defconfig
ModuleNotFoundError: No module named 'defconfig'
Traceback (most recent call last):
File "./tools/config.py", line 3, in
import defconfig
ModuleNotFoundError: No module named 'defconfig'
Makefile:101: recipe for target 'menuconfig' failed
make: *** [menuconfig] Error 1
The EFI stub enables to boot Resea directly without EFI boot loaders.
servers/experimental/hdaudio/main.c:9:10: fatal error: 'sound_data.h' file not found
#include "sound_data.h"
^~~~~~~~~~~~~~
1 error generated.
its on the gitignore, is this intentional?
Currently, Resea Kernel uses its internal static-sized heap for allocating kernel stacks and page tables. However, as described in microkernel papers authored by L4 folks, handling page faults in (bulk) IPC messes up its implementation and it's not good for the separation of mechanism and policy.
I'm wondering if we could move to a better approach like the following (this is inspired by MINIX3 and seL4):
libresea
internally calls the bootstrap server to copy bulk payloads in advance.page_table
and kernel_heap
parameters to spawn
system call to specify the task's page table address and task's kernel heap respectively.kmalloc()
and kfree()
) and page mapping interfaces (vm_link()
).The disadvantage of this approach is that it degrades the bulk IPC performance since it requires additional IPC with bootstrap server. That said, since the bootstrap server has authority to manipulate page tables, I believe we can implement zero-copy bulk IPCs in the future for performance without modifying the kernel.
The current malloc()
implementation uses a naive algorithm and it easily creates heap fragmentation. Obviously we need to fix this.
Lines 66 to 88 in 93d3f4d
IRET/SYSRET instructions cause #GP if the RIP address is not canonical and thus it leads to a kernel panic.
my config is: https://gist.github.com/milisarge/51ff85e055f4033553026a1ca7a9faed
I am talking about TCPIP service, am i missing some configuration?
https://github.com/nuta/resea/blob/master/docs/kernel/debugging.md says st
command for stats but haven't implemeted?
It might be interesting to provide a RDBMS-based data store server.
When I build the latest version (on Windows WSL) I get:
CC kernel/boot.c
clang: error: unknown argument: '-fstack-size-section'
Makefile:250: recipe for target 'build/kernel/kernel/boot.o' failed
make: *** [build/kernel/kernel/boot.o] Error 1
Currently, tcpip server supports only IPv4.
Hi. I'm using Debian 10.3 / amd64. When I tried to build, I got this error:
kernel/arch/x64/trap.S:428:5: error: invalid operand for instruction
jmp 1b
Ideas?
A live update feature allows updating the operating system without rebooting the computer. Unlike kernel live patch in Linux, on Resea, it might be feasible to implement complicated updates beyond bug and vulnerability fixes.
Running hello completely blocks the shell.
static error_t run_app(const char *name) {
char path[128];
strncpy(path, "/apps/", sizeof(path));
strncpy(&path[6], name, sizeof(path) - 6);
TRACE ("AT %d", __LINE__);
// Open the executable file.
struct message m;
m.type = FS_OPEN_MSG;
m.fs_open.path = path;
m.fs_open.len = strlen(path) + 1;
TRACE ("AT %d", __LINE__);
error_t err = ipc_call(fs_server, &m);
TRACE ("AT %d", __LINE__);
My test code shows
[shell] AT 202
[shell] AT 209
In servers/shell/main.c at line 208
error_t err = ipc_call(fs_server, &m);
doesn't return.
Interestingly, Resea Kernel is used to be written in Rust. However, it soon be rewrote in C because of the following drawbacks:
I do think C is better for a minimalistic microkernels as of this writing, that said, I also believe that the problems might be addressed someday.
Flutter is an open-source GUI component developed by Google. It supports iOS, Android, and their own non-Linux operating system called Fuchsia.
Currently, Resea requires at least two message copies even in the IPC fast path and what's worse, out-of-line payloads (analogous to the pair of buf
and len
in UNIX's read(2)
) involves additional IPC with the pager task and the buffer copy. (Please note that this design decision is for making the microkernel simple as much as possible).
As the kernel allows the pager task to map memory pages, I'm wondering if we could implement a zero-copy IPC using the map
system call and the notifications, a asynchronous IPC mechanism like UNIX's signals.
This issue tracks ideas and the progress of this feature.
libs/resea
) supportWe now have a hardware-accelerated hypervisor and it supports booting Linux. Let's improve it so that we can develop on Linux on Resea.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.