gwsystems / composite

A component-based OS

The Composite Component-Based OS

This is the source code for the Composite component-based OS. Even low-level system policies such as scheduling, memory mapping, and synchronization are defined as discrete user-level components. Each component exports an interface used to harness its functionality, and components are composed together to form an executable system.

Please see http://composite.seas.gwu.edu for publications and more information.

Branches

• main is the original system with a full user-level set of components.
• ppos is the Speck kernel emphasizing scalable predictability.
• tcaps has mainly been integrated into ppos, but a few unrelated pieces remain.

Research features of Composite

See a summary of the research directions of Composite at http://composite.seas.gwu.edu.

Where to start -- a tour of the source code

• Please read the Composite posts.

• Join the [email protected] mailing list. We use a #slack for our internal development, so this is exceedingly low throughput (1 email every 6 months).

• To run Composite, you start by reading the installation and usage summary in docs/installation_usage_summary.md.

Composite system support

• x86-32
• x86-64
• Qemu with 32/64 bit, x86 support

Important note

The code is pre-alpha quality. Some parts are quite solid, many others are absolutely not. Please consult with us to determine if it is right for your use-case.

Licensing

This code is licensed under the GPL version 2.0 with the class path exception unless otherwise noted (significant portions of user-level are BSD):

The Composite Component-Based OS
Copyright (C) 2009 Gabriel Parmer

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

Linking this library statically or dynamically with other
modules is making a combined work based on this library. Thus,
the terms and conditions of the GNU General Public License
cover the whole combination.

As a special exception, the copyright holders of this library
give you permission to link this library with independent modules
to produce an executable, regardless of the license terms of
these independent modules, and to copy and distribute the resulting
executable under terms of your choice, provided that you also meet,
for each linked independent module, the terms and conditions of
the license of that module. An independent module is a module which
is not derived from or based on this library. If you modify this
library, you may extend this exception to your version of the
library, but you are not obligated to do so. If you do not wish to
do so, delete this exception statement from your version.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.

This license is not set in stone, and we would be willing to negotiate on a case-by-case basis for more business-friendly terms. The license should not prevent you from using this OS, as alternatives can be arranged. It should prevent you from stealing the work and claiming it as your own.

Support

We'd like to sincerely thank our sponsors. The Composite Component-Based OS development effort has been supported by grants from the National Science Foundation (NSF) under awards CNS 1137973, CNS 1149675, and CNS 1117243.

composite's Issues

TLB flush API

To do:

Fix the TLB flush API -- we should switch to a dedicated TLB flush thread (in a separate protection domain) to flush the TLB on that core. Async invocations should be used to flush remote TLBs.

Currently a hack is used to get the job done.

Restructuring the no_interfaces directory to compensate for multiple booters.

Currently within the components/implementation/no_interface/ directory, there are two separate directories for booters: boot/ & llboot/.

llboot/ currently has a softlink to boot/ of booter.c and implements its own boot_deps.h

---

Different booting implementations should be moved to a common boot/ master directory, with common code residing inside of this master directory and different implementations within their respective sub directories.

Eg.
boot/shared code
boot/llboot/code
boot/unikernboot/code

With this, the cos_linker/loader can whitelist the components with names starting with boot.* instead of the current hack.

Build system: warn on cos_init without scheduler

The build system needs to check to see if a component specifies a cos_init() function but fails to depend on a scheduler, and issue a warning to that effect. This will prevent hours of frustration as to why your new component doesn't seem to be running (it isn't, because it's not being assigned a thread).

Structure of struct tcap

Rename budget and budget_local to something more intuitive, perhaps pool and budget, respectively.

Consider changing the structure of tcap_ref to be a generic reference to a capability (could share code with async invocations).

Can struct tcap_budget simply be replaced by an s64_t?

Remove epoch as its role will be handled by liveness.

Sane stack handling 1: guard pages

The stkmgr should coordinate with the valloc component to allocate two pages worth of virtual address space, though we use only one so that we can catch stack overflows (in libraries).

Splitting cbuf and cbufp

The dependency between cbuf_c and cbufp should be removed. The implementation of some cbufp paths is in the cbuf_c interface, especially the meta and allocation management. The two implementations should be separated so that cbufp does not depend on cbuf_c. This implies moving the cbufp-related code and structures out of cbuf_c, and also splitting the include/cbuf.h file into two different includes. Coinciding with Issue #62 the cbuf functions should be turned into scoped cbufs and will likely eventually be hidden from the public-facing API. For any code that is the same between the two kinds of cbufs, the code can be shared with either sym-links or providing a generic header that both can include.

Stack allocation: ABA problem.

In the asm_get_stack path, we use atomic instructions for the linked list operations. However this cannot avoid ABA problem. A possible solution to this is integrating a generation number in the lower 12 bits of the pointers.

Remove the atomic section from spd structure.

Cross-core Priority-Inheritance

When we do priority inheritance (through sched_block), the dependency thread could be on a remote core, in which case cross-core priority inheritance protocol (PIP) is needed. Currently the remote dependency is ignored.

assert()s in composite kernel and user-level

assert()s in composite are not working at present in most places except for platform/i386/kernel.c and platform/i386/boot_comp.c.

I've saved my work here : phanikishoreg@9f4753a

After enabling assert()s correctly, vm.c assert()s seem to fail. Need time to take a look into those. Will get to it eventually. For now, moving on without fixing this at all.

Hardware device identification

Currently in cos_hw_attach(), hwid_t is actually the 'IRQ line number' and the API supports IRQ numbers 32-63 requiring users to actually pass the hwid_t number 32-63 which isn't a good design for a API to expose the internal or machine-level details to the users.

ex: cos_hw_attach(BOOT_CAPTBL_SELF_INITHW_BASE, 32 + i, irq_arcvcap[i]);

This needs to be fixed in kernel/include/hw.h and kernel/include/shared/cos_types.h by allowing kernel API to abstract away the machine level details perhaps by handling the offsets from within the kernel. That would allow users to use the API starting hwid_t at 0:

like: cos_hw_attach(BOOT_CAPTBL_SELF_INITHW_BASE, i, irq_arcvcap[i]);

Bug in cbuf_alloc slow path

I am observing a bug in the following scenario:

1. cbuf_alloc() calls__cbuf_alloc_slow() which calls __cbufp_alloc_slow()
2. __cbufp_alloc_slow() does not collect any cbufs, so it calls cbuf_create().
3. cbuf_create() makes a new cbuf but does not find the meta, so it returns cbid * -1. The cbuf was added to the bin->c list.
4. __cbuf_alloc_slow() detects cbid < 0 and expands meta vector, calls __cbufp_alloc_slow() again and will overwrite the previously returned negative cbid with the return value.
5. __cbufp_alloc_slow() has cbid < 0, and amnt == 0, so it again calls cbuf_create() discarding the negative cbid that was passed in as an argument. The cbid that was returned from the previous cbuf_create is now lost.
6. cbuf_create() makes a new cbuf and gets a meta this time
7. cbuf_alloc() is happy.

Problem: The first call to cbuf_create is not getting handled properly, and the cbid gets 'leaked'. I detected this bug because later on, in cbuf_collect(), there is a loop over bin->c, and a new check was added recently for (CBUF_IS_IN_FREELiST(cbi->owner.m)), that causes a NULL pointer dereference in case cbi->owner.m was not set, as in the above case of the leaked cbid.

I'm pretty sure this bug existed before our recent changes to cbuf. It was just exposed now because of the added check about whether the cbuf is in the freelist.

Change CK to a submodule of Composite.

Currently I added ck to the include directory for simplicity. I need to change it to a submodule.

brand_upcall system call is outdated

cos_syscall_brand_upcall is an outdated system call in current Composite repository.

1. According to what Gabe said, its downside is low efficiency and has not been used since 2007
2. It should be removed
3. it is in cos_component.h, inv.c and ipc.S

Multicore TMem

Currently for each type of TMem (stack and cbuf) we only have a global free-list per component. We need to use per-core free-list for TMems.

High invocation cost with SMP Linux host

The invocation cost is ~600 cycles on uni-processor Linux while it's ~800 cycles with Linux SMP enabled. Check this!

Web server lws_static.sh not working

We are getting return value 0 from tread in HTTP component (to ro_tar component).

Just realized, this might be related to cbuf and cbuf_p. In the original lws_static.sh script, no cbuf_p component existed. Added it in but may not working properly.

Remove the shared region.

The shared region between Components is not used anymore. All cos_argreg_ functions should be removed. The only place that still requires this is the hierarchical scheduler, which uses the region to pass events between parent and children schedulers. This should be modified as well.

Sane stack handling 3: proper faulting on stack absence

When a thread invokes a component, and no thread is available on the freelist, it currently jumps through ugly hoops to save the current registers. Instead, it should use int to save all registers and invoke the stack manager.

Cobj-centric link/load process

Currently there are special cases for if a component is linux-loaded (e.g c0 and llboot), or composite/booter-loaded (the rest) that use the cobj format. We should just product cobjs for each component, and later do a post-processing phase for the linux-loaded components (which we have to do anyway) to load them into mmapped memory.

There is very little immediate pay-off for this currently.

component constructors/destructors

ELF/POSIX/C++ require that the .ctor and .dtor sections which are an array of function pointers have these functions invoked on startup and shutdown (respectively). This will be a convenient way to run any initialization code (e.g. specified with attribute((constructor))), and destruction code (attribute((destructor)) as well (e.g. to avoid the crap in the CBUF_TAKE macro).

This requires changes in the cobj format to include these special sections (easy), and the invocation of these functions on initialization (also easy). This is a big step (e.g. the major one) toward supporting C++.

Cbuf meta next pointer

next pointer leads to a nasty cast warning. Suggested solution is to create it as a union that contains two elements, one a union of next and cbid_tag, and the other a word-sized pointer.

Improving tests codebase structure

Unit tests and microbenchmarks could use some better organization. I propose the following be done:

1. Group the interfaces at components/interface/tests/
2. Group related implementations under respective subfolders in implementation/tests the same as how other subdirs in implementation are grouped that provide multiple implementations for the same interface.

Print component with multi-core

The print component does not work when multiple cores print at the same time. Need to fix it!!!

IPC path optimizations

The mandatory TLB accesses are:

• Instructions for invocation logic (1 superpage)
• Core's kernel stack (1 superpage, unless the TLB is unified, then this is the same entry as above)
• Captbl (1 page) for the SINV invocation side (not the SRET, return side) assuming a low capability number (<128, so that both first and second level fit onto the same page).
• Liveness table (1 superpage, unless it is on the same page as the core's kernel stack, which is likely)

So if we can lay everything out correctly, a call and return should access a total of 2 TLB entries.

The mandatory data cache accesses are:

• X misses for the invocation instructions
• Kernel stack and local core cache + invocation stack (4 cache-lines)
• Capability table (2 cache-lines for a 2 level table), only on the SINV, invocation path

The branches are:

• capid == SRET
• lookup: capentry size == SINV_SZ
• lookup: capentry is allocated (can maybe be avoided, see below)
• captype == SINV (can maybe be wrapped into the check above)
• component is live
• return path: preemption or not (we should be able to get rid of this).

Pending optimizations to the SINV path include:

• Remove the NULL check for if we properly resolve to an active capability by setting not-allocated capability slots to the CAP_FREE type. This is only going to be a win if we also...
• Update captbl layouts to not have NULL ptrs for the next level if there is no Nth level in the captbl. Instead, have a single captbl node that is pointed to by all others that don't have a next level. That node should point to itself so that any recursive resolutions stay within the "NULL node". Resolving if we have a valid capability at the end of lookup is equivalent to detecting if we are in that node.
• The compiler needs to be educated that the two special case paths (for SINV) in the lookup and in the capinv fast path are checking for the same thing so that a single branch compensates for both.
• The kernel stack needs to inline more information. At the very least, this should include the current component's information (captbl + pgtbl), and thread id. At the limit, this might include at least a portion of the thread's invocation stack. This would trade TLB footprint (don't touch the thread) for context switch overhead (memcpy of invocation stack along with registers).

Clean up the kernel v2

The new kernel still has remnants of the old system. A good example is the struct thread. Clean all of this up.

Cbuf: refcnt overflow

For refcnt, nsent and nrecv, as we use faa to atomically increase those counter, check overflow before increment is useless, but check it after the increment it maybe too late.
Find a better way to prevent or handle the overflow.

Proper documentation of tcap_consume()

The comment says tcap_consume() returns 0 on success or 1 on failure. The code also has a path that returns -1. Either change to only return 0 or 1, or document the return value as 0 or not 0 and verify all callers behave appropriately.

Liveness table -> resource table

The liveness table is currently globally visible. it needs to be treated exactly like the other resource tables with range checks (#163).

docs: add to the design document the details about brand/upcall

cstub using C wrappers can be optimized more

When client stubs are generated using the CSTUB_INVOKE macro from cstub.h, the compiler will generate code to save and restore registers that are also being saved and restored by the inline assembly. With a little bit of digging, we can eliminate some of these instructions in the inline assembly.

Refactor: cbuf naming

cbufs are currently named too awkwardly. In the future, persistent cbufs are going to be much more common (for programmers to use) than transient memory cbufs. However, the naming scheme makes it more difficult to use them.

Suggested change:

• cbuf_* -> scbuf_* for scoped cbufs which is a more relate-able name for transient memory cbufs.
• cbufp_* -> cbuf_* so now the namespace for persistent cbufs becomes easier to use.

Access control of IPIs

We need to have the access control of IPIs. When sending IPIs to remote cores, we need to do something to prevent malicious threads from interfering other cores.

Qi

New thread and initialization API

This is a RFC (request for comments) on a proposed thread API in Composite. This will be a low-level API that a pthread implementation could implemented on.

The current problem:

Thread management -- including creation, component initialization, termination, etc... -- is currently pretty ugly in Composite.

An example of this keeps biting new users: components are initialized at system boot-up time. The loader formulates a schedule of initialization under the constraint that a component's dependents must be initialized before it. The problem here is that we don't want to waste N threads (and pay the cost of their creation) for initializing N components, so we wait for the previous thread to complete before using that same thread to initialize the next component. The implication of this is that if a component never returns from cos_init, the dependent components will not be initialized. We created a hack such that if the initialization thread blocks, we assume it is not going to return from cos_init, thus in that case we create another thread to continue the initialization process in the next component. A problem arises when a component is initialized and goes into an infinite loop (as in the CPU component). In this case we will never continue initializing components later in the initialization schedule.

Another example of the hackiness is the thread creation API. Currently when a new thread is created via sched_create_thd. This creates a new thread that begins executing in cos_upcall_fn (which normally will call cos_init). It is not passed any particular arguments. This means that the first time cos_init is invoked, it is an initialization. Later, it is a new thread creation. Thus to emulate pthread_create which will begin execution at a specified function (not cos_init), and passes a specified void* argument, we have to add a branch in cos_init to accommodate this. This is all just a little too ugly.

The proposed solution:

1. To solve the component initialization problem, we should have a convention that you must always return from cos_init, and only then will the system understand that your initialization is complete. This implies that if you want to maintain execution in the given component after initialization, you must create a separate thread for that manually. Though this requires a little more work, it is saner, and clearly expresses a component's intention to the system.

The only problem with this is that lazy component developers might still use the initialization thread for main computation in a component. This will be a detectable bug for "system-level" components, as higher-level components won't initialize. However, for high-level (highest-level) components, it might be easy to avoid detecting this bug in some runscripts (e.g. when the component is the last one to be initialized). Unfortunately, depending on ordering of components in a runscript, this bug might manifest in another runscript where the component is not last. This is not a correctness issue, but might make debugging more annoying.

2. The new thread creation API in cthd.h will include:
   int cthd_create(void _(_fn)(void*), void *, sched_param_t) // return thdid
   int cthd_sched_params(int thdid, sched_param_t, sched_param_t)
   int cthd_exit(void)
   Note that sched_param_t will be a typedef of u32_t derived from union sched_param from https://github.com/gparmer/Composite/blob/master/src/components/include/res_spec.h

This API will make it much easier to manage threads, and take off much of the burden imposed by having to create threads if you want execution to continue beyond initialization.

As mentioned above, pthread.h will be implemented using this API (which will be expanded to include locks/condvars/etc...).

Implementation of solution:

(1) is pretty trivial. The booter needs only avoid creating threads when an initial thread blocks (i.e. this simplifies the code).

For (2), I think the only option is to change the scheduler API and include the fn and void *argument in the call to the scheduler. When the thread is created, in the scheduler, and it is upcalled ino the destination component, these arguments will be passed to cos_upcall_fn (which is the only upcall point in a component). In that function, we can directly jump to the passed function, and pass the argument. We might need to modify the kernel to properly pass arguments on thread creation and upcall.

It would be nice to avoid changing (complicating) the scheduler at all, and simply maintain a queue of pending thread creations including the fn/arg pairs in the component itself. When a new thread begins execution in cos_upcall_fn, it would simply pop one of these records off and use it. However, this would require dynamic memory allocation for all components, and, importantly, would make it impossible to know which thread id should be returned from cthd_create (i.e. which thread is going to get the proper fn/arg pair). So this isn't an option.

Any feedback on this change? As part of the change, I think I'm going to have to discourage all overrides of the cos_upcall_fn (as it is .weak), or else various things won't work such as constructors, destructors, and, now, thread creation.

TODO: new epoch-based `sched_{block,wakeup}`

Currently, sched_block takes the dependency thread as an argument for PI, and each thread that blocks must be sched_wakeuped. This imposes O(n) cost for n threads blocked on a resource. Though this is similar to how Linux does it, the increased cost of a scheduler invocation can make this into an issue (see microbenchmarks in tmem work with and without wakeup optimization). There is another option (worthy of investigation, not guaranteed to work):

Each resource associates an epoch counter with itself. When threads issue sched_block(dependency, epoch), they pass in the epoch. When the thread holding the resource releases it, and does epoch++; sched_wakeup(epoch). The scheduler tracks the current epoch (or 0 if it hasn't seen activity on the thread) for each thread (for resources it has), and for each dependent thread (i.e. the epoch of the resource). The scheduler will track down dependencies, but will always check to make sure they match the epoch counter has not been updated. If it has, then the thread is woken up. This avoids any O(n) cost, and means that sched_{block,wakeup} can now additionally replace sched_component_{take,release}.

tcap API consistency

Most tcap_* functions take a pointer to a tcap as the first parameter. tcap_delete() takes a pointer to spd as its first argument. It should probably have its arguments swapped so the tcap pointer is first for consistency.

Also, tcap_delegate() takes its prio argument as an int rather than u16_t, which would be consistent with tcap_split() and the structure definition.

Multicore Timer

On multicore platform, we are missing some timer interrupts randomly, and receiving multiple queued interrupts continuously. Seems like a Linux related problem. Will check details about mod_timer_pinned.

docs: add to the design document details about system bootup

Both for the llbooter, and the normal booter. Include details about how many 4M regions each component gets, etc...

Documentation: required data-structures, and the invocation path

I should document

1. how the kernel data-structures fit together.
2. which user-level data-structures are requisite
3. the control flow of the invocation path.

The diagrams will take time....

Change non-return upcall-into-scheduler to using fault handlers

In some cases, we need to upcall into scheduler to make the scheduling decisions (e.g. brand_execution_complete). But the upcall setup would overwrite the registers saved in the thread datastructure. The "correct" way to solve this is to not make an upcall into the scheduler, and instead use the new facilities for executing fault handlers in handler components (e.g. like page fault handlers) for an "upcall-can't-return-to-preempted-thread fault". This will make this automatically work by using the fault registers.

remove function calls for invocation path

In both call and return, we are calling spd_mpd_release|take which is not inlined into the invocation path. However, the common case in these functions is both small and likely. We should remove the unlikely case of mpd deallocation (making it unlikely), and inline the rest.

Possible deadlock between cbufs and component locking

The following situation can occur with two threads that may lead to deadlock:

1. thread 0 allocates a cbuf, and is preempted by thread 1.
2. thread 1 allocates a cbuf (there are only 2 in the component), and then takes a component-wide lock (this problem persists regardless of lock granularity so long as t 0 and t 1 take the same lock).
3. thread 1 allocates a cbuf which causes it to call the cbuf_mgr and block with a dependency on t 0.
4. t 0 runs and tries to take the same lock, causing a dependency to t 1.

--> Deadlock.

Possible solutions:

• impose a lock hierarchy including the one used in cbuf_alloc
• enforce that we allocate all cbufs we may need at the start, or none at all (would require dynamic checking to see if the programmer does not follow this convention)

Sane stack handling 2: low level components should use the same assembly-managed stack freelist

Scheduler, memory manager, etc... should not pre-allocate all stacks, and have them allocated in an array, per-thread. Make stack handling consistent across the system.

Incoherency of the TLB in the booter

When the booter maps past its first 4M region, it faults. The page-table is setup fine. Triggering a TLB flush fixes the fault. Thus it seems that for some reason the TLB is not flushed properly. To fix this, I need to check the invocation path, make sure it is flushing correctly, make sure that the global bit isn't getting set in any page table entry (or pgd entry), and hope one of those things is true.

For the time being, I've added a syscall to flush the TLB, and am using that to manually fix the problem in booter. This is a temporary bandaid not meant for the long-term.

cos_linker: failure to handle huge number of symbols

The cos_linker fails to get function addresses when there are very large amounts of symbols.
(tested with a .o that contained ~12,000 symbols).

This failure seems to stem from the bfd library, notably the bfd_canonicalize_symtab(bfd pointer, asymbol double pointer) invoked within loadall.c

The bug is observed when this function is called a large number of times and a proper fix entails investigating the bfd source code.

---

The current work around for this bug is to use a simple cache to store the return value of bfd_canonicalize_symtab(bfd pointer, asymbol double pointer) so it only needs to be called once for each .o file instead of for every symbol.

Failure detection should cause handler execution, not instant replay

Currently, Jiguo's code causes client replay when a fault is detected. The correct solution (which will support distributed migration, etc...) is to cause a fault handler (similar to the page fault handler) to execute in a server component.

Avoid continual insmod/rmmod

Composite has an issue when issuing make or make init that it can crash a random program. That program might be init in which case it crashes the entire system. To solve this problem would take quite a bit of debugging. However, we're at a point where we don't make as many kernel modifications anymore. Thus simply cleaning up the kernel-space (e.g. removing the spds, and threads) would enable us to rerun the system without reinserting the module. This would mean that if you can get a successful make init, then it shouldn't crash again.

So this issue is to do proper tear-down of composite, removing the created structures, and reclaiming memory.

Redefinition of sched_init

In very specific cases, where a scheduler (that implements sched_init already) depends on two components that export the scheduler interface, sched_init will be redefined (linked in twice). This is due to the is_transparent_capability function that will add in the sched_init function as a stub.

This needs to be solved by not adding in that function. For now, I'm just avoiding this situation.

Add range access control checks on higher-order resource tables

Right now access to a RT denotes complete access to its entries. This should not be so. Add range checks for all RT capabilities and accesses to prevent modifying RTs outside of allowed ranges. CPY needs to understand these ranges, and take parameters for how to alter the allowed ranges.

Timer interrupt interposition

Right now we jump back to the Linux timer handler after cos timer handling. Linux will freeze if we don't let its handler run. We should find a way to get rid of this, possibly by
using tickless kernel, which probably could survive timer hijacking.