GithubHelp home page GithubHelp logo

includeos / includeos Goto Github PK

View Code? Open in Web Editor NEW
4.8K 176.0 357.0 50.73 MB

A minimal, resource efficient unikernel for cloud services

Home Page: https://www.includeos.org

License: Apache License 2.0

C++ 89.38% C 0.94% Shell 1.08% Makefile 0.05% Assembly 1.42% Python 2.97% GDB 0.01% CMake 3.80% JavaScript 0.09% Dockerfile 0.03% Nix 0.23%
unikernel virtualization operating-system cpp

includeos's Introduction

IncludeOS Logo

IncludeOS is a minimal unikernel operating system for C++ services running in the cloud and on real hardware. Starting a program with #include <os> will include a tiny operating system into your service during link-time.

IncludeOS is free software, with "no warranties or restrictions of any kind".

Pre-release Apache v2.0 Join the chat

Note: IncludeOS is under active development. The public API should not be considered stable.

Key features

  • Extreme memory footprint: A tiny bootable C++17 hello world is currently 900 KB and needs only 5 MB RAM on x86_64.
  • KVM, VirtualBox and VMWare support with full virtualization, using x86 hardware virtualization, available on most modern x86 CPUs. IncludeOS will run on any x86 hardware platform, even on a physical x86 computer, given appropriate drivers. Officially, we develop for- and test on Linux KVM, and VMWare ESXi/Fusion which means that you can run your IncludeOS service on Linux, Microsoft Windows and macOS, as well as on cloud providers such as Google Compute Engine, OpenStack and VMWare vcloud.
  • Instant boot: IncludeOS on Qemu/kvm boots in about 300ms but IBM Research has also integrated IncludeOS with Solo5/uKVM, providing boot times as low as 10 milliseconds.
  • Modern C++ support
    • Full C++11/14/17 language support with clang 6 and later.
    • Standard C++ library (STL) libc++ from LLVM.
    • Exceptions and stack unwinding (currently using libgcc).
  • Standard C library using musl libc.
  • Virtio and vmxnet3 Network drivers with DMA. Virtio provides a highly efficient and widely supported I/O virtualization. vmxnet3 is the VMWare equivalent.
  • A highly modular TCP/IP-stack.

A longer list of features and limitations can be found on our documentation site.

Contents

Getting started

Dependencies

For building IncludeOS services you will need:

  • The conan package manager (1.13.1 or newer)
  • cmake, make, nasm (x86/x86_64 only)
  • clang, or alternatively gcc on linux. Prebuilt packages are available for clang 6.0 and gcc 7.3.

To boot VMs locally with our tooling you will also need:

  • qemu
  • python3 packages: psutil, jsonschema

The following command will configure conan to use our build profiles and remote repositories. (Note: this overwrites any existing conan configuration. Set CONAN_USER_HOME to create a separate conan home folder for testing.)

$ conan config install https://github.com/includeos/conan_config.git

Note: If you prefer to set up conan manually the full configuration can be found in the conan_config repository. You can browse our prebuilt conan packages in bintray.com/includeos.

Ubuntu

$ apt-get install python3-pip python3-dev git cmake clang-6.0 gcc nasm make qemu
$ pip3 install setuptools wheel conan psutil jsonschema
$ conan config install https://github.com/includeos/conan_config.git

macOS

If you have homebrew you can use our brew tap to install the dependencies.

$ brew tap includeos/includeos
$ brew install includeos
$ conan config install https://github.com/includeos/conan_config.git

Vagrant

If you want to use a Vagrant box to explore IncludeOS and contribute to IncludeOS development, you can read the getting started with Vagrant. See etc/vagrant.md

Hello World

First select an appropriate conan profile for the target you want to boot on. conan profile list will show the profiles available, including the ones installed in the previous step. When developing for the machine you're currently on, Linux users can typically use clang-6.0-linux-x86_64, and MacOS users can use clang-6.0-macos-x86_64. You can also make your own.

The following steps let you build and boot the IncludeOS hello world example.

$ git clone https://github.com/includeos/hello_world.git
$ mkdir your_build_dir && cd "$_"
$ conan install ../hello_world -pr <your_conan_profile>
$ source activate.sh
$ cmake ../hello_world
$ cmake --build .
$ boot hello

You can use the hello world repo as a starting point for developing your own IncludeOS services. For more advanced examples see the examples repo or the integration tests (under ./IncludeOS/test/*/integration).

Once you're done $ source deactivate.sh will reset the environment to its previous state.

Getting started developing packages

If you are interested in editing/building our dependency packages on your own, you can checkout our repositories at includeos/. Some of our tools and libraries are listed below under Tools and Libraries. You can find the external dependency recipes at includeos/conan. Currently we build with clang-6 and gcc-7.3.0 compiler toolchains. It is expected that these are already installed on your system (see Dependencies for details).

Managing dependencies

Prebuilt conan packages are uploaded to our bintray repository.

We upload to two channels:

  • stable: this channel has all the stable packages.
  • latest: this channel will have the latest packages in development/test phase (including stable releases).

Note: We only guarantee that the latest 10 packages are kept in the latest channel. All stable packages will be kept in the stable channel unless proven unsafe. One suggested workaround is to copy packages into your own repository.

To set up our remote, we recommend following the steps listed in Dependencies.

Search

If you want to check if a package exists you can search for it with conan search. To list all the available packages on our remote includeos, you can use:

$ conan search -r includeos

This should list all the packages we have uploaded to the includeos repository.

To find all the stable versions uploaded for a particular package:

$ conan search -r includeos '<package_name>/*@includeos/stable'

Prebuilt profiles

The profiles we are using for development can be found in the includeos/conan_config repository under conan_config/profiles/.

The target profiles we have verified are the following:

These profiles should have prebuilt packages available and are tested in CI. If you create a custom profile (or use a different profile provided by us) the dependencies may have to be rebuilt locally.

Building with IncludeOS in editable mode

If you are a kernel developer or are simple interested in fiddling with our kernel code, you can use the includeos-package in editable mode. When you rebuild the package will then automatically be updated so it can be used by other packages locally. This is useful when working with several interconnected components and you would like to test your changes across several libraries.

You can read more about how editable mode works in the Conan documentation.

Below we have written down a few steps to get you started with editable packages and IncludeOS.

Note: Currently this is an experimental feature on conan version 1.13 and they have mentioned that for future releases the feature is subject to breaking changes.

Start by cloning the IncludeOS source code and create a build folder. You have to edit etc/layout.txt in the source code to point to the build folder you created, by updating the build_dir variable.

The layout will look similar to the example below. You only have to update build_dir.

  {% set simple=true%}

  {% set build_dir='build' %}

  {% if simple==false %}
  {% set arch=settings.arch|string %}
  {% set platform=options.platform|string %}
  {% set build_dir=build_dir+'/'+arch+'/'+platform %}
  {% endif %}

  [bindirs]
  {#This is so that we can find os.cmake after including conanbuildinfo.cmake #}
  cmake

  [includedirs]
  {#This is to ensure the include directory in conanbuildinfo.cmake includes our API#}
  api

  [libdirs]
  {#This is so that we can find our libraries #}
  {{ build_dir }}/plugins
  {{ build_dir }}/drivers
  {{ build_dir }}/lib
  {{ build_dir }}/platform

  [resdirs]
  {#This is so that we can find ldscript and search for drivers plugins etc#}
  {{ build_dir }}

Note: in the non simple form it is possible to have multiple build folders from the same source which allows multiple architectures and configurations to be tested from the same source however the complexity increases

You should now be able to set the package in editable mode. The following command will add the package as editable based on the specified layout. We inspect the package to get the version, as this has to match exactly.

$ conan editable add . includeos/$(conan inspect -a version . | cut -d " " -f 2)@includeos/latest --layout=etc/layout.txt

The package is now in editable mode and any dependencies of IncludeOS will pick this IncludeOS package from your local cache.

Here is an example on how it looks when its pulled into cache as editable:

$ conan editable list
  includeos/0.14.1-1052@includeos/latest
    Path: ~/git/IncludeOS
    Layout: ~/git/IncludeOS/etc/layout.txt

We are now ready to build the package. Assuming the build-folder is called build under the includeos source directory the following is enough.

$ cd [includeos source root]
$ conan install -if build . -pr <conan_profile> (-o options like platform=nano etc)
$ conan build -bf build .

After making changes to the code you can rebuild the package with

$ cd build && make
   or
$ cmake build --build

Once you have made your changes and the code is finalized you should verify that the conan package still builds. Remove the editable and do a conan create on the package:

$ conan editable remove includeos/<version>@includeos/test
$ conan create <source_path> includeos/latest -pr <conan_profile>

Libraries and tools

We have moved the libraries and tools created by IncludeOS outside the includeos-repository. You can now find them all in their own repositories inside the IncludeOS organization.

To build the libraries and tools, see build instructions in each repository. Typically, the instructions will be in the form:

$ git clone https://github.com/includeos/mana.git
$ cd mana
$ conan create . includeos/latest -pr clang-6.0-linux-x86_64

Below is a list of some of our Libraries/Tools:

  • Vmbuild - Vmbuild is an utility for building the IncludeOS virtual machines.

  • Vmrunner - Vmrunner is a utility developed for booting IncludeOS binaries.

  • Mana - Mana is a web application framework which is used to build a IncludeOS webserver. We have an example named acorn which demonstrates mana's potential.

  • Microlb - Microlb is a library written for building the IncludeOS load balancer. We have an example named microlb which demonstrates our load balancer.

  • Diskbuilder - Diskbuilder is a tool used for building disks for IncludeOS.

  • NaCl - NaCl is the configuration language tool we have tailored for IncludeOS to allow users to configure various network settings such as firewall rules, vlans, ip configurations etc.

Contributing to IncludeOS

IncludeOS is being developed on GitHub. Create your own fork, send us a pull request, and chat with us on Slack. Please read the Guidelines for Contributing to IncludeOS.

C++ Guidelines

We want to adhere as much as possible to the ISO C++ Core Guidelines. When you find code in IncludeOS which doesn't adhere, please let us know in the issue tracker - or even better, fix it in your own fork and send us a pull-request.

Security contact

If you discover a security issue in IncludeOS please avoid the public issue tracker. Instead send an email to [email protected]. For more information and encryption please refer to the documentation.

includeos's People

Contributors

andreasaakesson avatar andreashappe avatar annikah avatar bjornfor avatar dfct avatar fredrb avatar fwsgonzo avatar geisler avatar gurka avatar guztech avatar hansklabbersrabobank avatar haralddahle avatar ingve avatar johncoconut avatar knatten avatar kristianjerpetjon avatar magnuss avatar matchpointdarko avatar matiashf avatar mkinney avatar mnordsletten avatar niks3089 avatar oyviaase avatar ricarkol avatar ricoantoniofelix avatar staiyeba avatar szborows avatar tomasczipri avatar vittorioromeo avatar voultapher avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

includeos's Issues

Allow member functions to subscribe to IRQ

There might be more than one virtio driver, in which case we'd want different instances to be notified about different IRQ's. The nicest way (I think) would be to allow the IRQ subscription system from #6 to be able to handle delegates. Another solution is polymorphism, but this gives us a little overhead (since we'll always have to look up the sub type, as long as any subclass can subscribe to any IRQ. Yet another solution: Force IRQ's into a schema (we can overwrite the defaults on the PCI device). I really don't like that solution.

Network buffer management

At the very least the RX buffers should be rotated. At best, we should have a separate location for buffers, in a dynamic pool. Right now we're just growing, and seem to be colliding with ROM (panic at some point after 0xa0000)

Key decisions:

  • Who takes care of releasing RX- and TX buffers respectively
  • How do we allocate buffers
  • How do we decide when to increase / decrease the pool

Implement virtqueues

There might be some OOP to be done here, but maybe not, since this is a C standard.

STL emplace

EASTL has some missing functionality in most of its containers: namely emplace_* functionality.

Should I implement this functionality?
I have tried to implement std::vector::emplace_back, but it didn't work out since eastl::vector uses its own allocator and I couldn't figure out how all the function definitions are supposed to look.

I imagine emplace_back just takes a parameter pack, and inline constructs the element into a new element in the vector. But I don't know how, or how to avoid a copy/move. Yet.

Virtio IRQ handling

Virtio IRQ requests needs to:

  1. Not send EOI until the Virtio ISR has been read / reset
  2. Read / reset Virtio ISR
  3. Look into virtio ISR to determine which actions to take (i.e. notify qeueue, look for config changes etc.)

Image crashes on many platforms

Works on:

  • Nested inside virtualbox Ubuntu 14.4, on Qemu 2.0
  • Bare metal (Trident) Ubuntu 12.04, on Qemu 1.0

Crashes on:

  • The above, with -cpu pentium and -cpu 486
  • Bare metal (Trident) Ubuntu 14.4, on Qemu > 1.4

Direct boot from virtio block-device

Issue #4 shows how to make openstack hook up the disk image to the IDE bus so that our bootloader works. However, it would be nice if we could boot from a virtio drive directly. It seems to come up as a scsi-disk and PCI device no. 4. Seems hard to do this properly, directly from the bootloader without probing the PCI. Maybe people are using BIOS-calls to do it- the BIOS will know which devices are present and also which one is the boot device. In that case it's back to 16-bit loading.

Virtio device doesn't eat TX packets in virtualbox

In 79cf2ac I'm now adding a single, empty ethernet buffer with a valid (I think) ethernet header. The packet is submitted to the TX-queue and the queue is kicked using tx_q.kick(). In Qemu this results in a virtio interrupt and the packet gets placed in the used ring as expected. In virtualbox, nothing happens. RX-queue buffers, on the other hand, gets eaten just fine, on incoming data.

In other words, I can't send data using virtualbox. Yet.

Things I've tried:

  • Added "memory barrier" before and after updating the avail->idx.
  • Chekcing for notification supression on avail->flags

I think there's something wrong with:

  • A) The TX queue itself (memory alignment or something) - but the first issue I had, with queue addresses being wrong, caused a complete VirtualBox hang which could only be fixed by host reboot.
  • B) The way we kick i.e. notify the queue. The new standard mentions a multiplier notify_off_multiplier in § 4.1.4.4. Could that be used by virtio?
  • C) The packet we're trying to send. Could it be looking for valid Ethernet CRC?

Note: I've noticed that Qemu negotiates VIRTIO_F_RING_INDIRECT_DESC and VIRTIO_F_RING_EVENT_IDX, identified by this output:

  • [x] We can use indirect descriptors
  • [x] There's a Ring Event Index to use

While VirtualBox doesn't:

  • [0] We can use indirect descriptors
  • [0] There's a Ring Event Index to use

... That being said, the standard implies that the absence of these flags should be the simpler case, i.e. we could just go ahead and write to the PCI buffer as we're doing.

Exceptions

Provide support for exceptions inside the Service. This will be necessary in order to use the standard library, unless we use a non-exception version, like EASTL (source here)

IRQ handler indirection

It's necessary to redo the IRQ system: We can arbitrarily assign new IRQ-handlers via IRQ_manager::set_handler(irq, fptr) but this adds the handler directly to IDT. This means the handler has to be an asm-function (due to iret). That asm-function in turn has to call a handler.

So, we need to have:

asm_irq_n_handler -> irq_n_handler -> add_irq_to_task_queue 

Actually, we don't want any work to be done directly by the IRQ-handler, other than just push it to a qeueue, so IRQ_manager::set_handler should not register directly to IDT, but instead to a "subscribe queue". (So, let's leave the set_handler - or remove it - and make a IRQ_manager::subscribe(irq,callback). Every IRQ that has subscribers will then have to have proper "numbered" handlers, which sets a flag or a function pointer. This way, any subscribers will be notified after the IRQ has triggered, and the CPU has returned to the halt-loop.

Get output in OpenStack

The images now boot in OpenStack, and we get the screen output under "console", but no log from the serial port under "logs" in Horizon. Make it work - this way people can use IncludeOS in OpenStack, and we can start making it do useful stuff.

Cannot resize disk image

Resizing the disk image at all, will just corrupt the image, or the loading sequence is bad
Needed for SQLite

Missing IRQ's

The IRQ system which only sets a flag doesn't work; the flag will get cleared at every notification, regardles of whether it was just set.

std::bind

I tried, and I failed. Send help.

  1. Implement std::tuple properly so that indices can be rebuilt into a parameter pack
  2. Find out how SL intends the parameter pack to be visible to std::bind
  3. Uncomment std::bind operator() function
  4. Make tests

implement read()

By default we should red from the serial port. We can create mechnism for rerouting later, i.e. a vga class can reroute anything going to fd 0 or fd 1 to the screen.

Implement ethernet handler

The ethernet handler should parse the ethernet header and notify the corresponding protocol, IP4, IP6, ARP etc.

Syscall `write` needs to take the length into account

Printf is clearly reusing it's buffer and uses size to write partial strings form it.

NOTE: Commit before doing this, since I don't know exactly which functions are using write (I think/hope it's just the syscalls - but they provide random buffer sizes, so expect failures and fix them)

Implement ARP resolution

Now there's a panic if we get a request (say UDP) which we want to answer to, unless the request is preceeded by an arp request (in which case we have allready cached it)

Fetch CPUinfo in OS

Now OS::_CPU_mhz is just set to 2500. OS::uptime needs this to calculate seconds

std::async

Old signals used std::async to have deferred tasks running "later", which in a multi-threaded system means concurrently, but in IncludeOS (atm.) just means run later.

Should we implement std::async in some way, and implement some task-based handler in the background? It would be synchronous but that is beside the point. It just means "do these things later."

Module system

We need a proper module system. I guess "included" modules could be built into a separate archive, modules.a, to complement os.a. This way we can always link with the modules, but only the ones reference will be linked in.

Dynamically change disk layout for bochs

The vmbuilder used to create a disk image with a permanent ~1MB size (32 tracks/cyls@63spt 1 head). This has now changed due to #62. vmbuild will still create a disk image with a valid CSH-layout, where we always use 63 spt (which is the "standard") and 1 head, for simplicity. But, the number of cylinders will now change according to the size of the image (it will be the smallest sufficient number of cylinders).

This means that in order to get bochs to work, we either have to :

  1. Change './etc/bochsrc' dynamically, each time we rebuild an image
  2. Get bochs to work with a more modern type of image, i.e. one that can be created by qemu-image in ./seed/convert_image (It should be able to run vmdk images, but the one I make now didn't work right off the bat)
  3. Set bochs to boot with a command-line parameter specifying the disk layout, and have that paramter be output to a file by vmbuild.

Generalized delegates

Since #10 there's now a specialized delegate class in IRQ_handler which allows delegation of any IRQ to any member function with signature void (). This should be replaced with a generalized delegate class, possibly this one since the C++11 version relies heavily on the standard library. Alternatively, we can wait until we have the Std library.

Safe memory management

We're now loading the kernel into 0x8000 and it just grows from there. This leaves us approximately 589 kb, before we hit EBDA, and then video memory. It's not really a problem if we use video memory (hey - it's memory) unless we want to use the screen for graphics. (And we kind of do, see #17 ). I don't know the consequences of overwriting EBDA however.

I don't think we have to worry about the "ISA hole", since we're not using any ISA devices, but we should do proper memory tests accross all "supported" hypervisors (at least Qemu, virtualbox, vmware and xen should be supported).

A simple "partial" solution would be to place the heap after the 1MB limit, but at some point, for large (> 500Kib) services, we have to start doing partial loading, (or load everything after 1MB) so they don't end up on top of video memory / BIOS area)

Some sub-issues:

  1. Find out the exact memory layout of Seabios and the other platform bioses
  2. Make vmbuilder determine where to load the service, depending on its size.
  3. Make sure stack and heap placements are sane
  4. Consider implementing protected segments over unusable areas.

std::signal

I implemented it as if it was a part of the SL, when its not.

Should I move it elsewhere, and if so, where?

Decide on STL implementation

EASTL (source here) without exceptions or full-blown GCC/libstdc++

  • EASTL could be handy and safe inside the kernel
  • In userland we do want full blown at some point
  • We don't want two STL's...

std::istream, std::ostream

I have implemented a skeleton for istream and ostream, including the basic_stream variants cout and cin. It doesn't follow the standard in any way, since for that I would need a working implementation to work from. Still, right now std::cout uses printf() internally, and std::cin uses fgets() which uses read() and therefore fails.

How relevant are these functions, especially std::cin? Should we implement user input from console/serial?

kernel_start SSE

It's possible the SSE code is enabled too late, if we assume that init_serial() or OS::rsprint could use SSE/AVX because of compiler auto-vectorization. It's unlikely that the compiler will optimize rsprint, though.

A fix would be moving SSE higher up on initialization list or initializing SSE at an earlier stage.

IP Checksumming

We have to either delegate IP checksumming to virtio or do it ourselves. Since virtio many virtio features seems to have poor support we should do it ourselves - then rather turn it off later.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.