alexellis / firecracker-init-lab

Build a Firecracker microVM from a container image

firecracker-init-lab

Build a microVM from a container image

Many of the examples you'll find are broken due to changes in Firecracker 1.0 - the official quickstart guide doesn't cover the most interesting thing - working Internet access - or extracting a filesystem from a container. This lab extends the official quickstart so that you can explore what an init process does, and add networking.

Pre-reqs

• A bare-metal Linux host - where can you get bare metal?
• Or a VM that supports nested virtualisation such as on DigitalOcean or GCP.
• Docker installed

Browse:

• Go init process
• Makefile
• boot.sh - commands to start MicroVM
• Dockerfile - for building the root filesystem

Usage

Download and install Firecracker to /usr/local/bin/

Or, alternatively, Arkade can do this for you with:

curl -SLs https://get.arkade.dev | sudo sh
sudo arkade system install firecracker

Edit the IFNAME in setup-networking.sh to match your host's network interface.

Then run the script to create the ftap0 device, and to setup IP masquerading with iptables:

./setup-networking.sh

Download the quickstart Kernel:

make kernel

Make the init process binary, and package it into a container, extract the container into a rootfs image:

make all

In one terminal, start firecracker:

make start

In another, instruct it to boot the rootsfs and Kernel:

make boot

Play around in the first terminal and explore the system:

free -m
cat /proc/cpuinfo
ip addr
ip route

ping -c1 1.1.1.1

echo "nameserver 1.1.1.1" > /etc/resolv.conf
ping -c 4 google.com

apk add --no-cache curl

curl -i https://inlets.dev

Expose a TCP or HTTP service to the Internet

Once you've got something interesting running like a HTTP server, or an SSHD daemon, you can then get ingress from the public Internet using an inlets tunnel. Inlets is a static binary, and there are a couple of simple tutorials you can follow depending on what you want to expose.

• HTTPS tunnel with Let's Encrypt
• TCP tunnel for SSH, reverse proxy etc

Running on a Raspberry Pi

Edit Makefile, and change arch to aarch64

export arch="x86_64"

Live-event - A cracking time with Richard Case of Weaveworks

Richard Case will join me as we explain to you why we're so excited about Firecracker, what use-cases we see and try to show you a little of what can be done with it. Richard's been at the sharp end of this technology for months, and is working on a cutting edge bare-metal Kubernetes project called Liquid Metal.

You'll hear more about it on Friday lunch at 12:00pm BST.

Subscribe & remind

If you can't make it live, then you'll be able to jump onto the replay with your morning coffee.

Faster, more secure CI with Firecracker and actuated

We demoed actuated for fast and secure CI with Firecracker, since then it's being used in production and has launched over 100k VMs so far.

Read more on the website, on the blog or in the docs at: actuated.dev.

Watch a demo:

• Actuated for GitHub Actions
• Acuated for GitLab CI

firecracker-init-lab's Issues

Couldn't boot the VM (The kernel file cannot be opened: No such file or directory)

Ran as mentioned in the README, I'm stuck at the boot step:

> make start
sudo rm -f /tmp/firecracker.socket || :
sudo firecracker --api-sock /tmp/firecracker.socket
2022-06-22T19:25:37.802163000 [anonymous-instance:fc_api:ERROR:src/api_server/src/parsed_request.rs:192] Received Error. Status code: 400 Bad Request. Message: The kernel file cannot be opened: No such file or directory (os error 2)
2022-06-22T19:25:37.843471954 [anonymous-instance:fc_api:ERROR:src/api_server/src/parsed_request.rs:192] Received Error. Status code: 400 Bad Request. Message: Cannot start microvm without kernel configuration.

>>> make boot
sudo ./boot.sh
HTTP/1.1 400 
Server: Firecracker API
Connection: keep-alive
Content-Type: application/json
Content-Length: 92

{"fault_message":"The kernel file cannot be opened: No such file or directory (os error 2)"}HTTP/1.1 204 
Server: Firecracker API
Connection: keep-alive

HTTP/1.1 400 
Server: Firecracker API
Connection: keep-alive
Content-Type: application/json
Content-Length: 70

{"fault_message":"Cannot start microvm without kernel configuration."}

Is there anything that I'm missing (any more prerequisites)?

Booting the VM kernel gives "Internal error: undefined instruction: 0 [#1] SMP" on Volterra

Hi,

I was checking out if I could run firecracker-vm on the Volterra box (Windows Dev Kit 2023). It is booted into EL2, has /dev/kvm, and also an lxc VM running to check if VMs are generally available and working.

When booting it halts with the undefined instruction, not quite sure if it's a kernel config that needs to be changed.

[    0.005519] ASID allocator initialised with 32768 entries
[    0.005920] Hierarchical SRCU implementation.
[    0.006492] EFI services will not be available.
[    0.006825] smp: Bringing up secondary CPUs ...
[    0.007143] smp: Brought up 1 node, 1 CPU
[    0.007417] SMP: Total of 1 processors activated.
[    0.007714] CPU features: detected: GIC system register CPU interface
[    0.008131] CPU features: detected: Privileged Access Never
[    0.008494] CPU features: detected: User Access Override
[    0.008840] CPU features: detected: 32-bit EL0 Support
[    0.009208] Internal error: undefined instruction: 0 [#1] SMP
[    0.009578] Process migration/0 (pid: 10, stack limit = 0xffffff8008b78000)
[    0.010030] CPU: 0 PID: 10 Comm: migration/0 Not tainted 4.14.174+ #14
[    0.010443] Hardware name: linux,dummy-virt (DT)
[    0.010739] task: ffffffc0068a0000 task.stack: ffffff8008b78000
[    0.011147] PC is at arm64_set_ssbd_mitigation+0x64/0xa0
[    0.011492] LR is at arm64_set_ssbd_mitigation+0x18/0xa0
[    0.011851] pc : [<ffffff800808e154>] lr : [<ffffff800808e108>] pstate: 004000c5
[    0.012346] sp : ffffff8008b7bd20
[    0.012569] x29: ffffff8008b7bd20 x28: 0000000000000000 
[    0.012927] x27: ffffff800803bbd0 x26: 0000000000000001 
[    0.013282] x25: ffffff800871ee20 x24: ffffff8008854e66 
[    0.013637] x23: 0000000000000001 x22: 0000000000000040 
[    0.013997] x21: ffffff800803bc94 x20: ffffff800887c4a8 
[    0.014353] x19: 0000000000000001 x18: ffffffffffffffff 
[    0.014710] x17: 0000000000000007 x16: 0000000000000001 
[    0.015072] x15: ffffff80087fad08 x14: ffffff808888af37 
[    0.015416] x13: 0000000000000000 x12: 0000000000000001 
[    0.015765] x11: 0000000000000000 x10: 0000000000000a00 
[    0.016110] x9 : ffffff8008b7bd70 x8 : ffffffc0068a0a60 
[    0.016455] x7 : 0000000000000000 x6 : 00000000ffffffff 
[    0.016793] x5 : 0000003fff6eb000 x4 : 0000000000000004 
[    0.017089] x3 : 0000000000000000 x2 : 0000000000000001 
[    0.017432] x1 : 0000000000000001 x0 : 0000000000000001 
[    0.017776] Call trace:
[    0.017941] Exception stack(0xffffff8008b7bbe0 to 0xffffff8008b7bd20)
[    0.018363] bbe0: 0000000000000001 0000000000000001 0000000000000001 0000000000000000
[    0.018878] bc00: 0000000000000004 0000003fff6eb000 00000000ffffffff 0000000000000000
[    0.019342] bc20: ffffffc0068a0a60 ffffff8008b7bd70 0000000000000a00 0000000000000000
[    0.019845] bc40: 0000000000000001 0000000000000000 ffffff808888af37 ffffff80087fad08
[    0.020325] bc60: 0000000000000001 0000000000000007 ffffffffffffffff 0000000000000001
[    0.020750] bc80: ffffff800887c4a8 ffffff800803bc94 0000000000000040 0000000000000001
[    0.021178] bca0: ffffff8008854e66 ffffff800871ee20 0000000000000001 ffffff800803bbd0
[    0.021646] bcc0: 0000000000000000 ffffff8008b7bd20 ffffff800808e108 ffffff8008b7bd20
[    0.022099] bce0: ffffff800808e154 00000000004000c5 ffffff8008652778 ffffff800808a95c
[    0.022592] bd00: ffffffffffffffff ffffff800808e108 ffffff8008b7bd20 ffffff800808e154
[    0.023098] [<ffffff800808e154>] arm64_set_ssbd_mitigation+0x64/0xa0
[    0.023505] [<ffffff800808ef14>] cpu_enable_ssbs+0x74/0xa0
[    0.023846] [<ffffff800808e6b0>] __enable_cpu_capability+0x10/0x20
[    0.024233] [<ffffff800813535c>] multi_cpu_stop+0x8c/0x110
[    0.024572] [<ffffff8008135634>] cpu_stopper_thread+0xc4/0x148
[    0.024893] [<ffffff80080c3d30>] smpboot_thread_fn+0x1a0/0x1d0
[    0.025261] [<ffffff80080bf80c>] kthread+0x12c/0x130
[    0.025567] [<ffffff8008084c50>] ret_from_fork+0x10/0x18
[    0.025898] Code: d4000002 f9400bf3 a8c27bfd d65f03c0 (d503403f) 
[    0.026279] ---[ end trace 66dc7e40a2c28e42 ]---
[    0.026574] note: migration/0[10] exited with preempt_count 1

Attached is the complete log, it boots the VM. And balks on the code.

start.txt

To get the vmlinux kernel I have set export arch=aarch64, maybe it must be more specific?

Question about makefile syntax

What do the characters after the command mean on this line: https://github.com/alexellis/firecracker-init-lab/blob/master/Makefile#L34

I've never come across them in this sequence before.