crc-org / vfkit Goto Github PK

vfkit has some unit tests, but there are no tests starting a VM and checking the VM has the expected devices.
https://github.com/Code-Hex/vz makes use of https://github.com/Code-Hex/puipui-linux to start a very small VM and get a shell into the VM. vfkit could reuse this to implement various tests (check lspci output, test file sharing, test vsock listen and connect, ...) to catch regressions early.
https://github.com/cfergeau/vfkit/tree/puipui/ has code which downloads puipui, starts a VM and shuts it down through ssh.
We can build on that to add more useful tests.

More projects than just crc are using vfkit (podman, #38, ...). Having vfkit available in brew would make it a lot easier to install.
I made a formula as part of crc-org/crc#3476 , it's still available at https://github.com/cfergeau/homebrew-crc

This needs to be submitted upstream.

When creating a vfkit network device and then creating the vfkit command using .Cmd(), an error is kicked out stating the network device requires an fd or NAT to be enabled.

According to the commit log, this should not be required.

    term1$ gvproxy  --listen unix:///tmp/gvproxy-http.sock --listen-vfkit unixgram:///tmp/gvproxy.sock
    term2$ vfkit --device virtio-net,unixSocketPath=/tmp/gvproxy.sock,mac=5a:94:ef:e4:0c:ee [...]

The code I used is:

	netDevice, err := vfConfig.VirtioNetNew("5a:94:ef:e4:0c:ee")
	if err != nil {
		return err
	}
	// Set user networking with gvproxy
	netDevice.SetUnixSocketPath(m.GvProxySock.GetPath())

This will be needed by podman to redirect all network traffic from the VM to a file descriptor, which will then be exposed on the host as a unix socket. This socket will be passed to gvproxy to handle the VM networking.

This is split from #19

With QEMU, ignition gets its config from the firmware configuration (fw_cfg). We cannot use the same mechanism with vfkit as the virtualization framework does not provide the needed APIs.
What is possible instead is to use the same mechanism as for s390/ppc64 and pass a disk with the ignition id. The raw content of this disk is the ignition config. vfkit only misses a binding for the BlockDeviceIdentifier property of virtio-blk devices. I've already added support for this property to my config-vz-split branch.
One limitation with this approach is that it's s390/ppc only, though there were plans to extend it, see
coreos/ignition#999 and coreos/ignition#928.

This would help for issues such as crc-org/crc#3483
The vfkit side should be relatively easy, quite close to the file sharing support.
The guest will need to mount the shared rosetta directory, and to configure binfmt support https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta?language=objc

This new version of the bindings wraps the virtualization framework APIs added in macOS 13.
Of interest to us:

• EFI support, which means we should no longer need external kernel/initrd, but directly boot from the disk image as on the other platforms
• virtio-serial support, which qemu-guest-agent uses by default. This would allow us to remove the selinux changes/custom systemd unit from crc-org/snc#595

As of now we

https://developer.apple.com/documentation/virtualization/vzfilehandleserialportattachment?language=objc => this api allows bidirectional communication using file handles but we don't know how to consume it.

This is important for crc side to debug some of the issues when ssh connection is broken.

File sharing will be needed in vfkit if we want to test virtiofs support with macOS virtualization framework.
This is implemented in an unmerged Code-Hex/vz PR: Code-Hex/vz#28

Initial testing showed some VM hangs while running https://github.com/pjd/pjdfstest both on 12.0 and 12.3
Testing needs to be redone on 12.4

This is preliminary work for crc-org/crc#3180

https://koji.fedoraproject.org/koji/buildinfo?buildID=2000811 has been used in the latest crc's podman bundle and this kernel was working fine.
However, I've been unable to boot anything newer than this on my Mac M1 using Code-Hex/vz. vfkit would have the same issue.
I tried kernel-5.18.13-200.fc36 , https://koji.fedoraproject.org/koji/buildinfo?buildID=2020964, several newer 5.19.x versions, kernel-6.0.0-54.fc38, ... none of these seemed to be able to boot :(

When making a release, the version in cmd/vfkit/root.go needs to be updated even if we push a tag with the same information.
We could adapt https://github.com/git/git/blob/master/GIT-VERSION-GEN and use golang ldflags in order to automatically generate the correct version.

I tried following the following instructions. Downloading the fedora-coreos-38.20230414.3.0-qemu.x86_64.qcow2 image (then converting it with qemu-img and crating an overlay). But I only get the following error:

$ ./out/vfkit --cpus 2 --memory 2048 --device virtio-blk,path=overlay.img --device virtio-serial,logFilePath=vfkit.log --device virtio-net,nat,mac=72:20:43:d4:38:62 --device virtio-rng --bootloader efi,variable-store=efi-store,create
INFO[0000] &{2 2048    {[efi variable-store=efi-store create] true}  [virtio-blk,path=overlay.img virtio-serial,logFilePath=vfkit.log virtio-net,nat,mac=72:20:43:d4:38:62 virtio-rng] none:// }
INFO[0000] boot parameters: &{EFIVariableStorePath:efi-store CreateVariableStore:true}
INFO[0000]
INFO[0000] virtual machine parameters:
INFO[0000]      vCPUs: 2
INFO[0000]      memory: 2048 MiB
INFO[0000]
Error: unsupported macOS version
Usage:
  vfkit [flags]

Flags:
  -b, --bootloader strings      bootloader configuration (default [])
  -c, --cpus uint               number of virtual CPUs (default 1)
  -d, --device stringArray      devices
  -h, --help                    help for vfkit
  -i, --initrd string           path to the virtual machine initrd
  -k, --kernel string           path to the virtual machine linux kernel
  -C, --kernel-cmdline string   linux kernel command line
      --log-level string        set log level
  -m, --memory uint             virtual machine RAM size in mibibytes (default 512)
      --restful-uri string      URI address for RestFul services (default "none://")
  -t, --timesync string         sync guest time when host wakes up from sleep
  -v, --version                 version for vfkit

unsupported macOS version

My arch is:

$ arch
i386

In vz and the applehv, you can have multiple shared directories under a mount tag. If you dig deeper into the code, there is a use of []shared buried in there.

I discovered this bug because I was trying to pass multiple virtio-fs devices and when doing so, none of them work.

i.e. --device virtio-fs,sharedDir=/Users/brentbaude/foobar1,mountTag=Users. --device virtio-fs,sharedDir=/Users/brentbaude/foobar2,mountTag=somedir

I then tried putting them all under a single mountTag, but multiple devices.

i.e. --device virtio-fs,sharedDir=/Users/brentbaude/foobar1,mountTag=podmanHomeDir --device virtio-fs,sharedDir=/Users/brentbaude/foobar2,mountTag=podmanHomeDir

i guess we need to either parse all virtfs devices to keep the command line the same and when we have multiple sharedDirs under the same mountTag, we would need to group them?

Before podman-machine can make use of vfkit instead of QEMU

Required:

• To avoid the need to extract kernel/initrd from disk images.
  #18
• A way to pass ignition config from the host to the guest
  #35
• Serialization of the VM to disk
  #46
• for easier early-boot debugging
  #44
• Usermode networking support without running gvforwarder in the guest
  #45

Possible improvements:

• #23

vfkit commandline, even a basic VM, is quite long. It could be useful/more user-friendly to be able to describe the VM using a yaml file.

Maybe add a link to the FOSDEM 2023 talk to the readme so that people who want to learn more about vfkit got somewhere to go look?

https://github.com/cfergeau/vfkit/tree/config-vz-split has support for file descriptor or unix socket backed network interfaces through the -device virtio-net fd=xx and -device virtio-net unixSocketPath=xx options.

vfkit could go one step further and provide a way to use directly gvisor-tap-vsock with the VM.

It's suggested to use htps://github.com/machine-drivers/vz as the upstream for the bindings.

Ref:

• crc-org/crc#3362

Perhaps it might even be beneficial if parts of this codebase becomes part of the machine-drivers codebase?