Immutable OpenBSD on AWS about aws-openbsd

[reyk]

Hi, so an imaginary tool means that it doesn't exist?

You can already use the userdata as an overlay option to deploy custom VMs from generic AMIs (just embed the files as heredocs or shell archive in the userdata script).

Antoine and me are currently experimenting with different ways to create OpenBSD VMs, but I think we're both trying to create something that matches the default OpenBSD installation as closely as possible.

A few comments:

• I agree that swap should be disabled on the main OS disk - it is even a requirement for marketplace VMs. Some machine types in AWS and all VM sizes in Azure offer ephemeral/resource disks that can be used for swap.
• It is not recommended to simplify the filesystem layout. We have certain mount options (eg. wxallowed on /usr/local) and assumptions on OpenBSD's auto-layout. OpenBSD supports simplified layouts like a single partition but you have been warned that they are less secure.
• We are even considering the opposite: auto-installing AMIs. The problem with pre-provisioned VM images is that they all have the same randomisation in the filesystem, kernel, and libraries where the installer usually makes each installation unique. This contradicts of what the project is doing with efforts like KARL (http://undeadly.org/cgi?action=article&sid=20170701170044).
• I don't get the benefits of using the tmpfs hackery here if you don't intend to reboot the server. These things are usually done for appliances that have to be fail-safe on power loss. Also the generation of server keys has been solved in OpenBSD: it is done when you start your newly-installed system for the first time. cloud-init implementations like ec2-init.sh or cloud-agent print the host key fingerprints on first boot.
• We usually don't use --long-options in OpenBSD ;-)

I agree that such immutable images would be helpful for specific use cases, but people have to know these implications. After all, running any kind of software in a public cloud is already some kind of a security compromise, isn't it? But we're still trying to get the best out of OpenBSD.

[ajacoutot]

@sarnowski to add to what @reyk said, once you have access to working OpenBSD AMI, it's easy to modify it to your specific needs using tools like packer (https://www.packer.io/), which is also available as a port :-)
I'd rather concentrate on providing a generic OpenBSD cloud image that would match a standard installation as much as possible while still being cloud-friendly.
There's still some ongoing discussion about the best way to do that.

[sarnowski]

Thanks for your answers. I took some time to prototype what I described and looked at all your code. I think that using packer is a valid compromise to prevent me chasing your tunings ;-) As I wrote before, I know and agree that you want to provide a vanilla/default AMI.

I figured out the aspects why I came up with this issue to begin with. You actually do some choices for the user that are not bound to the default installation:

• Sets: You install all sets automatically. I often don't need more than base and man. I almost never need comp and games and only sometimes need the x* sets (for example for Java). I think it would be trivial to allow customization here. Either provide this as an option in this tool to select the sets so that I could create my own base AMIs or also provide a minimal AMI that I can extend with the other sets if required. (probably providing a "full" AMI or a "minimal" AMI) The problem with all the unnecessary sets is that I need to apply security updates for compliance even if those tools are not in use. This is not a showstopper but it would feel "cleaner" to not have that stuff on the server (as I do on onpremise boxes as well)
• Partition setup: You made your point about the default partitioning scheme. There is one important aspect for me: auto resizing. As far as I can tell, this is not implemented yet in the ec2-init script. As I am using EC2 instances exclusively for server workloads, the size of /var matters the most. If I choose an EBS volume size for my disk, I align that with the space needs I have in /var. Will you implement auto resizing? From my vague understanding of OpenBSD's initialization procedure, there is no natural way to hook such a functionality in right now. Not having this functionality is a showstopper for me.

The vague comment about auto-installing VMs sounds very obscure but also interesting ;-) My interpretation of that statement is that you would actually want to do the installation on boot of the VM. This would of course give the opportunity to dynamically implement the two features I mentioned very naturally (no resizing of partition after the fact). Is there some more information about your plans?

[sarnowski]

Do you think of something like http://man.openbsd.org/autoinstall.8 ? That sounds like a good idea as a first impression. Most questions can probably be answered by the AWS environment (like user, ssh key, ...).

[reyk]

Not fully related to aws-openbsd, but I'm hijacking this issue to ponder about the options.

I can think of three methods to make new VM instances more "unique" and I'm intending to implement all of them to figure out what works best. I also think that running syspatch on -stable VMs is a good idea too because we could avoid that a new instance spawns up with a potentially unpatched kernel, web server, or sshd. Running automatic syspatch in AWS for 6.1 with 14 patches currently takes about 1.5 minutes, see time output in dmesg below.

1. Running some initialization from the standard rc script on first boot. The /etc/rc.sysmerge file seems to provide a good place to do such things. I tested a simple version that runs a) KARL (creating a unique kernel on -current) and b) syspatch (installing updates on -stable).
   See reyk@db1fd8b.

2. Running a custom /etc/rc script on first boot before anything else. This is slightly more complicated but allows to do more first boot actions, including a) KARL, b) syspatch, c) randomization of inode generations number (using fsirand(8) instead of the randomization in newfs). See reyk@abb47e8.

3. Create an auto-installing image using autoinstall(8). I see two problems here: This is more complicated than it sounds because it would most probably need modifications of OpenBSD's installer, for example to find the install script and sets on the disk image and to install the cloud agent/init tool, and would probably take a bit too long on first boot to install all sets. I will give it a try and test how long it takes to auto-install sets in AWS. The benefit is that this would be closest to a "real" installation process.

For the 2nd method, I created two experimental public AMIs in eu-west-1:

• ami-6d03e214 for 6.1-current using KARL (see dmesg in https://gist.github.com/reyk/5a6bc5b8ea052f396c1e72b10b1f2175) for 6.1-current with KARL
• ami-2300e15a for 6.1-stable using syspatch (see dmesg in https://gist.github.com/reyk/99110eae8e5db14c3cd35c1b006bf63d).

[reyk]

• ami-5e846427 for 6.1 auto-installer with syspatch (see dmesg in https://gist.github.com/reyk/55fa601a5b8461e2040d00d5d5f787f4).

Time to close this issue.