ajacoutot / aws-openbsd Goto Github PK

View Code? Open in Web Editor NEW

186.0 28.0 47.0 159 KB

AWS OpenBSD image builder (AMI) and cloud-init replacement

Shell 100.00%

aws openbsd cloud-init

aws-openbsd's People

Contributors

Stargazers

Watchers

aws-openbsd's Issues

mirror name not consistent (ie: not used as variable) in create-ami.sh

L132:
echo $MIRROR >${_MNT}/etc/installurl
instead of:
echo "https://ftp.fr.openbsd.org/pub/OpenBSD" >${_MNT}/etc/installurl

S3 Upload Issue

bash-4.3$ ./create-ami.sh -r 6.0 -s 2 -d "OpenBSD 6.0 AMI"
===> creating image container
===> creating image filesystem
===> mounting image
===> fetching sets from ftp.fr.openbsd.org (can take some time)
===> fetching ec2-init
===> extracting sets
===> installing MP kernel
===> installing ec2-init
===> removing downloaded files
===> creating devices
===> storing entropy for the initial boot
===> installing master boot record
===> configuring the image
===> unmounting the image
===> image available at:
     /tmp/aws-ami.OhD7s0xUuN/openbsd-6.0-amd64-20161013T200431Z

===> uploading image to S3 (can take some time)
Requesting volume size: 2 GB
ERROR: Unable to create signed manifest URL. Could not verify the region for bucket openbsd-6.0-amd64-20161013t200431z ; received error: com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket

Need to wait for volume converted before creating snapshot

After uploading the image to S3 and converting the volume, a snapshot is taken. However, after the image is uploaded to S3, we must wait until the volume is converted before the snapshot is taken.

mount_ffs: /dev/wd0k on /tmp: Device busy

thanks for putting this together.
i ran into something i dont understand - trying both openbsd-5.3 and openbsd-snapahot.

openbsd-63# umount -f /tmp
openbsd-63# mount -o "rw,nodev,nosuid" /dev/wd0k /tmp
mount_ffs: /dev/wd0k on /tmp: Device busy

openbsd-63# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/wd0a     1005M   75.9M    879M     8%    /
/dev/wd0k      9.9G   22.0K    9.4G     0%    /home
/dev/wd0f      1.9G    694M    1.2G    37%    /usr
/dev/wd0g     1005M    178M    777M    19%    /usr/X11R6
/dev/wd0h      4.2G   60.0M    4.0G     1%    /usr/local
/dev/wd0j      3.8G    2.0K    3.6G     0%    /usr/obj
/dev/wd0i      1.7G    2.0K    1.6G     0%    /usr/src
/dev/wd0e      2.9G    6.0M    2.7G     0%    /var
/dev/wd0d      1.8G   12.0K    1.8G     0%    /tmp

openbsd-63# disklabel wd0
# /dev/rwd0c:
type: ESDI
disk: ESDI/IDE disk
label: VBOX HARDDISK
duid: 6fabe8bb88d4fb82
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 3916
total sectors: 62914560
boundstart: 64
boundend: 62910540
drivedata: 0

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  a:          2097152               64  4.2BSD   2048 16384 12958 # /
  b:          2588544          2097216    swap                    # none
  c:         62914560                0  unused
  d:          3945920          4685760  4.2BSD   2048 16384 12958 # /tmp
  e:          6176576          8631680  4.2BSD   2048 16384 12958 # /var
  f:          4155776         14808256  4.2BSD   2048 16384 12958 # /usr
  g:          2097152         18964032  4.2BSD   2048 16384 12958 # /usr/X11R6
  h:          9034944         21061184  4.2BSD   2048 16384 12958 # /usr/local
  i:          3587424         30096128  4.2BSD   2048 16384 12958 # /usr/src
  j:          8141536         33683552  4.2BSD   2048 16384 12958 # /usr/obj
  k:         21085440         41825088  4.2BSD   2048 16384 12958 # /home

looks like /dev/wd0k is mounted as home..

minimum 12GB free space of /tmp (8GB for disk image and ~4GB for temporary files).

so /tmp is supposed to.. come from where?

Enchance user-data handling

Don't use 'print' to create the user-data script; it collides with EOL and other stuff.
Just ftp(1) it directly into a file.

[Question] ARM image

Hi @ajacoutot

What would help to create an ARM image? Do I need a physical ARM box with OpenBSD on it?

I am willing to offer some help and I know AWS well but my OpenBSD knowledge is very limited.

Search for volume when registering snapshot

When registering a snapshot, add a search for the correct volume to ensure we have the correct snapshot.

I attempted a patch.

File download in autoinstall

I have to run this in a VmWare machine (because I have no hardware running OpenBSD).

The download of the system sets is extremely slow (possibly because of the doubled virtualization), so I wonder if it would be better to download the installation ISO inside the script first and do the autoinstall in the VM from there.
Downloads inside the installation machine (my VmWare machine) run fast.

Any thoughts about this?

Missing region definition in volume_ids()

There is missing region definition in volume_ids(). Fixed in: kolargol@061ecea

Can not install on t3.micro Instance (free tier)

Hi, after uploaded to the my aws account (Region: HK), I can see the AMI, but I cannot install it on t3.micro instance (free tier)?
And only can install it on d2 instance.
Any idea why will that?

Potentially corrupt drive?

I am not sure if this is the proper place, but since I was using your image and saw your talk I figured you may be interested.

4 days ago I install v6.1 of the OpenBSD AMI on AWS in Virginia. it worked well until I decided to install the ports.

I wanted to test out net/unifi. After installing the ports system I ran make && make install. it took quite a while and compilation stopped mid-course, or so it seemed. I could still see the packages in /usr/ports/packages/amd64/all and running make install would fail due to too little disk space. so I ran df and saw negative space left.

I then ran make clean && make install thinking it would free some memory but I got kicked out of my ssh session and wasn't able to log back in until I rebooted.

My last session before the VM became completely un-responsive is as below, rebooting doesn't let me back into the system.

I plan to shutdown the VM and won't delete it just yet if you'd like to know more or have me try steps to debug this further, as I would like to use OpenBSD in the cloud.

Last login: Sat Oct 14 22:31:35 2017 from 73.223.151.196
OpenBSD 6.1 (GENERIC) #26: Wed Oct  4 18:41:35 CEST 2017

Welcome to OpenBSD: The proactively secure Unix-like operating system.

Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code.  With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.

# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd0a      5.9G   74.9M    5.5G     1%    /
/dev/sd0i      1.7G   18.0K    1.6G     0%    /home
/dev/sd0d      1.5G    8.0K    1.5G     0%    /tmp
/dev/sd0f      3.9G    3.9G   -200M   105%    /usr
/dev/sd0h      3.7G    112M    3.4G     3%    /usr/local
/dev/sd0e      5.9G    243M    5.4G     4%    /var
# du -cks * | sort -rn | head
2       total
2       install_ports.sh
# du -cks /usr/ | sort -rn | head
4238216 total
4238216 /usr/
# du -ckhs /usr/ports/* | sort -rn
910K    /usr/ports/astro
894K    /usr/ports/books
854K    /usr/ports/news
756K    /usr/ports/benchmarks
736K    /usr/ports/plan9
726K    /usr/ports/tests
712K    /usr/ports/inputmethods
500K    /usr/ports/shells
438K    /usr/ports/biology
430K    /usr/ports/education
360K    /usr/ports/chinese
356K    /usr/ports/java
348K    /usr/ports/meta
344M    /usr/ports/distfiles
173M    /usr/ports/packages
108K    /usr/ports/plist
78.0K   /usr/ports/korean
59.8M   /usr/ports/devel
52.3M   /usr/ports/x11
30.8M   /usr/ports/games
30.7M   /usr/ports/www
29.1M   /usr/ports/net
18.7M   /usr/ports/textproc
18.2M   /usr/ports/lang
14.6M   /usr/ports/sysutils
14.2M   /usr/ports/print
13.1M   /usr/ports/graphics
12.4M   /usr/ports/audio
11.4M   /usr/ports/security
9.6M    /usr/ports/mail
9.4M    /usr/ports/databases
6.0M    /usr/ports/editors
6.0K    /usr/ports/CVS
5.1M    /usr/ports/math
4.5M    /usr/ports/misc
4.4M    /usr/ports/multimedia
4.3M    /usr/ports/productivity
4.0K    /usr/ports/bulk
4.0K    /usr/ports/Makefile
3.4G    total
2.8M    /usr/ports/base
2.7M    /usr/ports/INDEX
2.5M    /usr/ports/geo
2.5G    /usr/ports/pobj
2.4M    /usr/ports/emulators
2.4M    /usr/ports/archivers
2.3M    /usr/ports/converters
2.1M    /usr/ports/fonts
2.1M    /usr/ports/comms
2.0K    /usr/ports/README
1.8M    /usr/ports/telephony
1.6M    /usr/ports/cad
1.3M    /usr/ports/infrastructure
1.1M    /usr/ports/japanese
# cat /etc/fstab
35819fb22bf0711e.b none swap sw
35819fb22bf0711e.a / ffs rw 1 1
35819fb22bf0711e.i /home ffs rw,nodev,nosuid 1 2
35819fb22bf0711e.d /tmp ffs rw,nodev,nosuid 1 2
35819fb22bf0711e.f /usr ffs rw,nodev 1 2
35819fb22bf0711e.h /usr/local ffs rw,wxallowed,nodev 1 2
35819fb22bf0711e.e /var ffs rw,nodev,nosuid 1 2
# fsck -f /usr
** /dev/sd0f (35819fb22bf0711e.f) (NO WRITE)
** Last Mounted on /usr
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
265332 files, 2061909 used, 794 free (650 frags, 18 blocks, 0.0% fragmentation)

all roads lead to hang mid installation

the automated installation fo the guest always hangs at the last line of the following.

OpenBSD 6.5 (RAMDISK_CD) #804: Tue Apr 2 20:45:28 MDT 2019
[email protected]:/usr/src/sys/arch/amd64/compile/RAMDISK_CD
real mem = 520093696 (496MB)
avail mem = 500412416 (477MB)
waiting for vm openbsd-current-amd64-20190405T070721Z: mainbus0 at root
bios0 at mainbus0
acpi at bios0 not configured
cpu0 at mainbus0: (uniprocessor)
cpu0: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz, 3698.31 MHz, 06-3a-09
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,LONG,LAHF,ITSC,FSGSBASE,SMEP,ERMS,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
pvbus0 at mainbus0: OpenBSD
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio0
virtio0: irq 3
virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: address fe:e1:bb:d1:3a:a0
virtio1: irq 5
virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio2
scsibus0 at vioblk0: 2 targets
sd0 at scsibus0 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd0: 10240MB, 512 bytes/sector, 20971520 sectors
virtio2: irq 6
virtio3 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk1 at virtio3
scsibus1 at vioblk1: 2 targets
sd1 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd1: 1024MB, 512 bytes/sector, 2097152 sectors
virtio3: irq 7
virtio4 at pci0 dev 5 function 0 "OpenBSD VMM Control" rev 0x00
vmmci0 at virtio4
virtio4: irq 9
isa0 at mainbus0
com0 at isa0 port 0x3f8/8 irq 4: ns16450, no fifo
com0: console

< at this point we are now hung ... i will let it run the rest of the night in case I'm impatient... but i have tried for a at least 30 minutes earlier ... doesnt matter the release or anything else.

Immutable OpenBSD on AWS

I would like to introduce you to a new imaginary tool called create-immutable-openbsd-ami. This tool creates OpenBSD AMIs on AWS with an immutable design. Just like mine, many companies use immutable AMIs to spin up servers in the cloud. Preprovisioned systems have some advantages in dynamic clouds like playing much nicer with tools like AWS AutoScaling Groups and allow you to lock down systems from boot on.

This new imaginary tool has a --help output like this:

Usage: create-immutable-openbsd-ami [options]

Options:
  --mirror      The OpenBSD mirror to download packages from. Look up mirrors at
                https://www.openbsd.org/ftp.html.
                Example: "https://ftp.example.org/pub/OpenBSD/" [required]
  --release     The OpenBSD release like "6.1" or special case "snapshot".
  --sets        A comma separated list of OpenBSD sets to install.
                Defaults to: base,comp,game,man,xbase,xfont,xserv,xshare
  --packages    A comma separated list of packages to install. Packages will be fetched
                from the given mirror. Also supports  referencing build-local packages
                like "./mypackage-1.0.tgz" or remote packages like
                "https://example.com/mypackage-1.0.tgz".
  --root-size   Size of the root partition. Defaults to 1G. Might be tuned to save space
                or to increase space to install more packages.
  --overlay     References a build-local directory which will be copied over to the AMI
                (and replacing potential existing files). Commonly used to provision the
                "/etc/rc.conf.local" file and similar configurations.
  --postexec    A build-local file that will be executed after all build steps on the
                final AMI filesystem to provide generic provisioning capabilities in
                case the other options weren't sufficient.

Examples:

Build the default OpenBSD AMI:

    create-immutable-openbsd-ami --mirror https://ftp.example.org/pub/OpenBSD/ \
                                 --release 6.1

Build the most minimal OpenBSD AMI possible (but with documentation):

    create-immutable-openbsd-ami --mirror https://ftp.example.org/pub/OpenBSD/ \
                                 --release 6.1 \
                                 --sets base,man

Build an OpenBSD AMI with Java available:

    create-immutable-openbsd-ami --mirror https://ftp.example.org/pub/OpenBSD/ \
                                 --release 6.1 \
                                 --packages jdk-1.8

Build an AMI that runs "httpd" by default:

    mkdir -p files/etc

    cat > files/etc/rc.conf.local << "EOF"
    httpd_flags=""
    EOF
    
    create-immutable-openbsd-ami --mirror https://ftp.example.org/pub/OpenBSD/ \
                                 --release 6.1 \
                                 --overlay ./files

Using this imaginary tool would create an AWS AMI with the desired setup. The resulting disk layout would look like this:

# cat /etc/fstab
/dev/sd0a   /      ffs     ro                        0   1
none        /tmp   tmpfs   rw,-s100M                 0   0
/dev/sd0b   /var   ffs     rw,softdep,nodev,nosuid   1   2

On boot, the AMI would resize the /var partition up to the full size that the root disk has free. Since everything except /var will be read-only, home directories of human users need to link to the writeable partition:

/root  ->  /var/root
/home  ->  /var/home

swap is not desired on cloud instances (very opinionated..). cloud-init like initialization is (almost) independent of this tool and should be installed as a package if desired. Generation of SSH server keys needs some thoughts - a mini partition of 1M could be created and mounted for persistent storage of keys, the key directory could be just linked to /var as home directories or the directory will mount a tmpfs, assuming that servers are not restarted/reused in an immutable infrastructure anyway.

All of that can be made pretty generic as almost nothing of it is cloud dependent. AWS specific pieces can be modularized and also potentially extended for other cloud providers.

Why are you telling me this?

This repository contains a great tool to bootstrap OpenBSD AWS AMIs. I am very excited about this possibility and want to pick it up. The current tool creates very generic AMIs that resemble a default OpenBSD installation. This is great in order to offer the most flexibility. It covers a lot of details and will most likely add further tweaks and tricks in the future for optimal cloud operations.

I would like to create a more opinionated AMI design for my operations as described here. But I would like to avoid reinventing the wheel. With this request I would like to open a discussion what the right way to approach this is. Is it desired by you to make your tooling more extendable and configurable or is setting up a dedicated tool a better way to explore this path? Have you any interest in this kind of undertaking to begin with?

Thanks for your great work!

An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation

I'm running OpenBSD 7.1 GENERIC.MP#448 amd64 with latest packages, when running the script I got an error:

================================================================================
| uploading image to S3
================================================================================

An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location-constraint is not valid

awscli returns proper region:

root:/root:402# aws configure get region
us-east-1

Googling shown that when using default region (us-east-1) one don't need to specify

                --create-bucket-configuration \
                LocationConstraint=$(aws configure get region)

Such approach has worked:

diff --git a/obsd-img-builder.sh b/obsd-img-builder.sh
index 41aebe4..92bb6da 100755
--- a/obsd-img-builder.sh
+++ b/obsd-img-builder.sh
@@ -26,9 +26,10 @@ create_ami() {
        vmdktool -v ${IMGPATH}.vmdk ${IMGPATH}
 
        pr_title "uploading image to S3"
-       aws s3api create-bucket --bucket ${_BUCKETNAME} \
-               --create-bucket-configuration \
-               LocationConstraint=$(aws configure get region)
+#      aws s3api create-bucket --bucket ${_BUCKETNAME} \
+#              --create-bucket-configuration \
+#              LocationConstraint=$(aws configure get region)
+       aws s3api create-bucket --bucket ${_BUCKETNAME}
        aws s3 cp ${IMGPATH}.vmdk s3://${_BUCKETNAME}
 
        pr_title "converting VMDK to snapshot"

running this from a obsd < v6.5 (name too long and no wait argument errors)

The version of vmctl in previous versions of obsd (sic. less than 6.5 which at this time is still beta) requires you to alter the file obsd-img-builder.sh by replacing "vmctl wait" with "vmctl status" (and code to keep checking OR increase the previous sleep to 30 seconds or so) as wait was not a argument of vmctl until the version in 6.5. You will likely have to use the -i argument as well as the default time stamp contained in the name constructed by the script is too long for the old versions of vmctl

Log file for ec2-init.sh

It would be great if all outputs from ec2-init were logged into a file (to avoid parsing dmesg)
(similar behavior to cloudinit-output.log)

ajacoutot / aws-openbsd Goto Github PK

aws-openbsd's People

Contributors

Stargazers

Watchers

Forkers

aws-openbsd's Issues

Why are you telling me this?

Recommend Projects

Recommend Topics

Recommend Org

Jobs