jbrzusto / liwixi

linux where I can see it - create a linux blob that RPi and others can boot from a "normal" μSD card

liwixi - linux where I can see it

Generate an easy-to-install linux blob from a running system. Easy means:

Known working computer / OS:

2017-04-28: Raspberry Pi 2/3 with Raspbian 8

For Users (the picture above in words)

• insert a brand new SD card into your laptop

or

• format a used SD card as VFAT using your laptop's OS or with one of the tools from

  https://www.sdcard.org/downloads/formatter_4/index.html

• download the liwixi archive from your trusted source e.g. COOLSTUFF_LIWIXI.ZIP

• unzip COOLSTUFF_LIWIXI.ZIP to the SD card

• delete COOLSTUFF_LIWIXI.ZIP - it's not needed on the SD card

• safely remove the SD card

• boot the Raspberry Pi with the SD card and...

Marvel at the Splendour!

• the SD card still has the right size, because instead of a hidden linux partition, there is a big fat file called COOLSTUFF_LIWIXI_IMAGE_DO_NOT_DELETE right where you can see it

• if you want your SD card back to normal, just delete the files you extracted from the LIWIXI archive

• you didn't have to download, install, and learn to use Win32DiskImager or dd or ...

• the extra space on your SD card is available, and you can write files to it from your laptop, or from the running pi (in /boot).

• you didn't have to wait for some silly program to write zeroes to the 50% or more of your SD card that is still available.

• the same LIWIXI archive could have been unpacked onto any SD card of sufficient size, and no parititon resizing is required.

• you can have multiple kernels and versions of linux on the same SD card; select which one is booted by editing config.txt to change these lines:

  kernel=kernel7.img
  initramfs=COOLSTUFF_LIWIXI_INITRAMFS_DO_NOT_DELETE
  

  to whatever is appropriate for a different version. The "...INITRAMFS..." and "...IMAGE..." for a particular version of linux have to be used together; the first knows the name of the second, and causes linux to use it.

• if COOLSTUFF needs extra storage for a linux-style filesystem, it might create more big fat files called COOLSTUFF_LIWIXI_STORAGE_N_DO_NOT_DELETE where N is 1, 2, ..., However, we encourage the COOLSTUFF developer to store media and data files directly on the SD card's VFAT filesystem, so you can access them if you pop the SD card into your laptop.

For Developers

Linux can use a loopback device for its root file system; you just have to make sure an appropriate initramfs script takes care of setting it up on a filesystem image before the kernel needs it. This repo provides a tool to create such an image from a working system.

After booting, the entire SD card will be mounted as a VFAT filesystem at /boot and at /dev/sdcard. The linux image file is then recursively accessible as /boot/BRAND_LIWIXI_IMAGE_DO_NOT_DELETE. The small initramfs used to set-up /dev/loop0 is then /boot/BRAND_LIWIXI_INITRAMFS_DO_NOT_DELETE

The image created includes an additional 25% of free space within the ext4 filesystem, to support the growth of logfiles, etc. You can of course allocate further VFAT-file-backed loopback filesystems on the SD card if you need more storage for software packages and so on. But if your linux is intended for data acquisition, consider writing data files to the VFAT filesystem directly - do you really need the features of a linux filesystem enough to justify having the data files invisible to your non-linux users when they pop the SD card into their own computer?

None of this is original or clever, but it might lower the barrier for people just starting in small computer linux, and ease the tedium for developers creating images for them.

Requirements

• a small computer running a recent linux, e.g. Raspbian on the Pi 2/3, whose image you wish to distribute as a LIWIXI archive. You should be logged into a root shell on this machine.

• a USB memory stick or other additional storage should be mounted for use as both temporary and destination storage for the LIWIXI archive.

• make sure you have the zip and initramfs-tools packages installed; e.g.:

apt-get install zip initramfs-tools

• grab liwixi:

git clone https://github.com/jbrzusto/liwixi/master

• run the script:

cd liwixi
./pliwixi BRAND DEST [TEMP]

where:

• BRAND: a string identifying your linux distribution; whitespace is probably toxic

• DEST: path to a folder on a device with enough room to store your compressed image plus boot files

• TEMP: optional path to a folder on a device with enough room to store your uncompressed image plus boot files; defaults to $DEST, which must then have enough room for both the compressed and uncompressed images combined.

DEST and TEMP should be on a USB memory stick or other external storage that you have already mounted, because the SD card itself is comparatively slow for large transfers, and might not have sufficient storage anyway.

Do ./pliwixi for more details and options.

After some chugging, and a chance to customize the image from a shell, you end up with a file called ${DEST}/${BRAND}_LIWIXI.ZIP

which your users can unzip onto a fresh VFAT SD card and use directly; see: For Users (above).

Deployment === Testing - use at your own risk

liwixi's Issues

add -a option to pliwixi for unattended creation

pliwixi gives the user opportunities to cleanly abort the image creation process. This option would
skip those prompts.

remove entry for mounting /dev/mmcblk0p1 on /boot in /etc/fstab

/dev/mmcblk0p1 is now mounted on /dev/sdcard by initramfs, so to prevent double-mounting
that device, we should do the above and make a symlink

/boot -> /dev/sdcard

• fix README.md to reflect this

should /dev/mmcblk0p1 be mounted with any particular options

• flush; probably; we had always mounted FAT partitions with this option on beaglebone SGs, to minimize
  data loss on unclean shutdown. Enabling this might make the FAT filesystem backing the looped-back
  extfs behave more like the real hardware that extfs is expecting, without too much additional wear (?)

• sync: too drastic; much write wear on VFAT?

• dirsync?: good for keeping structure uncorrupted?

pliwixi: let user provide list of additional paths to exclude from image

enlarging the liwixi image to allow space for installing other software?

From B. Noort:

Is there any way to enlarge the Liwixi image, I really like the idea,
but the diskspace is limmited.
I checked the liwixi github page but couldnt find anything about it.

...

I would like install monitoring tooling, so I can be alerted if anything
goes wrong.

The file pliwixi can be modified at line 98 to increase the ratio of free space created in the image:

export IMAGE_MB=$(( `df -BM --output=used / | tail -1l | tr -d 'M'` * 5 / 4))

The fraction 5 / 4 means the image will contain enough space for the operating system plus an additional 25% free disk space. You could change that to e.g. 2 / 1 to reserve 100% additional
free space, etc. You could also just calculate the number of megabytes you want
for the entire image, and replace line 98 with

export IMAGE_MB=2150

for example, which would create an image of size 2.15 GB

Note: I think the liwixi script is broken (see #6) with the most recent raspbian distributions
that are needed for using the Pi 3B+ or Pi Zero Wireless, though. This needs some debugging.

next steps

Goals

• something combining buildroot and a distro; want a curated set of
  packages, some unpackaged code run from onboard git repos, plus
  random hits to config and other files.

• no surprises: once blob is unpacked, space used on SD card and
  bootability should be guaranteed, so that a card taken into the
  field for swapping out, possibly without the chance to test it,
  works - right away

• ability to reset running system or SD card alone (i.e. when
  mounted in a laptop,rather than in target) to "factory" blob

• easy ability to check digests of blob, initramfs,etc. to verify
  install, integrity

• create a clone of running system (including local changes) or just
  of factory blob into archive on external storage

• one-step conversion to/from "developer" blob (which includes all
  tools); from presumably requires network access

• single card that can boot in either beaglebone or Pi?

• creation of either blob type from the other, with user->developer
  again requiring net access

Improvements

• smaller image for user to download and unpack

• track overall system setup in git + scripts, to allow reconstruction
  and validation:

  • base is existing distribution e.g. Raspbian or debian Jessie armhf

  • a text file records any changes made to:

    • on-board git repos

    • distro package removal / install / update

    • changes to any other text files in filesystem are tracked as if
      one big repo, with a separate file to track perms, ownership,
      and other attributes, using lines like: "path" USER:GROUP MODE
      [extented attributes, if necessary]

    • something similar for binaries, except just track SHA digest?

maybe-howtos

• squashfs for factory image with overlain ext4fs to track local
  changes?

• overlain ext4fs in a separate looped-back image file on vfat;
  if that image file is deleted, initramfs generates overlay
  in small ramfs.

• web interface gives user option to create another vfat image
  file as the ext4 overlay for making local changes persistent;
  this can be initialized to current contents of ramfs overlay

pi zero w: not booting

it appears the mount command available inside the initramfs needs different options:
Dropping into the initramfs shell and doing:

 mount -w -o flush,dirsync -t vfat /dev/mmcblk0p1 /dev/sdcard

worked.

But this is with a later version of Raspbian 8 (2017-4) than the 04-30 liwixi release is based on.
Bring full liwixi image up to date...

Also need to delete the "resume" script which with the latest initramfs-tools has the system spinning
waiting for a non-existent resume partition to load (or was that only when mount was failing?)