rpi-distro / pi-gen

Tool used to create the official Raspberry Pi OS images

pi-gen

Tool used to create Raspberry Pi OS images, and custom images based on Raspberry Pi OS, which was in turn derived from the Raspbian project.

Note: Raspberry Pi OS 32 bit images are based primarily on Raspbian, while Raspberry Pi OS 64 bit images are based primarily on Debian.

Dependencies

pi-gen runs on Debian-based operating systems released after 2017, and we always advise you use the latest OS for security reasons.

On other Linux distributions it may be possible to use the Docker build described below.

To install the required dependencies for pi-gen you should run:

apt-get install coreutils quilt parted qemu-user-static debootstrap zerofree zip \
dosfstools libarchive-tools libcap2-bin grep rsync xz-utils file git curl bc \
gpg pigz xxd arch-test

The file depends contains a list of tools needed. The format of this package is <tool>[:<debian-package>].

Getting started with building your images

Getting started is as simple as cloning this repository on your build machine. You can do so with:

git clone https://github.com/RPI-Distro/pi-gen.git

--depth 1 can be added afer git clone to create a shallow clone, only containing the latest revision of the repository. Do not do this on your development machine.

Also, be careful to clone the repository to a base path NOT containing spaces. This configuration is not supported by debootstrap and will lead to pi-gen not running.

After cloning the repository, you can move to the next step and start configuring your build.

Config

Upon execution, build.sh will source the file config in the current working directory. This bash shell fragment is intended to set needed environment variables.

The following environment variables are supported:

• IMG_NAME (Default: raspios-$RELEASE-$ARCH, for example: raspios-bookworm-armhf)

  The name of the image to build with the current stage directories. Use this variable to set the root name of your OS, eg IMG_NAME=Frobulator. Export files in stages may add suffixes to IMG_NAME.

• PI_GEN_RELEASE (Default: Raspberry Pi reference)

  The release name to use in /etc/issue.txt. The default should only be used for official Raspberry Pi builds.

• RELEASE (Default: bookworm)

  The release version to build images against. Valid values are any supported Debian release. However, since different releases will have different sets of packages available, you'll need to either modify your stages accordingly, or checkout the appropriate branch. For example, if you'd like to build a bullseye image, you should do so from the bullseye branch.

• APT_PROXY (Default: unset)

  If you require the use of an apt proxy, set it here. This proxy setting will not be included in the image, making it safe to use an apt-cacher or similar package for development.

• BASE_DIR (Default: location of build.sh)

  CAUTION: Currently, changing this value will probably break build.sh

  Top-level directory for pi-gen. Contains stage directories, build scripts, and by default both work and deployment directories.

• WORK_DIR (Default: $BASE_DIR/work)

  Directory in which pi-gen builds the target system. This value can be changed if you have a suitably large, fast storage location for stages to be built and cached. Note, WORK_DIR stores a complete copy of the target system for each build stage, amounting to tens of gigabytes in the case of Raspbian.

  CAUTION: If your working directory is on an NTFS partition you probably won't be able to build: make sure this is a proper Linux filesystem.

• DEPLOY_DIR (Default: $BASE_DIR/deploy)

  Output directory for target system images and NOOBS bundles.

• DEPLOY_COMPRESSION (Default: zip)

  Set to:

  • none to deploy the actual image (.img).
  • zip to deploy a zipped image (.zip).
  • gz to deploy a gzipped image (.img.gz).
  • xz to deploy a xzipped image (.img.xz).

• DEPLOY_ZIP (Deprecated)

  This option has been deprecated in favor of DEPLOY_COMPRESSION.

  If DEPLOY_ZIP=0 is still present in your config file, the behavior is the same as with DEPLOY_COMPRESSION=none.

• COMPRESSION_LEVEL (Default: 6)

  Compression level to be used when using zip, gz or xz for DEPLOY_COMPRESSION. From 0 to 9 (refer to the tool man page for more information on this. Usually 0 is no compression but very fast, up to 9 with the best compression but very slow ).

• USE_QEMU (Default: 0)

  Setting to '1' enables the QEMU mode - creating an image that can be mounted via QEMU for an emulated environment. These images include "-qemu" in the image file name.

• LOCALE_DEFAULT (Default: 'en_GB.UTF-8' )

  Default system locale.

• TARGET_HOSTNAME (Default: 'raspberrypi' )

  Setting the hostname to the specified value.

• KEYBOARD_KEYMAP (Default: 'gb' )

  Default keyboard keymap.

  To get the current value from a running system, run debconf-show keyboard-configuration and look at the keyboard-configuration/xkb-keymap value.

• KEYBOARD_LAYOUT (Default: 'English (UK)' )

  Default keyboard layout.

  To get the current value from a running system, run debconf-show keyboard-configuration and look at the keyboard-configuration/variant value.

• TIMEZONE_DEFAULT (Default: 'Europe/London' )

  Default keyboard layout.

  To get the current value from a running system, look in /etc/timezone.

• FIRST_USER_NAME (Default: pi)

  Username for the first user. This user only exists during the image creation process. Unless DISABLE_FIRST_BOOT_USER_RENAME is set to 1, this user will be renamed on the first boot with a name chosen by the final user. This security feature is designed to prevent shipping images with a default username and help prevent malicious actors from taking over your devices.

• FIRST_USER_PASS (Default: unset)

  Password for the first user. If unset, the account is locked.

• DISABLE_FIRST_BOOT_USER_RENAME (Default: 0)

  Disable the renaming of the first user during the first boot. This make it so FIRST_USER_NAME stays activated. FIRST_USER_PASS must be set for this to work. Please be aware of the implied security risk of defining a default username and password for your devices.

• WPA_COUNTRY (Default: unset)

  Sets the default WLAN regulatory domain and unblocks WLAN interfaces. This should be a 2-letter ISO/IEC 3166 country Code, i.e. GB

• ENABLE_SSH (Default: 0)

  Setting to 1 will enable ssh server for remote log in. Note that if you are using a common password such as the defaults there is a high risk of attackers taking over you Raspberry Pi.

• PUBKEY_SSH_FIRST_USER (Default: unset)

Setting this to a value will make that value the contents of the FIRST_USER_NAME's ~/.ssh/authorized_keys. Obviously the value should therefore be a valid authorized_keys file. Note that this does not automatically enable SSH.

• PUBKEY_ONLY_SSH (Default: 0)

• Setting to 1 will disable password authentication for SSH and enable public key authentication. Note that if SSH is not enabled this will take effect when SSH becomes enabled.

• SETFCAP (Default: unset)

  • Setting to 1 will prevent pi-gen from dropping the "capabilities" feature. Generating the root filesystem with capabilities enabled and running it from a filesystem that does not support capabilities (like NFS) can cause issues. Only enable this if you understand what it is.

• STAGE_LIST (Default: stage*)

  If set, then instead of working through the numeric stages in order, this list will be followed. For example setting to "stage0 stage1 mystage stage2" will run the contents of mystage before stage2. Note that quotes are needed around the list. An absolute or relative path can be given for stages outside the pi-gen directory.

A simple example for building Raspberry Pi OS:

IMG_NAME='raspios'

The config file can also be specified on the command line as an argument the build.sh or build-docker.sh scripts.

./build.sh -c myconfig

This is parsed after config so can be used to override values set there.

How the build process works

The following process is followed to build images:

• Interate through all of the stage directories in alphanumeric order

• Bypass a stage directory if it contains a file called "SKIP"

• Run the script prerun.sh which is generally just used to copy the build directory between stages.

• In each stage directory iterate through each subdirectory and then run each of the install scripts it contains, again in alphanumeric order. These need to be named with a two digit padded number at the beginning. There are a number of different files and directories which can be used to control different parts of the build process:

  • 00-run.sh - A unix shell script. Needs to be made executable for it to run.

  • 00-run-chroot.sh - A unix shell script which will be run in the chroot of the image build directory. Needs to be made executable for it to run.

  • 00-debconf - Contents of this file are passed to debconf-set-selections to configure things like locale, etc.

  • 00-packages - A list of packages to install. Can have more than one, space separated, per line.

  • 00-packages-nr - As 00-packages, except these will be installed using the --no-install-recommends -y parameters to apt-get.

  • 00-patches - A directory containing patch files to be applied, using quilt. If a file named 'EDIT' is present in the directory, the build process will be interrupted with a bash session, allowing an opportunity to create/revise the patches.

• If the stage directory contains files called "EXPORT_NOOBS" or "EXPORT_IMAGE" then add this stage to a list of images to generate

• Generate the images for any stages that have specified them

It is recommended to examine build.sh for finer details.

Docker Build

Docker can be used to perform the build inside a container. This partially isolates the build from the host system, and allows using the script on non-debian based systems (e.g. Fedora Linux). The isolation is not complete due to the need to use some kernel level services for arm emulation (binfmt) and loop devices (losetup).

To build:

vi config         # Edit your config file. See above.
./build-docker.sh

If everything goes well, your finished image will be in the deploy/ folder. You can then remove the build container with docker rm -v pigen_work

If you encounter errors during the build, you can edit the corresponding scripts, and continue:

CONTINUE=1 ./build-docker.sh

To examine the container after a failure you can enter a shell within it using:

sudo docker run -it --privileged --volumes-from=pigen_work pi-gen /bin/bash

After successful build, the build container is by default removed. This may be undesired when making incremental changes to a customized build. To prevent the build script from remove the container add

PRESERVE_CONTAINER=1 ./build-docker.sh

There is a possibility that even when running from a docker container, the installation of qemu-user-static will silently fail when building the image because binfmt-support must be enabled on the underlying kernel. An easy fix is to ensure binfmt-support is installed on the host machine before starting the ./build-docker.sh script (or using your own docker build solution).

Passing arguments to Docker

When the docker image is run various required command line arguments are provided. For example the system mounts the /dev directory to the /dev directory within the docker container. If other arguments are required they may be specified in the PIGEN_DOCKER_OPTS environment variable. For example setting PIGEN_DOCKER_OPTS="--add-host foo:192.168.0.23" will add '192.168.0.23 foo' to the /etc/hosts file in the container. The --name and --privileged options are already set by the script and should not be redefined.

Stage Anatomy

Raspbian Stage Overview

The build of Raspbian is divided up into several stages for logical clarity and modularity. This causes some initial complexity, but it simplifies maintenance and allows for more easy customization.

• Stage 0 - bootstrap. The primary purpose of this stage is to create a usable filesystem. This is accomplished largely through the use of debootstrap, which creates a minimal filesystem suitable for use as a base.tgz on Debian systems. This stage also configures apt settings and installs raspberrypi-bootloader which is missed by debootstrap. The minimal core is installed but not configured. As a result, this stage will not boot.

• Stage 1 - truly minimal system. This stage makes the system bootable by installing system files like /etc/fstab, configures the bootloader, makes the network operable, and installs packages like raspi-config. At this stage the system should boot to a local console from which you have the means to perform basic tasks needed to configure and install the system.

• Stage 2 - lite system. This stage produces the Raspberry Pi OS Lite image. Stage 2 installs some optimized memory functions, sets timezone and charmap defaults, installs fake-hwclock and ntp, wireless LAN and bluetooth support, dphys-swapfile, and other basics for managing the hardware. It also creates necessary groups and gives the pi user access to sudo and the standard console hardware permission groups.

  Note: Raspberry Pi OS Lite contains a number of tools for development, including Python, Lua and the build-essential package. If you are creating an image to deploy in products, be sure to remove extraneous development tools before deployment.

• Stage 3 - desktop system. Here's where you get the full desktop system with X11 and LXDE, web browsers, git for development, Raspberry Pi OS custom UI enhancements, etc. This is a base desktop system, with some development tools installed.

• Stage 4 - Normal Raspberry Pi OS image. System meant to fit on a 4GB card. This is the stage that installs most things that make Raspberry Pi OS friendly to new users - e.g. system documentation.

• Stage 5 - The Raspberry Pi OS Full image. More development tools, an email client, learning tools like Scratch, specialized packages like sonic-pi, office productivity, etc.

Stage specification

If you wish to build up to a specified stage (such as building up to stage 2 for a lite system), place an empty file named SKIP in each of the ./stage directories you wish not to include.

Then add an empty file named SKIP_IMAGES to ./stage4 and ./stage5 (if building up to stage 2) or to ./stage2 (if building a minimal system).

# Example for building a lite system
echo "IMG_NAME='raspios'" > config
touch ./stage3/SKIP ./stage4/SKIP ./stage5/SKIP
touch ./stage4/SKIP_IMAGES ./stage5/SKIP_IMAGES
sudo ./build.sh  # or ./build-docker.sh

If you wish to build further configurations upon (for example) the lite system, you can also delete the contents of ./stage3 and ./stage4 and replace with your own contents in the same format.

Skipping stages to speed up development

If you're working on a specific stage the recommended development process is as follows:

• Add a file called SKIP_IMAGES into the directories containing EXPORT_* files (currently stage2, stage4 and stage5)
• Add SKIP files to the stages you don't want to build. For example, if you're basing your image on the lite image you would add these to stages 3, 4 and 5.
• Run build.sh to build all stages
• Add SKIP files to the earlier successfully built stages
• Modify the last stage
• Rebuild just the last stage using sudo CLEAN=1 ./build.sh
• Once you're happy with the image you can remove the SKIP_IMAGES files and export your image to test

Troubleshooting

64 Bit Systems

Please note there is currently an issue when compiling with a 64 Bit OS. See #271

A 64 bit image can be generated from the arm64 branch in this repository. Just replace the command from this section by the one below, and follow the rest of the documentation:

git clone --branch arm64 https://github.com/RPI-Distro/pi-gen.git

If you want to generate a 64 bits image from a Raspberry Pi running a 32 bits version, you need to add arm_64bit=1 to your config.txt file and reboot your machine. This will restart your machine with a 64 bits kernel. This will only work from a Raspberry Pi with a 64-bit capable processor (i.e. Raspberry Pi Zero 2, Raspberry Pi 3 or Raspberry Pi 4).

binfmt_misc

Linux is able execute binaries from other architectures, meaning that it should be possible to make use of pi-gen on an x86_64 system, even though it will be running ARM binaries. This requires support from the binfmt_misc kernel module.

You may see one of the following errors:

update-binfmts: warning: Couldn't load the binfmt_misc module.

W: Failure trying to run: chroot "/pi-gen/work/test/stage0/rootfs" /bin/true
and/or
chroot: failed to run command '/bin/true': Exec format error

To resolve this, ensure that the following files are available (install them if necessary):

/lib/modules/$(uname -r)/kernel/fs/binfmt_misc.ko
/usr/bin/qemu-arm-static

You may also need to load the module by hand - run modprobe binfmt_misc.

If you are using WSL to build you may have to enable the service sudo update-binfmts --enable

pi-gen's Issues

Raspbian Stretch?

Now that Stretch has been officially released, do we know what's happening with Raspbian being updated to Stretch?

Question: What to build on

Hello All,

Not really a bug (unless filled under documentation update request) but my question is when building a raspberry pi image can this be done on any Linux PC or does it need to be build on the pi itself (I know certain dependencies need to be installed)?

To not finish with a question 3 quick things I noticed:

• Tried doing a build which gave me some errors (only to realize I was starting to script with sh build.sh instead of bash build.sh)
• config file contents where unclear to me (figured it out after reading some previous bugs might be nice to add this to the docs :-))
• After checking out I noticed the prerun.sh scripts weren't running after a bit of research noticed the sh files needed to be chmodded to have execute rights else they aren't executed.

Not really good things but would be nice stuff to add to the docs (ow yeah and of course my question ;-))

With kind regards

StaticCube

ioctl failures

When doing successive build, I get ioctl errors using docker or build.sh script.
I tried using virtualbox or a native debian distro

I think that the chroot created is not properly unmounted.

Force password change on first login

I tried to see if there was already a request for this, but saw none.
I do not know if this will impact a GUI environment, but in a lite environment, it would be perfect.

The passwd command has an option to force the password to be changed on the next login. Is it possible this can be toggled for a fresh image, to help mitigate compromised machines?

as root: passwd -e pi
or edit /etc/shadow, and change the 3rd field to a 0
Example: pi:xxx:0:0:99999:7:::

This, by default, will force the person logging in as pi to change the password, and will not allow the setting of the current password.

Build script hangs when trying to install libglib2.0-cil

Hey Team,
I have added few packages to the build script that I want to have installed. In stage2 in one of the files containing list of packages to install I have added libglib2.0-cil. It downloads and unpacks fine but it hangs up on Setting up libglib2.0-cil (2.12.10-5.1) ....

What could be the reason for it ?

Error on build: "Invalid filesystem option set: ^metadata_csum,^huge_file" with docker build

Hey,
I am trying to build pi-gen with the docker build and without it I get the following error. There is no image at the end in the deploy folder.
Using Ubuntu 17.04:

I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb2a851a7  

Device                                                                      Boot 
Start     End Sectors  Size Id Type
/pi-gen/work/2017-07-13-Raspbian/export-image/2017-07-13-Raspbian-lite.img1      
 8192   94207   86016   42M  c W95 FAT32 (LBA)
/pi-gen/work/2017-07-13-Raspbian/export-image/2017-07-13-Raspbian-lite.img2      
94208 3241835 3147628  1.5G 83 Linux


Command (m for help): The partition table has been altered.
Syncing disks.

/boot: offset 4194304, length 44040192
/:     offset 48234496, length 1611585536
mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
mke2fs 1.42.12 (29-Aug-2014) 
Invalid filesystem option set: ^metadata_csum,^huge_file
Done! Your image(s) should be in deploy/
/home/ubuntu
/var/lib/docker/volumes/66fb28dda4b5110681d778b8eacbb516798772ed703b99341caa9334b
de50762/_data/2017-07-13-Raspbian/export-image

current workaround is to build without docker

resize2fs_once.service startup fails at boot

Noticed in the boot sequence for a freshly built image that resize2fs_once was failing to get started by systemd. Checked its status and found that resize2fs_once.service simply didn't exist. Running raspi-config and expanding the filesystem fixes it and the message doesn't appear again.

Systemd services

This may be irrelevant/too hard/wrong approach.

Some of my Pi provisioning mechanisms interact with running services to confirm that updates have worked as expected. The approach requires systemd within the container. Is this an approach that is a/ useful in general, b/ likely to be supported?

tia
Tim

The build date is always UTC, even with different time zone settings

This took me some time to find out.
My timezone - PDT - is UTC-7
So if I start a build and it fails after 5pm, the date will change...
That happens very often in the US West coast.
there should be a better strategy for this.

Controller or keyboard not detected

hey just downloaded and installed the image,

it boots and i end up on the meny, but none of my remote (SNES or PS3) are detected, and, also, my keyboad...does not seem to be detected...any way to help me out?

Changing default language gives errors

Perhaps the locale package should be installed eralier in the process?

Set the locale to en_US.UTF-8 UTF-8

A number of errors appear for several packages before seeing "generating locale"

/usr/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_MESSAGES to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory

Build failed at stage 4

ownership of `/home/harshad/pi-gen-dev/work/2016-12-04-harry/stage4/rootfs/home/pi/python_games' retained as harshad:harshad

rsync: - - chown=1000:1000: unknown option
rsync error:syntax or usage error (code 1) at main.c(1453) [client=3.0.9]

What has gone wrong here?

Suggestion: Use sudoers.d instead of patching /etc/sudoers

Currently stage2/01-sys-tweaks applies a patch to /etc/sudoers, which is a Debian conffile. If at all possible, we shouldn't be modifying Debian conffiles. The reason for that is that should /etc/sudoers ever change in Debian, the user is going to be asked whether they want to keep "their" local changes or adopt Debian's.

Instead, I propose the solution I use on my own systems:

diff --git a/stage2/01-sys-tweaks/00-patches/00-sudoers.diff b/stage2/01-sys-tweaks/00-patches/00-sudoers.diff
deleted file mode 100644
index 3fda3d3..0000000
--- a/stage2/01-sys-tweaks/00-patches/00-sudoers.diff
+++ /dev/null
@@ -1,10 +0,0 @@
-Index: jessie-stage2/rootfs/etc/sudoers
-===================================================================
---- jessie-stage2.orig/rootfs/etc/sudoers
-+++ jessie-stage2/rootfs/etc/sudoers
-@@ -25,3 +25,5 @@ root ALL=(ALL:ALL) ALL
- # See sudoers(5) for more information on "#include" directives:
- 
- #includedir /etc/sudoers.d
-+
-+pi ALL=(ALL) NOPASSWD: ALL
diff --git a/stage2/01-sys-tweaks/00-patches/series b/stage2/01-sys-tweaks/00-patches/series
index b1381c8..5cb4c55 100644
--- a/stage2/01-sys-tweaks/00-patches/series
+++ b/stage2/01-sys-tweaks/00-patches/series
@@ -1,4 +1,3 @@
-00-sudoers.diff
 01-useradd.diff
 02-swap.diff
 03-console-setup.diff
diff --git a/stage2/01-sys-tweaks/01-run.sh b/stage2/01-sys-tweaks/01-run.sh
index 9356ce6..7a9759c 100755
--- a/stage2/01-sys-tweaks/01-run.sh
+++ b/stage2/01-sys-tweaks/01-run.sh
@@ -8,6 +8,8 @@ install -d                      ${ROOTFS_DIR}/etc/systemd/system/rc-local.service.d
 install -m 644 files/ttyoutput.conf            ${ROOTFS_DIR}/etc/systemd/system/rc-local.service.d/

 install -m 644 files/50raspi               ${ROOTFS_DIR}/etc/apt/apt.conf.d/
+install -m 440 files/group_sudo                ${ROOTFS_DIR}/etc/sudoers.d
+


 on_chroot sh -e - <<EOF
@@ -25,7 +27,7 @@ on_chroot sh -e - << \EOF
 for GRP in input spi i2c gpio; do
    groupadd -f -r $GRP
 done
-for GRP in adm dialout cdrom audio users sudo video games plugdev input gpio spi i2c netdev; do
+for GRP in adm dialout cdrom audio users sudo video games plugdev input gpio spi i2c netdev sudo; do
   adduser pi $GRP
 done
 EOF
diff --git a/stage2/01-sys-tweaks/files/group_sudo b/stage2/01-sys-tweaks/files/group_sudo
new file mode 100644
index 0000000..44f8752
--- /dev/null
+++ b/stage2/01-sys-tweaks/files/group_sudo
@@ -0,0 +1,2 @@
+# Allow members of group sudo to execute any command without a password
+%sudo  ALL=(ALL:ALL) NOPASSWD: ALL

The file /etc/sudoers.d/README explains the rules for this.

NB: This patch does provide a small functionality change. Instead of giving the pi user explicitly unlimited access to use sudo, it adds pi to the group sudo and then overrides the rule in /etc/sudoers to make that access passwordless.

You could put the group_sudo file into one of the Raspbian mods packages, but that might change behavior for existing systems. I'd suggest doing it only on new systems as I've done here. In any event, if you opt to add it to a package instead of doing it here as I've done, mark it as a conffile so that user changes to the file will be preserved unless they choose not to.

Missing dependencies

rsync and realpath are missing as dependencies in depends. I am trying to build inside a docker container pulled from ubuntu:trusty.

In addition, would the maintainers be interested in a PR for a docker-ized build system? It would be an addition in the form of a Makefile and Dockerfile only.

Question: read-only root filesystem

I'm trying to modify pi-gen to create an image which mounts / and /boot read-only. This means, for example, removing resize2fs_once. In a nutshell, I'm trying to take the steps in this article and do them at generation time so that the system is read-only from the very first boot.

Has anyone tried this, or am I in new territory? I'm sure I'll get it working eventually, but I wanted to see if anyone has any experience here before I spend too much time hacking away at it.

Suggestion: package changes

Should git-core be removed from stage3/00-install-packages/00-packages and replaced with git in either stage2 or stage4? In stage2 it makes -lite a little less lite, and it does fit with the more complete development tools suggested by stage4.

The gvfs-daemons package now depends on udisks2. Anything else that uses it in Raspbian requires that or udisks. Currently stage3 installs udisks. Perhaps it is time to update that to udisks2? Doing this will cause a few other packages not to be installed: hdparm, powermgmt-base, and cryptsetup-bin.

Python_Openzwave autoinstall fails (HASSBian 1.23)

Forgive my pull req... Should have been an issue.

Fresh flash of hassbian 1.23
added zwave: to configuration.yaml
Receive the following error in log:

Jun 07 12:35:49 hassbian hass[738]: 2017-06-07 12:35:49 INFO (Thread-4) [homeassistant.util.package] Attempting install of python_openzwave==0.4.0.31
Jun 07 12:38:19 hassbian hass[738]: 2017-06-07 12:38:19 ERROR (Thread-4) [homeassistant.util.package] Unable to install package python_openzwave==0.4.0.31: Command "/srv/homeassistant/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-92041o8k/python-openzwave/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-uses1i9y-record/install-record.txt --single-version-externally-managed --compile --install-headers /srv/homeassistant/include/site/python3.4/python-openzwave --home=/tmp/tmpyxonxd5x" failed with error code 1 in /tmp/pip-build-92041o8k/python-openzwave/
Jun 07 12:38:19 hassbian hass[738]: 2017-06-07 12:38:19 ERROR (MainThread) [homeassistant.setup] Not initializing zwave because could not install dependency python_openzwave==0.4.0.31
Jun 07 12:38:19 hassbian hass[738]: 2017-06-07 12:38:19 ERROR (MainThread) [homeassistant.setup] Setup failed for zwave: Could not install all requirements.

Slow builds

Building an image takes a very long time. Are there any suggestions for areas where improvements could be made to speed up the process.

Exporting at Stage 1 fails

Trying to export "truly minimal system" of Stage 1 fails:

[18:24:14] Begin /path/to/export-image/04-finalise/01-run.sh
/bin/bash: line 1: /etc/init.d/fake-hwclock: No such file or directory
/bin/bash: line 2: hardlink: command not found

Build fails at stage2/01-sys-tweaks/03-console-setup.diff

My build on Ubuntu 14.04.4 LTS is failing with the following error:

Applying patch 03-console-setup.diff
patching file rootfs/etc/default/console-setup
Hunk #1 FAILED at 6.
1 out of 1 hunk FAILED -- rejects in file rootfs/etc/default/console-setup
Patch 03-console-setup.diff does not apply (enforce with -f)

It seems like the patch is expecting the following:

-CODESET="Lat15"
-FONTFACE="Fixed"
-FONTSIZE="8x16"

but the file to patch actually contains:

CODESET="guess"
FONTFACE="Fixed"
FONTSIZE="8x16"

As a part of the build process there was a graphical chooser for the codeset where the default was "guess", so I chose that. Manually editing work/rootfs/etc/default/console-setup fixes the issue and lets the build continue.

Question: Best practices to customise it?

Hi all,
the project doesn't seem to have discussion group, so I will ask here.

1. Given I am working on my own step and I sometimes make mistakes, is there a way to avoid running all previous steps and speed up the development process?

2. If I was to use a devops tool like Ansible or Chef, what would be the best approach?

Thanks,
Franklin

apt-get failures

This may be an issue with the archive.raspberrypi.org server, but I'm inconsistently seeing build errors:

Now at patch 02-persistant-net.diff
[22:43:40] End /work/stage1/02-net-tweaks/00-patches
[22:43:40] Begin /work/stage1/02-net-tweaks/00-run.sh
Adding 'local diversion of /lib/udev/rules.d/75-persistent-net-generator.rules to /lib/udev/rules.d/75-persistent-net-generator.rules.distrib'
[22:43:40] End /work/stage1/02-net-tweaks/00-run.sh
[22:43:40] End /work/stage1/02-net-tweaks
[22:43:40] Begin /work/stage1/03-install-packages
[22:43:40] Begin /work/stage1/03-install-packages/00-packages
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  alsa-utils device-tree-compiler libasound2 libasound2-data libparted2 libsamplerate0 lua5.1 parted triggerhappy
Suggested packages:
  libasound2-plugins libparted-dev libparted-i18n parted-doc
The following NEW packages will be installed:
  alsa-utils device-tree-compiler libasound2 libasound2-data libparted2 libraspberrypi-bin libraspberrypi0 libsamplerate0 lua5.1 parted raspi-config triggerhappy
0 upgraded, 12 newly installed, 0 to remove and 0 not upgraded.
Need to get 4384 kB of archives.
After this operation, 11.4 MB of additional disk space will be used.
Get:1 http://archive.raspberrypi.org/debian/ jessie/main libasound2-data all 1.0.28-1+rpi1 [65.1 kB]
Get:2 http://archive.raspberrypi.org/debian/ jessie/main libasound2 armhf 1.0.28-1+rpi1 [321 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ jessie/main libparted2 armhf 3.2-7 [259 kB]
Get:4 http://archive.raspberrypi.org/debian/ jessie/main device-tree-compiler armhf 1.4.1-1+rpi1 [351 kB]
Get:5 http://mirrordirector.raspbian.org/raspbian/ jessie/main libsamplerate0 armhf 0.1.8-8 [939 kB]  
Get:6 http://mirrordirector.raspbian.org/raspbian/ jessie/main alsa-utils armhf 1.0.28-1 [960 kB]
Err http://archive.raspberrypi.org/debian/ jessie/main libraspberrypi0 armhf 1.20161215-1
  404  Not Found [IP: 93.93.130.104 80]
Get:7 http://mirrordirector.raspbian.org/raspbian/ jessie/main lua5.1 armhf 5.1.5-7.1 [75.1 kB]
Get:8 http://mirrordirector.raspbian.org/raspbian/ jessie/main parted armhf 3.2-7 [191 kB]
Err http://archive.raspberrypi.org/debian/ jessie/main libraspberrypi-bin armhf 1.20161215-1
  404  Not Found [IP: 93.93.130.104 80]
Get:9 http://mirrordirector.raspbian.org/raspbian/ jessie/main triggerhappy armhf 0.3.4-2 [31.2 kB]
Get:10 http://archive.raspberrypi.org/debian/ jessie/main raspi-config all 20161207 [17.7 kB]   
Fetched 3211 kB in 2s (1154 kB/s)                          
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/libraspberrypi0_1.20161215-1_armhf.deb  404  Not Found [IP: 93.93.130.104 80]

E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/libraspberrypi-bin_1.20161215-1_armhf.deb  404  Not Found [IP: 93.93.130.104 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Including `certbot` from jessie-backports fails authentication

I'm trying to include certbot, the Let's Encrypt certificate client, in stage3. I have included the jessie-backports repository in sources.list, but when the script tries to install certbot, the following error occurs:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  dialog python-acme python-certbot python-cffi-backend python-chardet python-configargparse python-configobj python-cryptography python-dialog python-dnspython python-enum34
  python-funcsigs python-idna python-ipaddress python-mock python-ndg-httpsclient python-openssl python-parsedatetime python-pbr python-pkg-resources python-psutil python-pyasn1
  python-pyicu python-requests python-rfc3339 python-setuptools python-six python-tz python-urllib3 python-zope.component python-zope.event python-zope.interface
Suggested packages:
  python-certbot-apache python-certbot-doc python-acme-doc python-configobj-doc python-cryptography-doc python-cryptography-vectors python-enum34-doc python-funcsigs-doc
  python-mock-doc python-openssl-doc python-openssl-dbg python-psutil-doc doc-base python-socks python-setuptools-doc python-ntlm
Recommended packages:
  letsencrypt
The following NEW packages will be installed:
  certbot dialog python-acme python-certbot python-cffi-backend python-chardet python-configargparse python-configobj python-cryptography python-dialog python-dnspython python-enum34
  python-funcsigs python-idna python-ipaddress python-mock python-ndg-httpsclient python-openssl python-parsedatetime python-pbr python-pkg-resources python-psutil python-pyasn1
  python-pyicu python-requests python-rfc3339 python-setuptools python-six python-tz python-urllib3 python-zope.component python-zope.event python-zope.interface
0 upgraded, 33 newly installed, 0 to remove and 46 not upgraded.
Need to get 2313 kB of archives.
After this operation, 11.1 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  python-cffi-backend python-pkg-resources python-configargparse python-six python-idna python-ipaddress python-pyasn1 python-setuptools python-cryptography python-dialog
  python-dnspython python-funcsigs python-openssl python-ndg-httpsclient python-pbr python-psutil python-urllib3 python-requests python-rfc3339 python-mock python-acme python-certbot
  certbot
E: There are problems and -y was used without --force-yes

I have tried adding the main Debian Jessie signing key to apt-key and running apt-key update but neither has worked so far.

Happy for any thoughts!

Build fails at export-image/04-finalise/01-run.sh

Building with current git version fails:

$ cd pi-gen-dev
$ echo "IMG_NAME='MyRaspbian'" > config
$ ./build-docker.sh
...
[10:25:03] Begin /pi-gen/export-image/04-finalise
[10:25:03] Begin /pi-gen/export-image/04-finalise/01-run.sh
Stopping fake hwclock: saving system time.
Mode:     real
Files:    2257
Linked:   344 files
Compared: 0 xattrs
Compared: 590 files
Saved:    9.44 MiB
Duration: 0.11 seconds
fatal: Not a git repository (or any of the parent directories): .git
Done! Your image(s) should be in deploy/

And there is no image in deploy.

[Suggestion] ${i}-run.sh must be .sh?

Noticed a few things looking at build.sh and scripts/common:

1. ${i}-run-chroot (no suffix) is a thing that could be used, but never is even where it makes sense.
2. ${i}-run.sh (.sh suffix) is used, and could be any language of course, though bash is obviously assumed.
3. on_chroot everywhere uses sh -e (dash) rather than bash -e.

While contemplating what if anything to do about any of the above, it crossed my mind: What if for some reason you wanted or needed to have a -run script that wasn't written in bash? How would that work?

Answer: Something like this:

#! /bin/bash

i=00

for run_script in ./${i}-run{.*,}; do
        if [ -x "$run_script" ]; then
                echo "Begin running ${run_script}"
                $run_script
                echo "End running ${run_script}"
        fi
done

Okay, let's try that out:

00-run.py: a /usr/bin/env python3 script, ASCII text executable
00-run.sh: Bourne-Again shell script, ASCII text executable
00-run:    a /bin/cat script, ASCII text executable

... /bin/cat!? Sure, why not.

Begin running ./00-run.py
./00-run.py has been run!
End running ./00-run.py
Begin running ./00-run.sh
./00-run.sh has been run!
End running ./00-run.sh
Begin running ./00-run
#! /bin/cat

I'm pretty useless, but I could exist and be anything at all...
End running ./00-run

At all wanted or cared about, or is this feature creep? I've got thoughts about how -run-chroot could be useful, if it were a file copied to the rootfs, run there, and then deleted, which would allow for more flexibility than an inline script. That's particularly useful to me for now since my Raspple II stage 5 presently uses -run to arbitrarily run random scripts from the chroot, though how that's done will change over time.

Error when executing build.sh

I am constantly getting IMG_NAME not set and can't get build process to start.

What can be the possible cause for this error? I even checked local env settings.

Code is cloned directly from github.

Suggestion to build.sh

This script adds a check to make sure the user is under root to run the script and adds all dependencies before starting the scripting process and if IMG_NAME is not set, will echo how to set IMG_NAME in the command.
build.txt

No recovery after failure

There is a horrible issue when the script fails from network issues during any stage of the apt-get processes.

When attempting to resume, the error is "WORK_DIR/?/apt' is not a directory or an apt error that cannot seem to be recovered.

Most of the time resuming is not possible, so a removal of the work directory is required.. this causes a cycle of issues since the original failure was a network issue (in this case DNS randomly failing) and the network issue may or may not occur again. I have also seen an error "cannot verify package" from apt.. which seems that it should be re downloaded instead of reporting an error..

Re downloading the packages every time the script is run with a new image name, or new date, or changing something else where the work directory has to be removed is very time consuming when trying to debug and build a deployable image that can be put in an embedded environment with no user intervention.

Creating an apt-cache directory for the downloaded packages in work/.cache or something and setting apt to use that instead of the temporary root fs being created is probably a good way around re downloading the same packages unnecessarily so the only packages being downloaded are the ones that failed before - even after cleaning the working image directory.

lightdm.conf patch fails at stage4

Using the 2016-09-23 release, the patch stage4/02-extras/00-patches/0-autologin.diff attempts to set lightdm for autologin-user=pi, but at least on my build, it fails because that has already been set at stage3.
Not sure what modified /etc/lightdm/lightdm.conf at stage3. The config file as delivered in lightdm_1.18.2-2_armhf.deb has autologin-user commented out as expected. Nothing obvious in the logs indicate that line being modified by another package install or by an explicit action of stage3.

dependencies_check fail

so i was testing the new build with sudo IMG_NAME="raspbian" ./build.sh and there is an error tossed from the dependencies_check file
/home/ssfdre38/pi-gen/scripts/dependencies_check: line 27: unexpected EOF while looking for matching `"'
Line 27 is echo "$missing"

./build-docker.sh failed raspbian release file

Hello,
I'm currently working on a custom image of Raspbian at stage2 level with some specific tweaks. It works well with ./build.sh
I would like testing it with docker support :

./build-docker.sh 
Sending build context to Docker daemon 1.597 MB
Step 1 : FROM debian:jessie
 ---> 8cedef9d7368
Step 2 : RUN apt-get -y update &&     apt-get -y install         git vim parted         quilt realpath qemu-user-static debootstrap zerofree pxz zip dosfstools         bsdtar libcap2-bin rsync grep     && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 19a0cb25957d
Step 3 : COPY . /pi-gen/
 ---> Using cache
 ---> 82bd079b4475
Step 4 : VOLUME /pi-gen/work /pi-gen/deploy
 ---> Using cache
 ---> bbe618019985
Step 5 : CMD /bin/sh
 ---> Using cache
 ---> 3211122be11f
Successfully built 3211122be11f
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
update-binfmts: warning: qemu-aarch64 already enabled in kernel.
[14:53:26] Begin /pi-gen
[14:53:26] Begin /pi-gen/stage0
[14:53:26] Begin /pi-gen/stage0/prerun.sh
I: Running command: debootstrap --arch armhf --foreign --components=main,contrib,non-free --no-check-gpg jessie /pi-gen/work/2017-04-20-Raspbian/stage0/rootfs http://mirrordirector.raspbian.org/raspbian/
I: Retrieving Release 
E: Failed getting release file http://mirrordirector.raspbian.org/raspbian/dists/jessie/Release
Done! Your image(s) should be in deploy/

when i launch container manually and tries to wget this file it works fine.

I think it is a very easy issue but don't find where i fail.

Thanks !

Please do not default to using Google nameservers

See discussion here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=761658

Printing "Done! Your image(s) should be in deploy/" after failure

After build fails to create raspbian image, pi-gen no matter prints that image should be in deploy/ folder. This is a small bug, but it is annoying when your build fails and you must look into deploy folder expecting that image is there... But it's not true =)

Shrink the size of the image file

The size of the final image files seems to be determined in "export-image/prerun.sh" script file with the following command.
IMG_SIZE=$(expr $BOOT_SIZE \* 2 \+ $TOTAL_SIZE \+ 512)M

I understand the boot size and total size part in this command, but I'm not sure why an extra 512M space is needed.

For now, I'm trying to create a image file with size as small as possible. I know that a lite version image is created in stage2, however, its size still over 1GB. How can I shrink the size more?

Raspberry Pi Debian release key is unchecked.

In the bootstrap function in scripts/common the --no-check-gpg flag is provided to deboostrap. Seems like it would be a little better to embed the key signature somewhere in the repository, get the key, check it against the signature, and then if it is good add it to a keyring so that deboostrap can validate that the repository is authentic.

How do you use this tool?

Question: What are steps 01 and 02 in export-image?

I notice there are two intermediate steps in the export-image stage called set-sources and network. Is there a reason these were added here as opposed to one of the earlier stages? 01-set-sources also has:

on_chroot sh -e - <<EOF
apt-get update
apt-get -y dist-upgrade
apt-get clean
EOF

The update and dist-upgrade seem to be done already in stage0/01-configure-apt/00-run.sh and the clean is done in stage2/03-cleanup/00-run.sh. Wondering if this ends up being redundant.

-bash: startx: command not found

After using pi-gen and build.sh to construct (what I am to believe) is an image of a desktop Raspbian system and subsequently copying this image onto a 16GB SD card, I am unable to access the GUI on my Raspberry Pi. Attempting to run startx returns the error in the title, whilst attempting to start the GUI on boot via raspi-config boots the pi into the CLI without providing any error messages/feedback.

I am relatively new to Rapsbian, and as such, I have two queries relating to this:

1. Am I missing something obvious in the Raspberry Pi installation process? After loading the image onto an SD card, I have been placing it in to my Raspberry Pi and powering it up.

2. How do I know which Raspbian Stage I have built to? In the "Stage Anatomy" section of readme.md, it is stated that the build of Raspbian is divided up into several stages from 0 to 4 inclusive, where stages 3 and 4 refer to the full Raspbian desktop systems as opposed to the Raspbian-Lite system. However, pi-gen contains stages 0-5 inclusive and I was unable to determine what Stage 5 relates to. I have constructed images up to (and including) Stage 3 and Stage 4 of pi-gen, and I experience the same issue detailed above with both images.

Thank you for all the great work!

export-noobs failure if you use /dev/mapper

I posted part of this in my run of pi-gen in #13, and I've noticed I always get the same result:

[05:13:41] Begin /home/tjcarter/Source/RPi-Distro/pi-gen/export-noobs
[05:13:41] Begin /home/tjcarter/Source/RPi-Distro/pi-gen/export-noobs/prerun.sh
umount: /: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)

real    72m34.969s
user    70m47.640s
sys     2m21.440s

Note this was my RPi-Distro tree, which has no patches.

You'll note that there is no End /home/tjcarter/Source/RPi-Distro/pi-gen/export-noobs/prerun.sh to match, and it ends with umount returning an error. (An error in any stage causes pi-gen to bail, leaving filesystems mounted and other issues, but that's a slightly more complex issue.) The prerun.sh is failing in its final line to unmount_image ${IMG_FILE}. At first glance, IMG_FILE was unmounted, but a second look spots the kpartx call which would've placed the image back in use.

Again, this is a problem I see in RPi-Distro's tree, however I'm going to use my own tree to try and diagnose the problem because it has a completely populated work/ from today, resulting in a 15 minute build instead of a 70-90 minute build depending on disk caching.

My debug procedure mostly consists of running build.sh with the following patch:

diff --git a/export-noobs/prerun.sh b/export-noobs/prerun.sh
index a531be6..bde31b7 100755
--- a/export-noobs/prerun.sh
+++ b/export-noobs/prerun.sh
@@ -1,4 +1,4 @@
-#!/bin/bash -e
+#!/bin/bash -ex

 IMG_FILE="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img"
 NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}"

Relevant output follows:

[00:35:45] End /home/tjcarter/Source/RasppleII/pi-gen/stage4
[00:35:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image
[00:35:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/prerun.sh
mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
mke2fs 1.43.3 (04-Sep-2016)
mount: /dev/mapper/loop1p2 mounted on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/rootfs.
mount: /dev/mapper/loop1p1 mounted on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/rootfs/boot.
[00:35:57] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/prerun.sh
[00:35:57] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun
[00:35:57] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun/00-run.sh
[00:35:57] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun/00-run.sh
[00:35:57] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun
[00:35:57] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources
[00:35:57] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches
Converting meta-data to version 2
Applying patch /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches/0-sources.diff
patching file rootfs/etc/apt/sources.list.d/raspi.list

Now at patch /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches/0-sources.diff
[00:35:57] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches
[00:35:57] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/01-run.sh
Hit http://archive.raspberrypi.org jessie InRelease
Get:1 http://mirrordirector.raspbian.org jessie InRelease [14.9 kB]
Hit http://archive.raspberrypi.org jessie/main armhf Packages 
Get:2 http://mirrordirector.raspbian.org jessie/main armhf Packages [8981 kB]
Hit http://archive.raspberrypi.org jessie/ui armhf Packages           
Ign http://archive.raspberrypi.org jessie/main Translation-en          
Ign http://archive.raspberrypi.org jessie/ui Translation-en
Get:3 http://mirrordirector.raspbian.org jessie/contrib armhf Packages [37.5 kB]          
Get:4 http://mirrordirector.raspbian.org jessie/non-free armhf Packages [70.3 kB]         
Get:5 http://mirrordirector.raspbian.org jessie/rpi armhf Packages [1356 B]               
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en                      
Ign http://mirrordirector.raspbian.org jessie/main Translation-en                         
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en                     
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en                          
Fetched 9105 kB in 20s (449 kB/s)                                                         
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[00:36:45] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/01-run.sh
[00:36:45] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources
[00:36:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network
[00:36:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network/01-run.sh
[00:36:45] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network/01-run.sh
[00:36:45] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network
[00:36:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise
[00:36:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise/01-run.sh
Stopping fake hwclock: saving system time.
Mode:     real
Files:    2228
Linked:   326 files
Compared: 0 xattrs
Compared: 568 files
Saved:    9.13 MiB
Duration: 0.19 seconds
14081/119135/321792
zip /home/tjcarter/Source/RasppleII/pi-gen/deploy/image_2016-09-16-RasppleII-lite.zip /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/2016-09-16-RasppleII-lite.img
  adding: 2016-09-16-RasppleII-lite.img (deflated 78%)
[00:37:24] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise/01-run.sh
[00:37:24] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise
[00:37:24] End /home/tjcarter/Source/RasppleII/pi-gen/export-image
[00:37:24] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image
[00:37:24] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/prerun.sh
mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
mke2fs 1.43.3 (04-Sep-2016)
mount: /dev/mapper/loop1p2 mounted on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/rootfs.
mount: /dev/mapper/loop1p1 mounted on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/rootfs/boot.
[00:39:06] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/prerun.sh
[00:39:06] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun
[00:39:06] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun/00-run.sh
[00:39:06] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun/00-run.sh
[00:39:06] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun
[00:39:06] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources
[00:39:06] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches
Converting meta-data to version 2
Applying patch /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches/0-sources.diff
patching file rootfs/etc/apt/sources.list.d/raspi.list

Now at patch /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches/0-sources.diff
[00:39:06] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches
[00:39:06] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/01-run.sh
Hit http://archive.raspberrypi.org jessie InRelease
Get:1 http://mirrordirector.raspbian.org jessie InRelease [14.9 kB]
Hit http://archive.raspberrypi.org jessie/main armhf Packages 
Get:2 http://mirrordirector.raspbian.org jessie/main armhf Packages [8981 kB]
Hit http://archive.raspberrypi.org jessie/ui armhf Packages            
Ign http://archive.raspberrypi.org jessie/main Translation-en          
Ign http://archive.raspberrypi.org jessie/ui Translation-en
Get:3 http://mirrordirector.raspbian.org jessie/contrib armhf Packages [37.5 kB]          
Get:4 http://mirrordirector.raspbian.org jessie/non-free armhf Packages [70.3 kB]         
Get:5 http://mirrordirector.raspbian.org jessie/rpi armhf Packages [1356 B]               
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en                      
Ign http://mirrordirector.raspbian.org jessie/main Translation-en                         
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en                     
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en                          
Fetched 9105 kB in 18s (483 kB/s)                                                         
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[00:39:52] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/01-run.sh
[00:39:52] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources
[00:39:52] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network
[00:39:52] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network/01-run.sh
[00:39:52] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network/01-run.sh
[00:39:52] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network
[00:39:52] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise
[00:39:52] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise/01-run.sh
Stopping fake hwclock: saving system time.
Mode:     real
Files:    6984
Linked:   1348 files
Compared: 0 xattrs
Compared: 3866 files
Saved:    36.81 MiB
Duration: 0.61 seconds
35687/105480/964096
zip /home/tjcarter/Source/RasppleII/pi-gen/deploy/image_2016-09-16-RasppleII.zip /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/2016-09-16-RasppleII.img
  adding: 2016-09-16-RasppleII.img (deflated 65%)
[00:42:09] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise/01-run.sh
[00:42:09] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise
[00:42:09] End /home/tjcarter/Source/RasppleII/pi-gen/export-image
[00:42:09] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-noobs
[00:42:09] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-noobs/prerun.sh
+ IMG_FILE=/home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
+ NOOBS_DIR=/home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII
+ unmount_image /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
+ sync
+ sleep 1
++ losetup -j /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
++ cut -f1 -d:
+ local LOOP_DEV=/dev/loop0
+ '[' -n /dev/loop0 ']'
++ mount
+++ basename /dev/loop0
+++ head -n 1
+++ cut -f 3 -d ' '
++ grep loop0
+ local MOUNTED_DIR=
+ '[' -n '' ']'
+ sleep 1
+ kpartx -ds /dev/loop0
+ losetup -d /dev/loop0
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs
+ cp /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/2016-09-16-RasppleII.img /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/
+ rm -rf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII
++ kpartx -asv /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
++ grep -E -o -m1 'loop[[:digit:]]+'
++ head -n 1
+ LOOP_DEV=loop0
+ BOOT_DEV=/dev/mapper/loop0p1
+ ROOT_DEV=/dev/mapper/loop0p2
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII
+ mount /dev/mapper/loop0p2 /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
+ mount /dev/mapper/loop0p1 /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-Raspple$
I/export-noobs/bootfs
+ tar -I pxz -C /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-no$
bs/bootfs -cpf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noo$
s/2016-09-16-RasppleII/boot.tar.xz .
+ tar -I pxz -C /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noo
bs/rootfs -cpf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noob
s/2016-09-16-RasppleII/root.tar.xz .
+ unmount_image /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noo
bs/2016-09-16-RasppleII.img
+ sync
+ sleep 1
++ losetup -j /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs
/2016-09-16-RasppleII.img
++ cut -f1 -d:
+ local LOOP_DEV=/dev/loop0
+ '[' -n /dev/loop0 ']'
++ mount
+++ basename /dev/loop0
+++ head -n 1
+++ cut -f 3 -d ' '
++ grep loop0
+ local MOUNTED_DIR=
+ '[' -n '' ']'
+ sleep 1
+ kpartx -ds /dev/loop0
+ losetup -d /dev/loop0
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs
+ cp /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/2016-09-16-RasppleII.img /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/
+ rm -rf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII
++ kpartx -asv /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
++ grep -E -o -m1 'loop[[:digit:]]+'
++ head -n 1
+ LOOP_DEV=loop0
+ BOOT_DEV=/dev/mapper/loop0p1
+ ROOT_DEV=/dev/mapper/loop0p2
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII
+ mount /dev/mapper/loop0p2 /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
+ mount /dev/mapper/loop0p1 /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs
+ tar -I pxz -C /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs -cpf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII/boot.tar.xz .
+ tar -I pxz -C /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs -cpf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII/root.tar.xz .
+ unmount_image /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
+ sync
+ sleep 1
++ losetup -j /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
++ cut -f1 -d:
+ local LOOP_DEV=/dev/loop0
+ '[' -n /dev/loop0 ']'
++ mount
+++ basename /dev/loop0
+++ head -n 1
+++ cut -f 3 -d ' '
++ grep loop0
+ local 'MOUNTED_DIR=/dev/mapper/loop0p2 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs type ext4 (rw,relatime,data=ordered)
/dev/mapper/loop0p1 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)'
+ '[' -n '/dev/mapper/loop0p2 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs type ext4 (rw,relatime,data=ordered)
/dev/mapper/loop0p1 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)' ']'
++ dirname /dev/mapper/loop0p2 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs type ext4 '(rw,relatime,data=ordered)' /dev/mapper/loop0p1 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs type vfat '(rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)'
+ unmount /dev/mapper . /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs . . . /dev/mapper . /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs . . .
+ '[' -z /dev/mapper ']'
+ DIR=/dev/mapper
+ mount
+ grep -q /dev/mapper
++ mount
++ grep /dev/mapper
++ cut -f 3 -d ' '
++ sort -r
+ local 'LOCS=/home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
/home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs/'
+ for loc in '$LOCS'
+ umount /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
+ for loc in '$LOCS'
+ umount /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs
+ for loc in '$LOCS'
+ umount /
umount: /: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)

real    14m32.352s
user    35m18.548s
sys     0m33.020s

As you can see, unmount_image is doing all kinds of not intended things there.

Remove dependency on kernel support for binfmt

In order to build for docker, the underlying kernel has to have support for binfmt, which is required by qemu-user-static. On Docker for mac, this works fantastically because the kernel already has the required support. However, when attempting to run on a fresh Debian VPS many kernels don't have this support.

I'm not sure if there's an easy solve. Many on the internet have suggested to use a version of qemu with the QEMU_EXECVE set so compiling for ARM doesn't escape out and cause the build to fail. A special docker image has been created that uses this prebuilt special version of qemu but I'm not sure how much use this is in this context.

Attempting to install qemu-user-static produces the following output:

Setting up qemu-user-static (1:2.1+dfsg-12+deb8u6) ...
update-binfmts: warning: Couldn't load the binfmt_misc module.
update-binfmts: warning: Couldn't load the binfmt_misc module.
...
update-binfmts: warning: Couldn't load the binfmt_misc module.
Setting up quilt (0.63-3) ...

And some people on circleci have documented similar behaviour as a failed result of cross-compilation for ARM architectures.

The actual error with running IMG_NAME=example ./build.sh occurs during the initial file extractions:

...
I: Extracting zlib1g...
I: Running command: chroot /root/pi-gen/work/2017-02-07-tiny/stage0/rootfs /debootstrap/debootstrap --second-stage
chroot: failed to run command '/debootstrap/debootstrap': Exec format error

While this isn't an issue inherent to pi-gen, it would be good to remove the dependency on underlying kernel support that isn't always present.

include usb-modeswitch in default install

This is required to get usb dongles internet devises like Tata Photon working.

http://www.tataphoton.com/photon-max-wifi.aspx its a very common method to get internet connection in India. It is included in desktop images of debian.

Trouble building when "pi-gen/work" dir is not a symlink after fix has been applied

Git commit 40b5af9 seems to be causing issues for me.

➜ ~ git clone https://github.com/RPi-Distro/pi-gen
Cloning into 'pi-gen'...
remote: Counting objects: 567, done.
remote: Compressing objects: 100% (61/61), done.
remote: Total 567 (delta 20), reused 0 (delta 0), pack-reused 506
Receiving objects: 100% (567/567), 641.79 KiB | 302.00 KiB/s, done.
Resolving deltas: 100% (193/193), done.
Checking connectivity... done.
➜ ~ cd pi-gen
➜ pi-gen git:(dev) sudo IMG_NAME="test" ./build.sh
mkdir: missing operand
Try 'mkdir --help' for more information.

The problem seems to be that WORK_DIR is now returning empty:

export WORK_DIR=`readlink -f "${BASE_DIR}/work/${IMG_DATE}-${IMG_NAME}"`

This was a fix that was added for symlink support, but is now interfering with my ability to run pi-gen without a symlinked work directory.

Repeated use of build-docker.sh eats disk space extremely fast

Each use of build-docker.sh leaves behind tens of GBs of files in /var/lib/docker which are never deleted, not even with docker rm pigen_work.

This eats up disk space extremely fast, wasting easily hundreds of GB, with no way to get rid of these files (except sudo rm -rf which might destroy integrity of docker).

Paths with spaces do not work

Upon inspecting the code and testing the build system, there are many many places in several scripts where if the current path (pwd) has spaces in it, the script will fail.
These should be cleaned up so we don't have to rely on the pwd having no spaces.

I have seen talk of changing the build script to python, but I have seen the dependency check script fail too. Looks like the whole build system needs to be modified to handle such paths.

First boot configuration file/script

This'd be a relatively major feature, and I know it's been suggested before but I'm unsure if there's an intentional reason it hasn't been implemented (and if so feel free to ignore the rest of this proposal).

Description of the problem

There are many reasons this would be beneficial, but the most commonly cited issue is configuring a Raspberry Pi through just the use of wifi. As it stands right now, you'd have to do one of the following:

• Compile your own image with the wifi configuration baked in
• Use a third party utility that relies on custom images such as PiBakery
• Connect ethernet on initial boot to add appropriate wifi configuration
• Use an OS with write support for ext4 (neither Mac or Windows can do this natively)

All of the above solutions have frustrating aspects to them; a native solution that requires no third party utilities or faffing around with ethernet cables would be ideal.

Feature proposal

In essence, a configuration file/script placed in the boot partition of the SD card. It's FAT which means that almost every operating system in existence can read and write to it natively.

Much like with the empty ssh file, this would only take effect on first boot (thus hopefully eliminating many of the security concerns with having such a configuration file).

File specifics

This could theoretically be implemented in two ways:

1. A config.json, config.txt or config.yml with structured config data
2. A config.sh of some kind that simply executes on first boot

It could be either/or, as there are advantages and disadvantages to both. A structured config file would be easier to setup and have fewer security concerns (with the limited number of properties that could be modified), but with the greater overhead of having to parse the file contents.

Likewise a config.sh would be slightly harder to configure but offer the user much greater customisability over their Pi. The security concerns would be slightly more-so but if the user takes care to make sure it's freshly written image (with validated checksums) it's unlikely it could be used maliciously. It'd be fast though.

Perhaps the option could be allowed for both, so that a user can take whichever approach they are most comfortable with.

Implementation

Placing a file that executes the script (or parses the structured config data) in /etc/rc.local or /etc/init.d/... and then deletes itself so that it is guaranteed to only run once would probably be the best way to go.

---

I personally think this would be a fantastic feature to have natively, and would eliminate a lot of headache with setting up a new Raspberry Pi. I would be more than happy to do all of the scripting, implementation and documentation for it myself. Just wanted to run it by you and see what your thoughts were before attempting 😄

Error --> sudo: unable to resolve host hassbian

The hostname file is correct but the hosts file should be updated to match. The last line in hosts should be 127.0.1.1 hassbian and that will eliminate the sudo: unable to resolve host hassbian error.

Thank you!
Jon

pi@hassbian:~ $ sudo systemctl restart [email protected]
sudo: unable to resolve host hassbian

pi@hassbian:~ $ cat /etc/hostname
hassbian

pi@hassbian:~ $ cat /etc/hosts
127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters

127.0.1.1	raspberry

pi@hassbian:~ $ sudo nano /etc/hosts

pi@hassbian:~ $ cat /etc/hosts
127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters

127.0.1.1	hassbian

pi@hassbian:~ $ 

 

EDIT: This is for the 2017-02-03-HASSbian.img

Suggest: Comments in packages files

Currently ##-packages and ##-packages-nr files are just lists of packages to include, one or more per line. Might it be useful to have comments in these files for later maintenance to indicate why packages are there, or to provide some logical grouping?

Here's what I have in mind:

diff --git a/build.sh b/build.sh
index 5b745e8..84daeda 100755
--- a/build.sh
+++ b/build.sh
@@ -16,7 +16,7 @@ EOF
        fi
        if [ -f ${i}-packages-nr ]; then
            log "Begin ${SUB_STAGE_DIR}/${i}-packages-nr"
-           PACKAGES=`cat $i-packages-nr | tr '\n' ' '`
+           PACKAGES=`sed -e '/[[:space:]]*#.*$/d' < ${i}-packages-nr | tr '\n' ' '`
            if [ -n "$PACKAGES" ]; then
                on_chroot sh -e - << EOF
 apt-get install --no-install-recommends -y $PACKAGES
@@ -26,7 +26,7 @@ EOF
        fi
        if [ -f ${i}-packages ]; then
            log "Begin ${SUB_STAGE_DIR}/${i}-packages"
-           PACKAGES=`cat $i-packages | tr '\n' ' '`
+           PACKAGES=`sed -e '/[[:space:]]*#.*$/d' < ${i}-packages | tr '\n' ' '`
            if [ -n "$PACKAGES" ]; then
                on_chroot sh -e - << EOF
 apt-get install -y $PACKAGES

That will strip out anything starting with # on a line (and leading whitespace). This will allow stuff like:

# X.org compositing effects for LXDE
libgl1-mesa-dri libgles1-mesa libgles2-mesa xcompmgr

Scripts can explain what they're doing so that the process is understood by people looking at it later on. Right now, packages files cannot. They ought to, just so that they can explain WHY certain packages are being installed and perhaps sort packages into logical groups.

The alternative to doing this would be to have multiple parts of any given stage that install packages, each named for the bundle of packages being installed. That's semantically quite possible and quite useful, however it's pragmatically a mess simply because Debian has approximately 2^512 packages (only slightly exaggerated) in the database which must be processed every time you run apt-get.

Processing databases takes so long in Debianish systems nowadays that sid's version of aptitude now asks you after finishing an install if you want to quit immediately, without bothering to grind through the package list to regenerate the state graph for the UI before allowing you to do so. It's ugly and inelegant, but even on fast systems it's become kinda welcome.

The comment removal is fast and orders of magnitude faster than running multiple instances of apt-get when not necessary.

Probably worthwhile to make sed also do the tr stage above, but I didn't bother for comments on the suggestion. :)

Way to specify which stage to run to?

Had a brief look over build.sh and was wondering if there was a good way to execute the build script up to a certain stage (stage 2, to get just the lite system, for example).

packages cannot be authenticated during stage1/03-install-packages/00-packages

When I run sudo IMG_NAME=test.img ./build.sh I get an error during 03-install-packages of stage1. I can add --force-yes to the apt-get in build.sh and the error goes away, but it is probably not the safest solution. Is there something else wrong?

Snippet of stdout from the above command.

[...]
[20:25:35] End /home/pi/pi-gen/stage1/02-net-tweaks/00-run.sh
[20:25:36] End /home/pi/pi-gen/stage1/02-net-tweaks
[20:25:36] Begin /home/pi/pi-gen/stage1/03-install-packages
[20:25:36] Begin /home/pi/pi-gen/stage1/03-install-packages/00-packages
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  alsa-utils device-tree-compiler libasound2 libasound2-data libparted2 libsamplerate0 lua5.1 parted triggerhappy
Suggested packages:
  libasound2-plugins libparted-dev libparted-i18n parted-doc
The following NEW packages will be installed:
  alsa-utils device-tree-compiler libasound2 libasound2-data libparted2 libraspberrypi-bin libraspberrypi0 libsamplerate0 lua5.1 parted raspi-config triggerhappy
0 upgraded, 12 newly installed, 0 to remove and 0 not upgraded.
Need to get 4384 kB of archives.
After this operation, 11.4 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  libparted2 libsamplerate0 alsa-utils lua5.1 parted triggerhappy
E: There are problems and -y was used without --force-yes

Here is some other information that may or may not be related.

pi@raspberrypi:~ $ cat /etc/apt/sources.list
deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
deb-src http://archive.raspbian.org/raspbian/ jessie main contrib non-free rpi
pi@raspberrypi:~ $ sudo apt-key update
gpg: key 90FDDD2E: "Mike Thompson (Raspberry Pi Debian armhf ARMv6+VFP) <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Warning: removed keys keyring  /usr/share/keyrings/raspbian-archive-removed-keys.gpg missing or not readable
pi@raspberrypi:~ $ sudo apt-get update
Hit http://mirrordirector.raspbian.org jessie InRelease
Hit http://archive.raspbian.org jessie InRelease
Hit http://archive.raspberrypi.org jessie InRelease
Hit https://dl.yarnpkg.com stable InRelease
Get:1 http://archive.raspbian.org jessie/main Sources [7,746 kB]
Get:2 http://archive.raspberrypi.org jessie/main armhf Packages [130 kB]
Hit http://archive.raspberrypi.org jessie/ui armhf Packages
Get:3 http://archive.raspbian.org jessie/contrib Sources [64.5 kB]
Get:4 http://archive.raspbian.org jessie/non-free Sources [113 kB]
Get:5 http://archive.raspbian.org jessie/rpi Sources [1,132 B]
Ign http://archive.raspberrypi.org jessie/main Translation-en_GB
Ign http://archive.raspberrypi.org jessie/main Translation-en
Ign http://archive.raspberrypi.org jessie/ui Translation-en_GB
Ign http://archive.raspberrypi.org jessie/ui Translation-en
Hit http://mirrordirector.raspbian.org jessie/main armhf Packages
Hit http://mirrordirector.raspbian.org jessie/contrib armhf Packages
Hit http://mirrordirector.raspbian.org jessie/non-free armhf Packages
Hit http://mirrordirector.raspbian.org jessie/rpi armhf Packages
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en
Ign http://mirrordirector.raspbian.org jessie/main Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/main Translation-en
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en
Fetched 8,054 kB in 3min 37s (37.1 kB/s)
W: Failed to fetch https://dl.yarnpkg.com/debian/dists/stable/InRelease  Unable to find expected entry 'main/binary-armhf/Packages' in Release file (Wrong sources.list entry or malformed file)

W: Failed to fetch http://archive.raspberrypi.org/debian/dists/jessie/main/binary-armhf/Packages  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

Building forward from Stage 2

I'm looking to go forward from a lite system toward a server-type setup (so, not a graphical direction). Is the right way to do this in pi-gen to just remove the existing Stage 3/4 and build a new Stage 3 that installs what I need with the config I'm looking for, or am I thinking about this wrong?