azimuth-cloud / azimuth-images

azimuth-images

This repository contains Packer template definitions for building images for use with Azimuth.

The templates work by provisioning machines in an OpenStack project and configuring them using Ansible before converting to an image. These images are then downloaded from OpenStack and uploaded to S3 for distribution in an Azimuth release.

Building an image manually

Before building an image, you must configure OpenStack authentication, using either an RC file or clouds.yaml, e.g.:

export OS_CLOUD=openstack

This repository has a concept of "environments", allowing builds to be customised for different clouds if required, for example in a fork. These manifest as directories under env, where environment are set that in turn can specify Packer var files under var.

Currently, the repository contains a base environment with cloud-agnostic configuration and an arcus environment that contains configuration for the continuous integration environment, which is kindly hosted on the Arcus cloud at the University of Cambridge.

To build images manually, outside of CI, first ensure you have Packer installed and the required providers installed:

packer init ./config.pkr.hcl

Next, create a Python virtual environment to use and install the Python and Ansible dependencies:

python -m venv .venv
source .venv/bin/activate

pip install -r ./requirements.txt

ansible-galaxy install -f -r requirements.yml

Finally, set up an environment using the arcus environment for inspiration.

You can then build images using the ./bin/build-image script. This script expects the following environment variables to be set:

• ENVIRONMENT - the environment to use
• PACKER_TEMPLATE - the Packer template in packer to use
• ENV_VAR_FILES - the environment variable files to include

The values for PACKER_TEMPLATE and ENV_VAR_FILES can be determined by looking at the defined builds.

For example, to build a workstation image, the following command can be used:

ENVIRONMENT=myenv \
PACKER_TEMPLATE=linux-desktop \
ENV_VAR_FILES=common,kvm,linux,ubuntu-jammy \
./bin/build-image

This will result in a private image being built in the target OpenStack project.

azimuth-images's Issues

Add apptainer to workstation

repo2docker version 2024.03.0 breaks appliance

The latest version of repo2docker - 2024.03.0 - breaks the repo2docker appliance with a Permission denied error.

For now, we pin to a known working version.

ubuntu@test-4r6nq-repo2docker:~$ sudo -u podman -i jupyter-repo2docker --image-name repo2docker-notbook-server --engine podman --no-run https://github.com/binder-examples/bokeh.git
Picked Git content provider.
Cloning into '/tmp/repo2dockerf_v0yv88'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 13 (delta 0), reused 9 (delta 0), pack-reused 0
Receiving objects: 100% (13/13), 15.71 KiB | 3.14 MiB/s, done.
Python version unspecified, using current default Python version 3.10. This will change in the future.Building conda environment for python=3.10
Using CondaBuildPack builder
Traceback (most recent call last):
  File "/usr/local/bin/jupyter-repo2docker", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/repo2docker/__main__.py", line 474, in main
    r2d.start()
  File "/usr/local/lib/python3.10/dist-packages/repo2docker/app.py", line 892, in start
    self.build()
  File "/usr/local/lib/python3.10/dist-packages/repo2docker/app.py", line 856, in build
    for l in picked_buildpack.build(
  File "/usr/local/lib/python3.10/dist-packages/repo2docker/buildpacks/base.py", line 671, in build
    yield from client.build(**build_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/repo2podman/podman.py", line 459, in build
    tarf.extractall(builddir)
  File "/usr/lib/python3.10/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.10/tarfile.py", line 2324, in _extract_one
    self._handle_fatal_error(e)
  File "/usr/lib/python3.10/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.10/tarfile.py", line 2403, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "/usr/lib/python3.10/tarfile.py", line 2448, in makefile
    with bltn_open(targetpath, "wb") as target:
PermissionError: [Errno 13] Permission denied: '/tmp/tmpw097iipk/src/.git/objects/pack/pack-a06434fcada51ff585872da911f357fc71f912e5.idx'

Add podman-compose to workstation

Prune images not part of a release

We should prune images from S3 that are not part of a release.

Kubernetes Jammy images not working with Harbor

CVMFS debs left behind in the home dir

Ideally we need to clean these up, not leave them behind:

ubuntu@johng-workstation:~$ ls
Desktop    Downloads  Pictures  Templates  cvmfs-config-eessi_latest_all.deb
Documents  Music      Public    Videos     cvmfs-release-latest_all.deb