dvandok / userspace-package-management

Handle packages for grid systems entirely in user space, based on NetBSD pkgsrc

User space package management

The PoC (Proof of Concept / Packages of Choice) has always been a centrally provided collection of packages for end users. The alternative–user space package management– allows the VO software managers to handle VO specific software installations per site, thus gaining control over versions and build parameters.

The pkgsrc system, by NetBSD, is being tried as the basic packaging toolset for managing updates, dependencies, etc.

Not all science software is readily available in pkgsrc, but with some work any regular package that follows the usual configure/make/make install mantra can be incorporated.

Extra tooling around pkgsrc is needed to interact with Grid systems, because software management is done by sending grid jobs with a software manager proxy (to gain the appropriate user mapping), and all packaging/updating/etc work needs to be done inside the grid job.

Grid jobs

The general layout of the grid jobs is as follows:

• test if we have the right permissions in the VO_SW_AREA (and set umask 002)
• test for the presence of pkgsrc
  • if not present: fetch and install
  • if present: upgrade if necessary
• run the vulnerability admin tool
• see to the actual request (install, remove, etc.)

management and reporting

As a VO software manager you are interested to keep track of all the installed versions of packages on all the available sites. So one level up from the pkgsrc-cmd.sh scripts run in grid jobs is a set of tools to automate the submission of jobs to a known collection of resources, and generating aggregated reports.

On top of this we could have a service that periodically runs these tools and maintains and serves the generated reports.

Design of get-site-status.sh

This script will probe sites to find out the status of pkgsrc installations. It will launch pkgsrc-cmd jobs, cycle to wait on the results, and collect the output in a human-readable form.

Some design principles:

• the script is meant to be run as a batch job, so it can be used in a crontab file.
• the output can be text, html, or otherwise
• resources are listed in a data file,
• credentials to run the job under are presumed to be available. The user is responsible for keeping credentials up-to-date.
• only one job per site should be run at a time; this goes especially for the jobs that install packages as simultaneous writes may result in corruption of the database.