docker-yum-mirror

Builds yum package mirrors for you in a containers via a yaml formatted config file.

Why?

In a few environments I've worked in, we've needed local yum mirrors, but needed to not only have up-to-date copies of them, but also point-in-time snapshots of some/all of them (HPC systems, kernel bound filesystems, etc. Basically, be able to bring up 1000s of systems "updated" to the exact same version of all RPMS.)

So, this container + script will help do this. It takes a config file and a directory mounted into it, and supports mirrors using rsync or reposync (and thus, anything yum itself can use).

Features

Supports rsync or reposync
Can create a large 'all' repo (all pacakges form all repos, smushed together + generated repodata)
This 'all' repo can be snapshot'ed, to remain frozen in time.
Likewise, individual repos can be snapshot'd, freezing them in time.
These 'snaps' are datestamped, and created (optionally) via hardlinks, to be space efficient.
hardlink (the program) is also used to file-level de-dupe your repos.

A resulting directory structure may look like:

- /mirror/
|__ centos-7-x86_64/
  |__all
  |__all.2016-06-14
  |__os
  |__updates
|__ centos-6_x86_64/
  ...

all would be all the rpms from os + updates, as hardlinks. all.2016-06-14 would be what all looked like on that date, all hardlinks to save space. os and updates would be mirrors of the upstream repos.

In production i would, most likley, enable all+datestamp_all, do a sync, then disable then until i wanted to make another snapshot. subsequent runs would sync upstream, but leave teh datestamped 'all' directory as is.

Usage

docker run -v /path/to/storage:/mirror -v /path/to/config.yaml:/config.yaml sjoeboo/docker-yum-mirror:latest

config example:

---
:hardlink: true
:hardlink_dir: '/mirror'
:all: true
:all_name: 'all'
:datestamp_all: true
:mirror_base: '/mirror'
mirrors:
  os:
    :dist: 'centos-7-x86_64'
    :type: 'rsync'
    :url: 'rsync://mirrors.kernel.org/centos/7/os/x86_64/'
  extras:
    :dist: 'centos-7-x86_64'
    :type: 'rsync'
    :url: 'rsync://mirrors.kernel.org/centos/7/extras/x86_64/'
    :datestamp: true
    :hardlink_datestamp: true
  plus:
    :dist: 'centos-7-x86_64'
    :type: 'reposync'
    :url: 'http://mirrors.kernel.org/centos/7/centosplus/x86_64/'
    :dest: '/some/other/location/'
    :datestamp: true
    :link_datestamp: true

Lets dive into the above config a little, it would:

Create mirrors in /mirror/centos-7-x86_64 (since centos-7-x86_64 is the dist for the mirrors listed)
Except for plus which would be created elsewhere (dist the then ignored)
plus would, after being sync'd, be moved to plus.YYYY-MM-DD, with plus becoming a symbloic link to plus.YYYY-MM-DD
extras would, after getting sync'd, have extras.YYYY-MM-DD created, as a tree of hardlinks back to extras. You could do this on multiple days to have multiple space efficent snapshots for extras as well as track upstream.
additionally, a all repo, named all would be created, containing al of the rpms from each dist listed. It would also have a hardlink tree created all.YYYY-MM-DD
finally, the hardlink program would be run on ``/mirror` to file-level de-dup the rpms.

The above config is....silly. I can't think of why one would want to datestamp all AND individual repos, but, you could. I would either datestamp all, or, datestamp individual repos and not create an all whatsoever.

options

:hardlink: Boolean. Should we run hardlink at the end to try to find duplicates. Default: true :hardlink_dir: String. What directory to run hardlink on Default: /mirror :all: true Boolean. Create an 'all' repo. Default: true :all_name: String. name for all repo. Default: all :datestamp_all: Boolean. To make a hardlinked, datestamped copy of 'all'. Default True. :mirror_base: String. Base directory to use to create destinations, dist will be appended. Default: /mirror :mirrors: Hash. List of mirrors to create. Format/parameters are:

name: (name of repo, will be appended to mirror_base + dist unless dest is specified)
  :dist: (distribution, whatever you want, basically, a grouping of repos. Appended to mirror_base unless dest is specified. Also how 'all' is created/grouped.)
  :type: (rsync or reposync)
  :url: (url to sync from. rsync:// for rsync, something yum supports for reposync)
  :datestamp: (Boolean, should a datestamped copy be made)
  :hardlink_datestamp: (Boolean, should that copy be made of hardlinks)
  :link_datestamp: (Should the original repo be turned into a link to the most recent datestamped version)

sjoeboo / docker-yum-mirror Goto Github PK