clearlinux / clr-distro-factory Goto Github PK
View Code? Open in Web Editor NEWClear Linux* Distro Factory
Clear Linux* Distro Factory
Delta packs and delta manifests (from here on 'deltas' refers to both packs and manifests) should be generated for the past 10 builds to give users quick updates from releases over the past week or two.
mixer build delta-packs --previous-versions 10
mixer build delta-manifests --previous-versions 10
Additionally, for systems that are on a very old version and need to update across format bumps, need to always generate deltas from the first build in the current format to help users get to the current build faster.
mixer build delta-packs --from ${FIRST_BUILD}
mixer build delta-manifests --from ${FIRST_BUILD}
When working in a full DevOps environment, a git repository projects/common will be used. This contains a packages file that keeps track of the current packages for the distro. This needs updated as a part of every build to ensure that when developers do a make pull/make clone from their tooling, they are able to automatically clone new packages. This should be done as a part of the publishing step, and be done if a config repo for the URL to this repository is set to allow operation without a full DevOps setup.
If the sign_image function is defined, the image certificate needs to be placed in the staged config folder alongside the Swupd_Root.pem.
A file named os-core-update-extra-files
needs to be created in mixer's server state directory containing the paths that need to be added to the os-core-update manifest. These files need to be created with the proper permissions inside mixer's full chroot. This needs done after mixer build bundles
but before mixer build update
. Without this, users are not able to perform swupd update
from the content that mixer creates when custom update signing is used.
At a minimum, need the following:
/usr/share/clear/update-ca
/usr/share/clear/update-ca/Swupd_Root.pem
This is because swupd-client has an option in it's configure.ac for:
default_cert_path="/usr/share/clear/update-ca/Swupd_Root.pem"
Most use cases will have some sort of QA, so the breakpoint would be required. This don't need to be in replacement to the workflow definition we have today. We can keep both options and user can pick the one.
Update our CI files to github actions that is performing much better.
To allow for one to maintain a Downstream that is just a subset of Upstream.
Today, it can be done on Distro-Factory side by handling the bundles repository to fit mixer needs. But the most correct approach is for mixer to support the same layout used by mainline (upstream) Clear Linux so we don't have to move files for nothing.
As a part of release note generation, bundle changes for the local bundles added to the mix should be shown. This means showing added, updated, and removed bundles. For now, this should only apply to the bundles for the particular distribution. If a downstream, it should not show changes for upstream bundles. While an improvement for the future, these may be resolved by looking at the release notes of the upstream on which the distribution is bases.
Content needs to be able to be staged on a remote machine. Rather than just moving content into a new location, it should be rsync'd to a path. This path may be local, or it may be another machine. This allows for more complex infrastructure setups and requirements when dealing with larger amounts of data that need mirrored as a part of the publishing process.
A docker folder should be made in the config repository containing folders for each image name. Each image folder name should have the following files inside of it:
This data should be used as input to a process that builds a Docker image from scratch. This process should use the following swupd command to create the build context:
sudo -E swupd os-install --wait-for-scripts --path="$CHROOT_DIR" --version="$VER" --statedir="$STATE_DIR" --format="$(< "$SWUPDWEBDIR/$VER/format")" --url=file://"$SWUPDWEBDIR" --no-boot-update --bundles "$(cat "$TYPE_BUNDLE_FILE" | tr '\n' ',' | sed 's/,$//')"
Variables for these guys needs to be defined, but if they exist, they should be staged into the following directory structure:
repo/
x86_64/
os/
Packages/
repodata/
debug/*
repodata/
source/*
SRPMS/
repodata/
* Indicates missing directories
There are one or two find command that become expensive as debuginfo build up over time. If we know the packages from the previous build, we might be able to identify which debuginfo packages have changed and generate debuginfo only for these in a staging directory, and then merge it into the final directory.
Prior to doing releases of tools on which the clr-distro-factory build process is dependent, those tools should be tested in this environment to identify any changes needed to the build process to support their release.
Given clr-distro-factory enables reproducible testing in namespaced environments on the same machine (or even running on a separate machine all together), we want to provide a test plan for these tools that if their updated versions allow for the build process to complete successfully, they may be released.
Given the integration with GitHub we have with clr-distro-factory now, I'm not sure if this is something that may be done as Travis checks on our dependent tools. I'm thinking the more automation, the better since we want to always make sure this is tested.
Today when packages are published, all ~85GB of content is copied and synced for each build. Over time, this accumulates a high amount of disk space and requires a robust hardlinking strategy for proper management. Only new packages that have been added from the last build should be synced out to a common location, a cache, and existing packages should remain in that cache.
Want to consider using a run_and_log
approach for these commands because the console output is too large to go through most times. For mixer and clr-installer, we want to archive the logs in the Jenkins job so they may be referenced in the future.
In order to create a wider array of images with special requirements, need to generalize post processing of image files. Rather than making assumptions about compression, make this step be a part of a finalize script. The finalize script may also be responsible for running qemu img-convert
on the resulting image to prepare it for publishing into the marketplace of various cloud providers. Or it may post-process the content in the chroot to create a PXE image. Any number of changes are possible and can be associated with the config of the image. This allows images to be easily created in a standalone manner with the clr-installer outside of clr-distro-factory and aligns to an issue open with clr-installer: clearlinux/clr-installer#531. Once clr-installer closes this issue, this code will need removed from clr-distro-factory, but should be maintained until then.
SimpleHTTPServer module has been merged into http.server in Python 3.0 which supports by Clear by default. I create a pull #30, please help to review :-)
Add header to license file that contains the following text to make it more clear as to what users are looking at:
# This list is automatically generated.
#
# To read the full license text for these licenses, please visit
# https://spdx.org/licenses. A few license identifiers in this list
# are not declared on the https://spdx.org/licenses website:
# - Copyright
# - Distributable
# - Public-Domain
#
# These are used for projects that have explicitly granted redistribution
# of the project source code, but don't have a typical OSI-approved
# license identifier.
SimpleHTTPServer was deprecated in favor of module http.server in Python 3.
I started from 27000 (format 26), and 27190 (format 26) ,and 27200 (format 26), all successful.
But It failed like when I tried 27210 (format 27).
My clear version on the build machine is 29280.
The failure log as below.
== Build 'Update' Content ==
The upstream version for this build (27210) is outside the format range of your last mix (format 26, upstream versions 25770 to 27200). This build cannot be done until you complete an upstream format build. Please run the following command to complete the format bump:
mixer build upstream-format
Once this has completed you can re-run this build you are attempting to create.
Some operations for building Clear Linux vary depending on whether the build is an upstream or a downstream. Create this flag in the config repository and use this when calling to mixer to properly create update content.
To start, the --offline
flags should be applied to mixer if an upstream.
A static files repository should be a variable in the config repo. This repository is expected to have the following directory structure:
That controls which files get placed in the toplevel directory and which files get places in the directory for each release.
make release
make: *** No rule to make target 'prologue', needed by 'release'. Stop.
You can use "mixer repo add " command to add new repos.
Add the repo url with file:// , http or https protocol
You can remove the "clear" repo from repo list if not needed using "mixer repo remove " command.
But "local" repo is auto added if removed with default url file:///local-yum.
You can set priorities to repos by editing ".yum-mix.conf file". We don't have mixer commands to set priority yet.
Improve the way we use and maintain our workspace to avoid the extreme duplication of artifacts on the disk that we have today.
Also, improve the code in such a way that a "reusable" (BUILD_DIR) workspace isn't required while keep mixer to reuse the required artifacts (swupd key pair and update stream for the deltas) between builds.
This will have to be split in sub tasks, because the final solution isn't possible with current mixer requirements in place.
When using the --no-signing
option on mixer, mixer will not sign the latest file. This is not being done today by our process and is required for supporting future enhancements to swupd. The first file is not used by mixer or swupd, but supports creating delta packs from the first build in every format, and also supports MCA. This file should also be signed.
This should be done in a separate pipeline running on an agent different than the build agent. Do not want debuginfo generation to be in the critical path of the build because it is optional for non-developers who are just running or hosting applications.
Start with the script from https://github.com/clearlinux/clr-debug-info in scripts/clr_debug_prepare along with the following code, replacing/removing undefined variables as necessary:
RAW=${DEBUGINFODIR}.raw
refresh_clr-debug-info() {
local "${@}"
TYPE=projects
PROJ=clr-debug-info
if [[ -d "${BUILDDIR}/${TYPE}/${PROJ}" ]] ; then
sudo rm -rf "${BUILDDIR}/${TYPE}/${PROJ}/"
fi
git clone "${GITHOST}:${TYPE}/${PROJ}" "${BUILDDIR}/${TYPE}/${PROJ}"
PROJDIR=${REPO_BASE_DIR}/${PROJ}
pushd ${PROJDIR}
# clr-debug-info HEAD moves forward at will, but releases are tagged
# with one, two , or three didgets if it ever reaches release 100.
TAG=$(git tag -l | grep -E "^[0-9]{1,3}$" | sort -n | tail -1)
git checkout ${TAG} || error "git special checkout failed"
autoreconf -fiv
if [[ $? -ne 0 ]]; then
error "autoreconf -fiv failed"
fi
${PROJDIR}/configure --prefix=${BUILDDIR}/usr
if [[ $? -ne 0 ]]; then
error "${PROJDIR}/configure ... failed"
fi
make
sudo make install
if [[ $? -ne 0 ]]; then
error "make install failed"
fi
popd
}
create_debug() {
local "${@}"
log "Creating debug info for ${VER} in $0"
#Begin debug data creation ----------------------
set +x
mkdir -p ${RAW}/done
pushd ${RAW}
DCNT=0
for i in ${RELEASEDIR}/${VER}/clear/x86_64/debug/*.rpm ; do
base=`basename $i`;
if [[ ! -e ${RAW}/done/$base ]] ; then
echo "Extracting ${base} ..."
# Extract to a temporary staging dir first
TMP="$(mktemp -d)"
sudo rpm2cpio "${i}" | ( cd "${TMP}"; sudo cpio -idu || : )
# Packages are now shipping debuginfo under
# /usr/share/debug/ instead of /usr/{lib,src}/debug/.
# Move back to /usr/{lib,src}/debug for compatibility.
sudo mkdir -p "${TMP}"/usr/lib/debug
sudo mkdir -p "${TMP}"/usr/src
sudo find "${TMP}"/usr/share/debug/ -mindepth 1 -maxdepth 1 -type d -regextype awk -regex '.*/(.build-id|boot|lib|sbin|usr)$' | xargs -r -I '{}' sudo mv -v '{}' "${TMP}"/usr/lib/debug/
sudo find "${TMP}"/usr/share/debug/ -mindepth 1 -maxdepth 1 -type d -name 'src' | xargs -r -I '{}' sudo mv -v '{}' "${TMP}"/usr/src/debug
sudo rm -rf "${TMP}"/usr/share
# Then copy only changed content into debuginfo.raw.
# Symlinks are copied verbatim, since new ones are
# needed whenever build IDs change, and some of the
# symlinks are expected to be broken (so we can't use
# the -L option) ...
sudo rsync --info=stats2 -i -rlc "${TMP}"/ "${RAW}"/
sudo rm -rf "${TMP}"
touch ${RAW}/done/$base
fi
done
popd
set -x
time nice sudo -E bash -x ${BUILDDIR}/usr/bin/clr_debug_prepare /home/releases/debuginfo.raw /home/releases/debuginfo || error "Unable to run clr_debug_prepare"
#Validate files updated
if [[ $(find ${RAW}/ -type f -atime 0 -amin -6 |wc -l) -eq 0 ]] ; then
log "ERROR: No new files in ${RAW}, debug is not working correctly!"
fi
if [[ $(find ${DEBUGINFODIR}/ -type f -atime 0 -amin -6 |wc -l) -eq 0 ]] ; then
log "ERROR: No new files in ${DEBUGINFODIR}, debug is not working correctly!"
fi
log "Finished creating debug info in $0"
}
main () {
local "${@}"
refresh_clr-debug-info ${@}
create_debug ${@}
}
main ${@}
content.sh should be able to look at a directory and add a rpm repository for each folder in the directory to mixer. This allows users to configure any number of rpm repositories to be used as sources to support more complex downstream use cases.
When starting a Jenkins job, or when running scripts via the Makefile, want to allow the user to override MIX_VERSION and DISTRO_LATEST variables to their choosing. This gives flexibility on how to start a build, which is useful for when issues arise.
Check the version of required tools meet the minimal requirements to work with a given clr-distro-factory release
Content needs to be aware of all inputs to mixer. This mean packages AND bundles. Because content.sh has a well-defined interface with content providers, need to create a bundle provider interface and implement this. This bundle provider interface should be able to clone bundles from a parameterized location to a defined location that will be pulled from in the mix. Whether the mix is an upstream or a downstream will determine if the mixing process also attempt to pull bundles from an external source during the build process.
build_bundles
functionBuilding
section, do not automatically perform/check for a format bump if an upstream.$clear_ver
and $mix_ver
are the same value if an upstream--offline
flag should be used everywhereTo help with conditionally performing work, create an IS_UPSTREAM flag in the config repository and key off this value.
The trick part is how to nicely populate the config.sh vars from the template. It is way easier to just dump a new config file from scratch.
In the other hand, making it from scratch makes trick to also provide an image template (or more than one).
Maybe we can clone from clr-distro-factory-config and fully replace 'config.sh' so we get the image template files but don't have a headache trying to populating the template with variable values.
If an image fails to generate, rather than catching the update in the next build of clear, want to have the option of selectively attempting to regenerate images again. This may come in the form of allowing re-execution through Jenkins pipeline, or maybe we simply put clr-installer into a loop attempting to build the image multiple times (no more than a total of 2 attempts maybe)?
Have a flag for the release workflow that would allow a format bump to be triggered by necessity of the Downstream itself instead of only tracking Upstream's format bumps.
This is required to allow for deletion of bundles that are exclusive to a given Downstream.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.