GithubHelp home page GithubHelp logo

ecm-distro-tools's Introduction

ECM Distro Tools

ECM Distro Tools is a collection of utilities that provide for easier administration, management, and interaction with the great Rancher ecosystems, including RKE2 and K3s.

Installation

The easiest way to install a single utility is to go to the release page, choose the release you want, and download the utility for your operation system and architecture.

Install Script

To install all executables and shell libraries, run the install script as follows:

Install the latest version

curl -sfL https://raw.githubusercontent.com/rancher/ecm-distro-tools/master/install.sh | sh -

Install a specific version

curl -sfL https://raw.githubusercontent.com/rancher/ecm-distro-tools/master/install.sh | ECM_VERSION=v0.31.2 sh -

This will download all binaries and shell libraries and install them to /usr/local/bin/ecm-distro-tools. You'll need to add that directory to your path after installation.

Release CLI

Configuration

New Configuration File

release config gen > $HOME/.ecm-distro-tools/config.json

Load config from custom path

release config view -c ./config.json

Load config from string

release generate rancher missing-images-list v2.7.15 -C '{"rancher": { "versions": {"v2.7.15": {"check_images": ["rancher/rancher:v2.7.15"]}}}}' -i "https://prime.ribs.rancher.io/rancher/v2.7.15/rancher-images.txt" --ignore-validate

Building

There's a mix of code in this repository. The shell scripts and shell libraries reside in the bin directory and are ready to use. The Go programs are rooted in the cmd directory and need to be compiled.

To compile the programs run the following in the root of the project:

make all

To compile the container image locally:

docker build . -t rancher/ecm-distro-tools

Utility Index

The following is a non-exausitve list of the utilities included in this repository and their corresponding usage. (see bin/ and cmd/ for all utility code)

List available utilities

utility_index

For details on specific utilities, review the script header or the README for the specific utility.

All utilities comes with help output.

GitHub action

This repository provides the "Setup ecm-distro-tools" GitHub action. It downloads the assets belonging to the specified release to a temporary directory, and adds the directory to the PATH.

Usage

The action can be run on ubuntu-latest runners. The version parameter is required. Providing the GH_TOKEN environment variable is recommended to avoid rate limiting by the GitHub API.

steps:
  - name: setup ecm-distro-tools
    uses: rancher/[email protected]
    with:
      version: v0.27.0
    env:
      GH_TOKEN: ${{ github.token }}
  - name: release
    run: release -h

Contributing

We welcome additions to this repo and are excited to keep expanding its functionality.

To contribute, please do the following:

Features and Bugs

  • Open an issue explaining the feature(s) / bug(s) you are looking to add/fix.
  • Fork the repo, create a branch, push your changes, and open a pull request.
  • Request review

Development

Expectations

A set of patterns have been established with the Go and shell code that need to be adhered to. Usage output and flags should be copied and pasted from other code files and adjusted to keep the UX as similar as possible to the rest of the utilities in the repo.

Library code has been written for Go and shell which to simpler access to Github, loggers, and means of validating common checks.

When a new utility is added or an API is changed, documentation needs to be updated to reflect that change. This needs to be done wherever that documentation lives, likely the utility's README.

Go

  • Go code additions are expected to have been linted, vetted, and fmt'd prior to pushing the code.
  • Prefer the standard library over 3rd party libraries when possible

Shell

  • Shell scripts are expected to be POSIX compliant, avoiding specific shell features for portability. We are currently using shellcheck to perform these checks and validations.

Building

When building locally, you may want to build just for your ARCH and OS. To do so, you can use one of the two methods below:

Using this method, you can set these variables in .bashrc or .zshrc to make it easier to alaways build for your ARCH and OS. Just be aware that if you do this, you'll need to unset them if you want to build for all ARCHs and OSs.

export ARCHS=amd64
export OSs=linux
make all

Using this method, you'll need to set the variables each time you want to build just for your ARCH and OS.

make ARCHS=amd64 OSs=linux all

License

ecm-distro-tools source code is available under the Apache Clause License.

ecm-distro-tools's People

Contributors

aganesh-suse avatar briandowns avatar brooksn avatar cwayne18 avatar dereknola avatar galal-hussein avatar johnatasr avatar jossemargt-3pillar avatar mallardduck avatar manuelbuil avatar matttrach avatar nicholassuse avatar nikolaishields avatar pjbgf avatar rancher-max avatar tashima42 avatar vestigej avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ecm-distro-tools's Issues

Verify Asset Script - Out of Date

It's been found that the verify assets script is out of date and is resuling in a "MISMATCH" result when ran against releases. The current value of assets it's checking for is 50 when it should be expecting 52.

KDM Update Script

It would be nice to have the ability to update the relevant KDM branches with a new release of RKE2 and k3s. At a minimum, adding a new block of YAML for the release in the dev-v2.6 branch and bumping the previous release in the dev-v2.5 branch, if relevant.

Extend Release Notes Program to Aggregate Packaged Component Versions

The release notes program currently generates release notes for a given milestone but will not aggregates the packaged component versions for that milestone. This functionality should be added so this process is all encompassing.

The documentation to find the packaged component versions can be found in the RKE2 documentation.

Use previous merged date to find pull requests for release notes

After reading the code for finding release notes I thought it would be cool to have a way to manually validate the release notes "changes since" section.
The code uses the compare API, so it was a bit more complex than I wanted to take on for manual verification, and I found a small issue query which finds the same information:

https://github.com/rancher/rke2/issues?q=base%3Arelease-1.23+merged%3A%3E2022-06-13+sort%3Aupdated-asc+
base:release-1.23 merged:>2022-06-13 sort:updated-asc

This may be slightly less accurate since it pulls PRs based on date rather than PRs for commits which merged between tags, but so far I have not found any practical cases where it has not matched up. This is a much easier to parse and limits the queries to a single call to GH vs the O(n) that the code currently operates on (it gets all commits, then queries every commit for a PR, then dedups the PR list ). It is not much of a practical improvement, since we don't really have thousands of commits to parse, but it does help conceptually with understanding what we are looking for and provides an easy way to manually validate, which helps with the learning/training process. It also reduces calls to the GH API, which does get rate limited after a certain number of queries.

You need two pieces of information, the target branch and the previous release date. I also sorted by last updated to get the same sorting as the current code. The failure mode for this is if someone merges a PR for the next release on the day the release is cut you might miss that PR in the release notes.

Automate the creation of PRs updating image-build-base when image-build-base releases

When image-build-base releases a new version, all the rancher/image-build-xxxx projects [1] should update their Dockerfiles. We are talking about ~24 projects. This process is not automated but manual and the consequence is that most of them are outdated.

It'd be so great that a PR would be created for each of them updating that parameter (e.g https://github.com/rancher/image-build-calico/blob/master/Dockerfile#L4) every time there is a new release of image-build-base.

Ideally, we should trigger this from drone/github_actions/etc in the image-build-base github project as soon as there is a new release. However, as we are still unsure what will be our CI at the end of the year, I think it would be great to start having the functionality here.

[1] https://github.com/orgs/rancher/repositories?language=&q=image-build-&sort=&type=all

Add Image Scan Script

it'd be helpful to have a script that performs image security scans the same way that rancher performs its scan.

Program for launching common rke2/k3s cluster configurations.

There has been a request to create a "simple" script that can deploy common k3s cluster configurations.

Goals:

  • Make it really, really easy for developers to test on common k3s/rke2 cluster configurations
  • Use existing E2E tests so if more advanced options are needed, developers can find them.
  • Enable developers to bring over their locally built k3s executable for PR testing.

Refactor CheckUpstreamRelease

The CheckUpstreamRelease methods needs to be refactored to return a map of string bool with the release tag as the key and it's availability represented as a bool.

Fix version references and url's on gen-release-notes

From past march RKE2 release we identified the following artifacts on gen-release-notes output

  • CNI Plugins standalone version reference is not longer needed for RKE2 Fixed on #67
  • Calico url's have the v3.22 fixed version on its path
  • etcd version should be pulled from version.sh instead of go.mod
  • helm-controller uses an outdated version

Fill out README

The README needs to be filled out to include the following:

  • Index of available utilities
  • Usage examples
  • Contributing guidelines
  • Feature request guidelines

As well as additional inconsequential sections.

Add Ability for Go Update Script to Perform VCS Ops

The update Go program needs to be be extended to include the functionality to create a branch, perform the update, add the changed files to staging, commit the changes, push to the origin branch, and create a PR filled out with the necessary information.

[update-rke2-charts] Ensure packageVersion is reset every time Chart version changes

Background

The rke2-charts are downstream updated each time a Kubernetes component or Kubernetes itself. The big picture desire has been documented on rke2#2077.

There are validation rules between the relationship specific values in Chart.yaml and package.yaml that currently are not supported.

Original request

Whenever we update a chart value, packageVersion should increase its value otherwise CI will fail. Example: https://github.com/rancher/rke2-charts/runs/4948436190?check_suite_focus=true
[...]
The reason for that is that otherwise we would be overwriting an existing helm chart and users might get confused. The name of the chart contains the packageVersion: https://github.com/rancher/rke2-charts/tree/main/charts/rke2-canal/rke2-canal.
To be fully accurate, when changing the version, we should go back to packageVersion: 1

Request

Ensure packageVersion (package.yaml) is reset to 1 every time the Chart version changes

Update Base Images for RKE2 Artifacts

We're currently using UBI7 for our base artifact images. These images are updated but not often and not all concerns are taken care of. We need to work on the following:

  • Investigate the move from UBI7 to something newer
  • Investigate the use of BCI, UBI8, alpine, scratch
  • Update the base image to not have all of the reported vulnerabilities from the image-scan process.

All experiments need to be scanned the same as Rancher does to make sure we're meeting customer expectations. This line below references how Rancher image scan performs its scan.

https://github.com/rancher/image-scanning/blob/main/run.py#L303

Add tool to automate rke2-charts update

Background

The rke2-charts are downstream updated each time a Kubernetes component or Kubernetes itself. The big picture desire has been documented on rke2#2077, this tool is part of it.

Request

Create a Go tool that updates the rke2 referenced versions within the existing rke2-charts. A version reference can be:

  • Package version
  • Helm chart version
  • Helm chart appVersion
  • Docker image tag within values.yaml file

Acceptance criteria

  • Working Go tool on ecm-distro-tools able to perform the behavior described above.

Port Shell Lib Code to Go

This logic needs to be ported to Go for functionality in OpDom and other CLI utilities.

  • rke2_list_image_repos
  • rke2_list_images
  • rke2_charts_get_index
  • rancher_list_local_repos

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.