airshipit / airshipctl Goto Github PK
View Code? Open in Web Editor NEWA CLI for managing declarative infrastructure.
License: Apache License 2.0
A CLI for managing declarative infrastructure.
License: Apache License 2.0
Problem description (if applicable)
The current implementation of redfish.RedfishConfigError
does not have an Error
method, and therefore doesn't implement the error
interface. This can be annoying for a user trying to type switch on the error
Proposed change
Implement the Error
method on redfish.RedfishConfigError
. This error could also benefit from a renaming to more closely match the other errors in airshipctl
Problem description (if applicable)
The document
pkg needs to expose/re-implement a few kustomize primitives to allow consumers of the document
pkg to filter documents and specify Gvk
and fs
objects without importing kustomize directly in their code. This ensures we define these interfaces in one place, and do not have a massive code change when we uplift kustomize and these interfaces change.
Proposed change
Create document
pkg local Gvk
, fs
, and types
so consumers of the document
pkg are not required to import kustomize at all.
Potential impacts
This will require updating a few existing places that are directly importing kustomize for this purpose, such as https://review.opendev.org/#/c/705890/8/pkg/remote/remote_direct.go and re-implementing them with the local document
types and functions.
Problem description (if applicable)
There should be a simple way to display what airshipctl
knows about its document set (target repositories, branches, target directories, airshipctl document pull
status).
Some of this can be pulled from the airship configuration, and some of it interpreted from cloned repository directories.
Proposed change
Implement airshipctl document info
to provide details of the document set currently targeted, such as the target repositories, branches, commit IDs, and the target clone directory. This could also provide some summary information available from the rendered document set, such as Group, Version, and Kinds found, their count, and so on which should be easily extracted from a Bundle
. Just as an example:
13 baremetalhosts.metal3.io/v1/BareMetalHost
4 core/v1/Secret
...
Potential impacts
None
Problem description (if applicable)
Currently documentation only exists at the command line for AirshipCTL functionality.
Proposed change
Enable web documentation builds from AirshipCTL source docs, and ensure those source docs contain information on the commands/options.
Potential impacts
None
Problem description (if applicable)
For both ephemeral and target invokations of airshipctl cluster initinfra
as well as a non-existent pivot command we intend to integrate the clusterctl pkg as a library.
While the clusterctl
pkg continues to develop, this issue represents the effort of beginning that integration to understand the specific gaps remaining, especially the work we need to accomplish upstream within clusterctl
.
Proposed change
For the airshipctl cluster initinfra
commands, leverage the clusterctl
library to perform the init, effectively equivalent to a clusterctl init
. This will initially be equivalent to a kubectl apply
of the provider resources, and we will build on the gaps remaining for leveraging metal3-io as a provider to just providing that - for instance, ensuring that the baremetal-operator
, ironic
, and so on are also instantiated, as well as BareMetalHost
records to support subsequent Machine
provisioning.
Within airshipctl
, there is a need to be able to feed provider resources to a library like clusterctl
. There has been work done upstream in clusterctl
to support pulling provider resources from a local file system. We need to understand whether this will work for the way airshipctl
renders documents through Kustomize and emits them as Document
objects as part of a bundle - see the following pull request in clusterctl. Will we create a fake filesystem and provide that to clusterctl
or enhance it upstream to support a YAML byte stream?
Potential impacts
There may be requirements for clusterctl
upstream that may need to be solved locally in airshipctl
.
Add a command in cluster module of airshipctl to power on, power off and validate the hosts in the cluster.
airshipctl remotedirect server -- name --poweroff/poweron/validate .....
Main use cases:
Problem description (if applicable)
The original implementation of airshipctl bootstrap remotedirect
occurred before the implementation of configurations in airshipctl. As a placeholder, CLI flags were used to testing and POC for the command.
Proposed change
CLI flags should be replaced with calls to the airshipctl configuration data for any data used that is stored there.
Source: TODO, cmd/bootstrap/bootstrap_remotedirect.go, L22 & L62
Authentication modules Kubernetes include Client Certificates, Password, and Plain Tokens, Bootstrap Tokens, and JWT Tokens (used for service accounts). Multiple authentication modules can be specified, in which case each one is tried in sequence, until one of them succeeds.
Target Kubernetes cluster, bootstrapped by airshipctl should be integrated with OpenID Connect Provider so that a user can be authenticated by the authentication back-end such as LDAP and github that are not directly supported by Kubernetes. OIDC provider sits between Kube api server and authentication back-ends to support authentication of users via LDAP etc.
OIDC provider is required to be deployed in the target cluster and be integrated with authentication back-ends.
Once the user is authenticated, Kubernetes RBAC mechanism can be used to authorize the users for what they are allowed / denied for.
DEX : https://github.com/dexidp/dex
Problem description (if applicable)
Windows linting is currently broken:
C:\aic\airshipctl> make test
mkdir -p tools/bin
./tools/install_linter
+ tools_bin_dir=./tools
+ download_url=https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh
+ version=v1.21.0
+ curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh
+ sh -s -- -b ./tools/bin v1.21.0
golangci/golangci-lint info checking GitHub for tag 'v1.21.0'
golangci/golangci-lint crit unable to find 'v1.21.0' - use 'latest' or see https://github.com/golangci/golangci-lint/releases for details
+ printf 'Something went wrong while installing golangci-lint\n'
Something went wrong while installing golangci-lint
+ exit 1
make: *** [Makefile:159: tools/bin/golangci-lint] Error 1
Proposed change
update the install_linter script to include a way to pull in the golangci-lint.exe for windows
Potential impacts
Windows is not technically an officially supported target architecture, but this is a nice to have in case anyone else is falling on this
Describe the bug
When running the make godoc
target, a user's go.mod
file is updated to reflect a dependency on the godoc
tool. However, since airshipctl
itself does not depend on godoc
, the make test
target will fail until the user reverts the changes to his go.mod
.
Steps To Reproduce
make godoc
make test
Expected behavior
Unit tests should pass
Problem description (if applicable)
Airshipctl needs the ability to render documents for a site.
Proposed change
Implement airshipctl render command as a document CLI command. This should be done using Kustomize for rendering.
Potential impacts
None.
Problem description (if applicable)
Add capability to filter documents collected in document bundle command.
Proposed change
When document bundle is called, allow filtering to limit the collection to specific documents and return that subset of documents as a bundle.
Potential impacts
Document bundle command
Problem description (if applicable)
The airshipctl client needs the ability to manage workflows using the Argo Workflows wrapper.
Proposed change
This is an epic; issues will be filed later with more details.
Problem description (if applicable)
The cmd/
directory should be strictly for CLI implementation, logic, and validation. Currently, some commands have some of their execution logic written in the same file as the command declaration in the cmd/
directory. Having the execution logic mixed with the command implementation can cause confusion for developers and possible namespace collisions. Having all logic within the pkg/
directory will also make it easier to use airshipctl as a library. The following files have mixed code:
Proposed change
All execution logic should be moved from the cmd/
directory to the pkg/
directory.
Problem description (if applicable)
Currently, most argument/flag validation occurs at the package level or is non-existent. This can lead to unnecessary code running up to argument/flag validation.
Proposed change
To avoid running unnecessary code and make the CLI as usable as possible, we should consider implementing validation at our CLI level. Many pre-built validators are available in the Cobra CLI package for both arguments and flags. For additional validation for parameters such as URLs or file paths, we can consider building custom shared validation functions callable from the Args
section of each targeted command.
Potential impacts
CLI validation could replace package level implementations of validation. However if airshipctl is to be used as both a CLI application and a library, validation may need to exist at both levels which potentially adds duplicated code.
Problem description (if applicable)
Airshipctl needs some way of initializing documents for a new site.
Proposed change
Implement document init command.
Potential impacts
None
Steps To Reproduce
diff --git a/pkg/document/bundle.go b/pkg/document/bundle.go
index 011c15d..3cd9843 100644
--- a/pkg/document/bundle.go
+++ b/pkg/document/bundle.go
@@ -16,6 +16,7 @@ import (
"sigs.k8s.io/kustomize/v3/pkg/target"
"sigs.k8s.io/kustomize/v3/pkg/types"
+ "opendev.org/airship/airshipctl/pkg/config"
docplugins "opendev.org/airship/airshipctl/pkg/document/plugins"
"opendev.org/airship/airshipctl/pkg/log"
utilyaml "opendev.org/airship/airshipctl/pkg/util/yaml"
@@ -25,6 +26,7 @@ func init() {
// NOTE (dukov) This is sort of a hack but it's the only way to add an
// external 'builtin' plugin to Kustomize
plugins.TransformerFactories[plugins.Unknown] = docplugins.NewTransformerLoader
+ _ = config.AirshipClusterDefaultType
}
// KustomizeBuildOptions contain the options for running a Kustomize build on a bundle
make test
Expected behaviour
airshipctl
binary has been successfully tested
Actual behaviour
Build error raised
Checking that go.mod is up to date...
go.mod is up to date
Performing linting step...
ERRO Running error: context loading failed: failed to load program with go/packages: internal error: go list gives conflicting information for package opendev.org/airship/airshipctl/pkg/config [opendev.org/airship/airshipctl/pkg/config.test]
make: *** [lint] Error 3
Environment
Problem description (if applicable)
The airshipctl cluster initinfra
command will need the ability to remove resources in a Kubernetes cluster matching a designated set of labels or annotations.
Proposed change
Add the ability to remove resources matching a specified set of labels or annotations to a Kubernetes (or k8s) package using the Kubernetes client.
Potential impacts
No anticipated security or performance related impacts.
Problem description (if applicable)
Writing to the airshipconfig and kubeconfig in PersistConfig()
can result in modifications to only the airshipconfig file if an error occurs. This will result in differences between the airshipconfig and kubeconfig.
Proposed change
A possible solution is to make the modifications to each file codependent. A backup of each file could be created before modifications are made and restored if an error occurs.
Source: FIXME(howell), pkg/config/config.go, L355-359
Describe the bug
Running the unit tests via make test
produces over 50 temporary directories. These quickly build up and slow down the host OS.
Steps To Reproduce
Run make test
and ls /tmp
.
Expected behavior
The unit tests should write to a /tmp
directory that gets cleaned up after execution.
Environment
v0.1.0
4.4.0-173-generic
1.13.7
Relates-To: #36
Problem description (if applicable)
We currently have an airshipbot user that posts when changes are merged for each issue (due to the fact that the mirror is updated upon change merge). This bot could be expanded to update issues with comments when changes are created or updated.
Proposed change
Write Zuul gates to update GitHub Issues when new, tagged changes are pushed or updated.
Need to go explore if there is a project to manage label, taints, etc for Node .
Community says this is not part of capi for now?
kubernetes-sigs/cluster-api#493
This is a question to define the direction we should take.
Do we create a new operator to do this?
Would this fall under the perview of the Day solution?
Problem description (if applicable)
The config package has a large number of functions contained within config.go
, some unrelated to each other. This can make it difficult to find certain workflows and segments of code for developers.
Proposed change
Functions could be broken up into several separate files or packages to help improve code clarity and organization.
Potential impacts
As this would be a large scale change, special care will need to be taken to ensure no circular imports are created and that functionality stays the same after the refactor.
The remoteDirect
command uses a custom data type populated by CLI arguments for configuration data. Change this to use the BootStrapInfo
config module.
Describe the bug
The airshipctl documentation contains various typos and grammar issues in its first draft.
Solution
Documentation should be reviewed and patched to fix and spelling and grammar mistakes found.
document validation explicitly will implement
validation rules beyond the simple yaml rendering validation done via kustomize mechanism.
This command will provide a way for enforcing validation policies such as :
Problem description (if applicable)
The airshipctl
end-to-end test is currently launched by Zuul infrastructure. We need to ensure these same gates can act as developer friendly scripts to launch a virtualized airshipctl
environment locally.
Proposed change
While the ansible gate playbooks can be launched by hand with a series of flags and environmental settings, we need to make this more approachable to new developers. This issue proposes a series of simple idempotent scripts that developers can run which will exercise the gate locally, without Zuul infrastructure, e.g.
tools/gate/01_create_iso.sh
tools/gate/02_launch_ephemeral_node.sh
...
Potential impacts
None
Describe the bug
In certain circumstances, airshipctl config get-cluster
ignores filters and will print all clusters. The known circumstances are from attempting to use the --cluster-type
option or attempting to specify more than one argument for the cluster name.
Steps To Reproduce
Case One: Specifying a Cluster Type
airshipctl config get-cluster --cluster-type=ephemeral
on a valid configuration with a mix of cluster types.Case Two: Specifying more than one cluster anem
airshipctl config get-cluster <NAME1> <NAME2>
on a valid configuration with clusters and .Expected behavior
When --cluster-type
is specified only clusters of that cluster type should be displayed.
When more than one cluster name is provided as an argument, airshipctl should either give an error or only print the clusters named as arguments.
Source: FIXME(howell), cmd/config/get_cluster_test.go, L80 & L96
This command initializes the configuration with some adequate seed values.
All airshipctl or airshipui behavior is driven by what exists in the configuration files.
We need to create an airship configuration file set to define the intentions the operator/user has for the site.
One goal in Airship 2.0 is to maintain a declarative framework and to help facilitate usage by allowing the operator/user the config init function to seed the configuration. These configurations will prepare aspect of the configuration such as Manifest/Repository, Bootstrap information with some default consumed from Treasuremap.
Expectation is that the operator/user will use the config cmd to define its initial target cluster and context intentions for the ephemeral cluster, and the target cluster.
Problem description (if applicable)
The current implementation of ImagePull()
always downloads images on each call. This results in additional data usage and run time even if the image is already downloaded.
Proposed change
The implementation should be modified to search for a matching local image before resorting to downloading a new copy from a remote source.
Potential impacts
Local images will need to be md5 or sha256 validated before use. Any discrepancies should still result in a full download of the image.
Source: TODO(D. Ukov), pkg/container/container_docker.go, L199-200
Problem description (if applicable)
Most Airship 1 projects automatically built and pushed docker images for each merged change to quay.io. Airship 2 projects should use the same process for easier retrieval of developmental and release images for developers and end users.
Proposed change
Implement a Zuul gate to run a docker image build and push to quay.io.
Relates-To: #27
Problem description (if applicable)
As part of #27, CLI helper functions were moved from the cmd/config package to the pkg/config package. In the associated Gerrit change for #27, @dukov pointed out that we should create a convention for where CLI helper functions are stored inside their respective packages. Creating and implementing this convention will help us better distinguish library code and CLI helper code.
Proposed change
Create and implement a convention for organization of CLI helper functions.
Describe the bug
airshipctl
defaults to using $HOME/.airship/config
and $HOME/.airship/kubeconfig
as the filepaths to the associated config files. As of https://review.opendev.org/706408, the Go code doesn't handle environment variable in these paths, which causes the default values to fail.
Problem description (if applicable)
Currently, there is no standard for how flags are added to a cobra.Command
Proposed change
We should create a standard means of adding these flags. I thing it would make sense for each command to have an associated Options
object which could be attached to the flags that are passed in.
Problem description (if applicable)
There should be a way to show bootstrap information from modules config. Current implementation of "config get-cluster" shows only name of bootstrapInfo module associated with cluster.
Proposed change
New CLI command "config get-module" should provide information about implemented modules.
Potential impacts
TBA
Problem description (if applicable)
Right now, airshipctl requires users to author a configuration file. Users should be able to initialize a default airshipctl config.
Proposed change
Add the airshipctl config init
subcommand, which will initialize the airshipctl config with default values. The user will build the rest of the configuration imperatively using other command line arguments. See the spec for more information.
Potential impacts
No potential security or performance related impacts anticipated.
Parent: #2
Problem description (if applicable)
airshipctl should be able to validate documents it manages before they are used. One way we can validate documents is by using JSON schema.
Proposed change
Add a command airshipctl document validate schema
to validate site documents using their associated schema.
Relates-To: #58
Problem description (if applicable)
Many functions are relatively undocumented in the airshipctl code base. Undocumented code can make it more difficult for developers to navigate code and efficiently work on enhancements and bug fixes.
Proposed change
We should improve function documentation to help improve the project's readability in its raw form and in godoc. There is an official godoc guide, but it is rather incomplete. Luckily, there are some third party resources available such as godoc-tricks that can help us create better code documentation.
Related-To: #36
Problem description (if applicable)
The OpenStack Storyboard is still enabled for airshipctl, but we have switched to using GitHub Issues. To avoid confusion, the OpenStack Storyboard should be disabled for this project.
Proposed change
Disable OpenStack Storyboard and try enabling GitHub Issues integration with the has-issues
flag.
Potential impacts
OpenStack Storyboard will no longer be available for airshipctl.
Problem description (if applicable)
We are currently using GitHub for our issues, but we still have OpenStack Storyboard enabled on our project in Gerrit. Ideally, the Storyboard for this project should be disabled and additional integration with GitHub (possibly via jeepyb) should be investigated.
Proposed change
Research ways to better integrate our GitHub Issues with OpenDev Gerrit.
Problem description (if applicable)
airshipctl needs to be able to pull site configuration documents from a remote source.
Proposed change
Implement an airshipctl document pull
command to pull documents to a target directory from a remote git repository.
Problem description (if applicable)
Testing is not currently being done by airshipctl to ensure that when a command expects a specific number of arguments, that only that number of arguments is accepted.
Proposed change
Implement tests to validate that expected number of args is enforced on commands that require a specific number of args.
Potential impacts
None
Describe the bug
The airshipctl config get-credentials
command outputs the credentials in random order. This could be confusing to a user who expects consistent output. It also causes issues with unit tests; tests will frequently pass, but occasionally fail.
Steps To Reproduce
Create an airship config with multiple credentials and run airshipctl config get-credentials
several times.
Expected behavior
Output should be the same every time.
Relates-To: #6
Problem description (if applicable)
The current planned implementation of airshipctl config init
creates a configuration document with blank or dummy data. This creates a roadblock to users that will need to both understand and fill in key/value data in the configuration.
Proposed change
To avoid extra challenge to users, we could introduce a series of interactive input prompts similar to Angular's ng new. These input prompts could provide key details for users as to the meaning of each configuration key/value pair and allow easier creation of configuration documents.
Problem description (if applicable)
The current implementation of tests in set_context_test.go
does not use table tests. As stated in our Testing Guidelines, we want to use table tests as often as possible to avoid code duplication.
Proposed change
Refactor the tests in set_context_test.go to use the table test model.
Problem description (if applicable)
As part of the cluster-api workflow, airshipctl
will require an airshipctl cluster move
implementation to move resources from the ephemeral to the target cluster.
Proposed change
This expands on integration with the clusterctl
integration (#29). It will leverage the clusterctl
pkg in order to perform a clusterctl move
but it must also care for the BareMetalHost
object of the target node which would be provisioned by the ephemeral node to ensure it is transferred to the target cluster along with the inventory data that was gathered. There is a known issue right now where the status of the node, namely the fact that it was successfully provisioned, is not transferred. There is a known out-of-band solution that exists, effectively creating the BareMetalHost
entry, and then patching the status as a subsequent step. This requires the following flow:
Potential impacts
None
Both the CLI and the UI will need to be able to leverage a standard k8ssecurity context for interacting with remote sites. It should also understandAirship site contexts so that it can be pointed at a local document set andwork with that during subsequent commands without needingto be fed the same document set with every command similar to interactingwith the same Kubernetes cluster throughout a kubectl session.
The initial proposed list of commands that will be implemented is defined below:
Every command will support :
GENERIC OPTIONS
-----------------------------------------------------------------+
NAME
USAGE
=================================================================+
help
Help for the appropriate command
it also lists available plugins
-----------------------------------------------------------------+
version
Version of airshipctl.
-----------------------------------------------------------------+
Problem description (if applicable)
Currently, the airshipctl iso-gen process makes a call to the cloud-init pkg to generate the cloud-init for the ephemeral host.
This looks up the appropriate user data and network data for the ephemeral host by finding a Kind: Secret
in the document set with the appropriate annotation.
This has several issues:
networkData
key, however we use a netplan
key the current implementation meaning we could not share/re-use this Secret
between the BMO and airshipctl, resulting in a duplicate ephemeral host network configuration somewhere.Proposed change
Originally, we wanted to drive ephemeral node selection by indicating a specific BareMetalHost
as the ephemeral node which would allow code such as the cloud-init pkg to back into the appropriate networkData and any other host-specific information easily without duplicating that data for ephemeral iso generation. If one of the hosts had such an annotation, walking its spec
back to the actual Secret
containing the appropriate networkData
for that host should be fairly straight forward and would not run into any conflict issues.
This would also allow us a path to get that hosts BMC information for power control as well, avoiding the need to specify this separately in Airship configuration.
Problem description (if applicable)
Airshipctl needs the functionality from Pegleg's collect command.
Proposed change
Add a command to airshipctl allowing the collection of documents. These documents are referred to as a bundle.
Potential impacts
None
Problem description
The Document type will be wrapping various Kubernetes objects of different types. The airshipctl
tool could use a client which wraps the Document, extracts the inner kubernetes objects, and sends them through the correct kubernetes clients.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.