GithubHelp home page GithubHelp logo

facebookincubator / go2chef Goto Github PK

View Code? Open in Web Editor NEW
38.0 25.0 25.0 185 KB

A Golang tool to bootstrap a system from zero so that it's able to run Chef to be managed

License: Apache License 2.0

Makefile 0.50% Go 98.30% Shell 1.20%

go2chef's Introduction

go2chef: "just enough Go to get to Chef"

What is go2chef?

go2chef is a Go tool for bootstrapping Chef installations in a flexible and self-contained way. With go2chef, our goal is to make bootstrapping any node in a Chef deployment as simple as "get go2chef onto a machine and run it"

Requirements

Build Dependencies

go2chef requires Go 1.12+ for appropriate module support. Library dependencies are enumerated in the go.mod file in the repository root.

go2chef has no runtime dependencies.

Quickstart Example

Installing

go2chef is packaged in Fedora as of Fedora 32. It can be installed with:

$ sudo dnf install go2chef

Building

Build go2chef using make on Unix platforms:

$ make all		# build all variants to build/$GOOS/$GOARCH/go2chef
$ make linux		# ...or build platform-specific binaries
$ make darwin
$ make windows

On Windows, just build it using go build:

PS> mkdir build/windows/amd64
PS> go build -o build/windows/amd64/go2chef.exe ./bin

Configuring

Create a configuration file. For example, to install Chef and then download and install a custom chefctl.rb bundle from a tarball on Fedora:

{
  "steps": [
    {
      "type": "go2chef.step.install.linux.dnf",
      "name": "install chef",
      "version": "15.2.20-1.el7.x86_64",
      "source": {
        "type": "go2chef.source.http",
        "url": "https://packages.chef.io/files/stable/chef/15.2.20/el/8/chef-15.2.20-1.el7.x86_64.rpm"
      }
    },
    {
      "type": "go2chef.step.bundle",
      "name": "install chefctl",
      "source": {
        "type": "go2chef.source.local",
        "path": "./chefctl.tar.gz",
        "archive": true
      }
    }
  ]
}

Executing

  1. Copy the appropriate binary from build/$GOOS/$GOARCH/go2chef, the config file, and the chefctl bundle to a single directory on the target host

  2. Execute go2chef with the config

    $ cd path/to/copy
    $ ./go2chef --local-config config.json
    

scripts/remote.go

A remote execution script is provided in scripts/remote.go. Example usage:

$ make windows && go run scripts/remote.go --binary build/windows/amd64/go2chef.exe -B examples/bundles/chefctl -B examples/bundles/chefrepo -B examples/bundles/whoami_exec --target 10.0.10.187 -W -c examples/config_install_msi.json

Design

go2chef has four basic building blocks, all of which are implemented using a plugin model:

  • Configuration Sources (go2chef.ConfigSource): fetch go2chef configuration from remote sources
  • Loggers (go2chef.Logger): send log messages and structured events to logging backends via a common plugin API
  • Steps (go2chef.Step): implement the building blocks of a go2chef workflow. Every action that needs to be taken to set up your Chef environment can be integrated into a go2chef step. See "Steps" for more details
  • Sources (go2chef.Source): implement a common API for retrieval of remote resources needed for Step execution

Configuration Sources

Configuration sources are the plugins which allow you to customize how go2chef retrieves its runtime configuration. We provide a couple configuration plugins out-of-the-box:

  • go2chef.config_source.local: loads configuration from a JSON file accessible on the filesystem. (this is the default configuration source)
  • go2chef.config_source.http: loads configuration source in JSON format from an HTTP(S) endpoint. Enable using go2chef --config-source go2chef.config_source.http
  • go2chef.config_source.embed: loads configuration source from an embedded variable. This probably isn't what you want, but if it is, have it.

New configuration sources can be registered with go2chef.RegisterConfigSource.

Loggers

Loggers are the plugins which allow go2chef users to report run information for monitoring and analysis, and provide plugin authors with a single API for logging and events.

For Users

Logging plugins are configured using the loggers key in go2chef configuration. An example configuration setting up the default go2chef.logger.stdlib looks like:

{
  "loggers": [
    {
      "type": "go2chef.logger.stdlib",
      "name": "stdlib",
      "level": "DEBUG",
      "debugging": 1,
      "verbosity": 1
    }
  ]
}

The loggers key is an array so that you can log to multiple places, which may be useful for the following scenarios:

  1. You want your raw log messages to go to syslog, but you also want to send specific events to a separate logging service using a custom plugin to trigger some downstream action (i.e. changing asset service state).
  2. You want to log to file and stderr and syslog at varying levels of verbosity (and so on and so forth)

For Developers

Logging plugins may skip parts of the interface specification by stubbing out the unneeded methods as no-ops.

The go2chef.MultiLogger implementation synchronously dumps messages out to backends at the moment, so delays in message sending in a Logger plugin may slow down execution of go2chef as well.

Steps

Steps are the plugins which actually "do stuff" in go2chef. These can do pretty much anything you want if you implement it, but we've intentionally limited the built-in plugins to the following initially:

  • Sanity checking: make sure that the runtime environment is sane before trying to install Chef -- are we root? Is the clock set right? Is there disk space?
  • Bundle exec: provide a simple abstraction for fetching and running some arbitrary scripts/binaries before/after installation. Do things like set up required certs, install chefctl.rb, etc.
  • Installers: provide installer implementations for each platform (and sub-platforms thereof, if necessary).

Many Step implementations will require some sort of remote resource retrieval; rather than leaving it up to each implementation to bring its own support code for downloads, we provide it to you using Sources (described next).

Sources

Source plugins implement a common API for resource retrieval for go2chef. This allows all steps to configure remote resource retrieval with the same idiom:

{
  "steps": [
    {
      "type": "go2chef.step.install.linux.apt",
      "name": "install chef",
      "source": {
        "type": "go2chef.source.http",
        "url": "https://example.com/chef-15.deb"
      }
    }
  ]
}

A source key inside a step configuration block defines how the remote resources for that step should be retrieved.

Code Layout

bin/        # go2chef binary source code
build/      # temporary directory for build outputs
cli/        # CLI implementation
plugin/     # plugins directory
  config/   # configuration source plugins
  logger/   # logger plugins
  source/   # source plugins
  step/     # step plugins
*.go        # go code for the base go2chef module

Contribute

See the CONTRIBUTING file for how to help out.

License

go2chef is Apache 2.0 licensed.

go2chef's People

Contributors

aleksandrmeta avatar chenry3 avatar chermehdi avatar dafyddcrosby avatar davide125 avatar demosdemon avatar joshuamiller01 avatar kcbraunschweig avatar natewalck avatar oko avatar skywalk7 avatar steelcowboy avatar svmastersamurai avatar williamtheaker avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go2chef's Issues

Clarify requirements

Trying to build this on Debian Stable, I get:

$ make
go test ./...
# github.com/facebookincubator/go2chef/plugin/step/install/darwin/pkg
plugin/step/install/darwin/pkg/pkg.go:127:61: exit.ExitCode undefined (type *exec.ExitError has no field or method ExitCode)
note: module requires Go 1.12# github.com/facebookincubator/go2chef/plugin/step/install/linux/apt
plugin/step/install/linux/apt/apt.go:111:75: err.(*exec.ExitError).ExitCode undefined (type *exec.ExitError has no field or method ExitCode)
note: module requires Go 1.12ok  	github.com/facebookincubator/go2chef	(cached)
ok  	github.com/facebookincubator/go2chef/plugin/config/embed	(cached)
ok  	github.com/facebookincubator/go2chef/plugin/config/http	(cached)
ok  	github.com/facebookincubator/go2chef/plugin/config/local	0.002s
ok  	github.com/facebookincubator/go2chef/plugin/lib/certs	(cached)
ok  	github.com/facebookincubator/go2chef/plugin/source/http	0.039s
ok  	github.com/facebookincubator/go2chef/plugin/step/sanitycheck	(cached)
ok  	github.com/facebookincubator/go2chef/plugin/step/winsanitycheck	(cached)
ok  	github.com/facebookincubator/go2chef/util/plugconf	(cached)
make: *** [Makefile:7: test] Error 2
$ go version
go version go1.11.6 linux/amd64

Either fix the module to work with go 1.11, or update the README to clarify the dependency on go 1.12

[Feature Request] `go2chef.step.install.linux.dnf` should allow specifying the repo path as source

The typical usage of installing a package in dnf is like the following

{
  "steps": [
    {
      "type": "go2chef.step.install.linux.dnf",
      "name": "install chef",
      "version": "15.2.20-1.el7.x86_64",
      "source": {
        "type": "go2chef.source.http",
        "url": "https://packages.chef.io/files/stable/chef/15.2.20/el/8/chef-15.2.20-1.el7.x86_64.rpm"
      }
    }
  ]
}

This is awesome, because it doesn't depend on any repo config on the const and make the bootstrap truly independent.
This works great for packages here the version doesn't change frequently - so the config is reasonably static.

However it might not be ideal in cases where the versions are changing quickly because of CI/CD. In that case it might be desirable to specify the path of the repo instead. Something like this:

{
  "steps": [
    {
      "type": "go2chef.step.install.linux.dnf",
      "name": "install chef-solo-tools",
      "version": "15.2.20-1.el7.x86_64",
      "source": {
        "type": "go2chef.source.repo",
        "baseurl": "https://packages.chef.io/centos/8.x/x86_64/"
      }
    }
  ]
}

Allow downgrading packages

Because Go2Chef does not allow downgrading, it alone does not help in a scenario where it might install a stable version of Chef and you'd want it to recover if a later version of Chef caused an issue.

For instance, Chef 15.6 works with our current code and Chef 16.2 doesn't. We can't use Go2Chef to get back to Chef 15.6 without first uninstalling Chef 16.2. There should be some kind of option, such as "allow_downgrade", in the config that enables this kind of functionality.

PackageName option to install step ineffective

If using the PackageName option to override the package to install from the default ("chef"), log messages will indicate the specified package is being installed but in reality it still tries to install a package called "chef" anyway. The option is not being respected. (tested on a host using DNF)

MSI Exit Codes that are non-zero

Right now we hunt for the 0 exit code by simply checking if err is non-nil. If you install an MSI that is already installed it will throw 1603 which is valid... kinda sorta.

If this should be idempotent we should be able to handle either known-"good" errors or let the user of the application specify the exit codes they want to accept.

[Issue] Add Makefile entry for Apple Silicon

It would be nice to have an easy Makefile entry for M1 and newer Macs.

I'm happy to do this and test it. Should we call it darwin-apple-silicon or darwin-arm or darwin-arm64?

[feature request] command step allow output redirection

Because of how commands are executed you can't do output redirection but you may want to. Request is to provide options to the command step to allow sending stdout and/or stderr to /dev/null (or I guess a file if someone wants?)

[Issue] `go2chef.step.install.linux.dnf` doesn't treat Version consistently

I am presently using go2chef.step.install.linux.dnf to install a package and version and relying on dnf to figure out the repo. But I am seeing inconsistency with how go2chef is honoring the "version" value.

go2chef doesn't honor version when the package is not installed at all.
go2chef does honor version when the package is already installed
go2chef doesn't downgrade the package if the version is installed

Repro:

########## Go2chef config ########## 

[centos@ip-172-16-1-244 ~]$ cat /etc/go2chef.config-broken.json
{
  "steps": [
    {
      "type": "go2chef.step.install.linux.dnf",
      "name": "install chef-13",
      "version": "13.10.0-1.el7",
      "package_name": "chef"
    }
  ]
}

########## Remove installed chef ##########
[centos@ip-172-16-1-244 ~]$ sudo dnf remove chef -y
Dependencies resolved.
================================================================================================================================================================================================================================================================================================================================================================================================================================
 Package                                                                                                   Architecture                                                                                  Version                                                                                                 Repository                                                                                                Size
================================================================================================================================================================================================================================================================================================================================================================================================================================
Removing:
 chef                                                                                                      x86_64                                                                                        14.15.6-1.el7                                                                                           @fb-site-packages                                                                                        108 M
Removing dependent packages:
 fb-chef-solo-tools                                                                                        noarch                                                                                        20220503-100441                                                                                         @any-noarch                                                                                              131 k

Transaction Summary
================================================================================================================================================================================================================================================================================================================================================================================================================================
Remove  2 Packages

Freed space: 108 M
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                                                                                                                                                                        1/1
  Erasing          : fb-chef-solo-tools-20220503-100441.noarch                                                                                                                                                                                                                                                                                                                                                              1/2
  Erasing          : chef-14.15.6-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              2/2
  Running scriptlet: chef-14.15.6-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              2/2
  Verifying        : chef-14.15.6-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              1/2
  Verifying        : fb-chef-solo-tools-20220503-100441.noarch                                                                                                                                                                                                                                                                                                                                                              2/2

Removed:
  chef-14.15.6-1.el7.x86_64                                                                                                                                                                              fb-chef-solo-tools-20220503-100441.noarch

Complete!
[centos@ip-172-16-1-244 ~]$ rpm -qa | grep ^chef-
[centos@ip-172-16-1-244 ~]$

##########  Run go2chef - doesn't honour version ########## 

[centos@ip-172-16-1-244 ~]$
[centos@ip-172-16-1-244 ~]$ sudo /usr/local/bin/go2chef --local-config /etc/go2chef.config-broken.json

          ___    _         __
 __ _ ___|_  )__| |_  ___ / _|
/ _` / _ \/ // _| ' \/ -_)  _|
\__, \___/___\__|_||_\___|_|
|___/

GO2CHEF 2022/06/08 21:27:34 loading config from source go2chef.config_source.local
GO2CHEF 2022/06/08 21:27:34 EVENT: LOGGING_INITIALIZED in go2chef.cli -
GO2CHEF 2022/06/08 21:27:34 EVENT: STEP_0_START go2chef.step.install.linux.dnf:'install chef-13' in go2chef.cli -
Last metadata expiration check: 0:42:16 ago on Wed 08 Jun 2022 08:45:19 PM UTC.
Dependencies resolved.
================================================================================================================================================================================================================================================================================================================================================================================================================================
 Package                                                                                         Architecture                                                                                      Version                                                                                                    Repository                                                                                                   Size
================================================================================================================================================================================================================================================================================================================================================================================================================================
Installing:
 chef                                                                                            x86_64                                                                                            14.15.6-1.el7                                                                                              fb-site-packages                                                                                             28 M

Transaction Summary
================================================================================================================================================================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 28 M
Installed size: 108 M
Downloading Packages:
chef-14.15.6-1.el7.x86_64.rpm                                                                                                                                                                                                                                                                                                                                                                    30 MB/s |  28 MB     00:00
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                                                                                                                                                                            30 MB/s |  28 MB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                                                                                                                                                                        1/1
  Installing       : chef-14.15.6-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              1/1
  Running scriptlet: chef-14.15.6-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              1/1
Thank you for installing Chef Infra Client! For help getting started visit https://learn.chef.io

  Verifying        : chef-14.15.6-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              1/1

Installed:
  chef-14.15.6-1.el7.x86_64

Complete!
GO2CHEF 2022/06/08 21:27:43 EVENT: STEP_0_COMPLETE go2chef.step.install.linux.dnf:'install chef-13' in go2chef.cli - completed successfully in 8 second(s)
GO2CHEF 2022/06/08 21:27:43 EVENT: ALL_STEPS_COMPLETE in go2chef.cli - 1 step(s) completed successfully in 8 second(s)
2022/06/08 21:27:43 temp dirs cleanup completed
[centos@ip-172-16-1-244 ~]$ rpm -qa | grep ^chef-
chef-14.15.6-1.el7.x86_64

##########  Install the desired chef version ##########
 
[centos@ip-172-16-1-244 ~]$ sudo dnf install chef-13.10.0-1.el7  -y

Last metadata expiration check: 0:44:26 ago on Wed 08 Jun 2022 08:45:19 PM UTC.
Dependencies resolved.
================================================================================================================================================================================================================================================================================================================================================================================================================================
 Package                                                                                         Architecture                                                                                      Version                                                                                                     Repository                                                                                                  Size
================================================================================================================================================================================================================================================================================================================================================================================================================================
Downgrading:
 chef                                                                                            x86_64                                                                                            13.10.0-1.el7                                                                                               cloud-common-64                                                                                             49 M

Transaction Summary
================================================================================================================================================================================================================================================================================================================================================================================================================================
Downgrade  1 Package

Total download size: 49 M
Downloading Packages:
chef-13.10.0-1.el7.x86_64.rpm                                                                                                                                                                                                                                                                                                                                                                    31 MB/s |  49 MB     00:01
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                                                                                                                                                                            31 MB/s |  49 MB     00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                                                                                                                                                                        1/1
  Downgrading      : chef-13.10.0-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              1/2
  Running scriptlet: chef-13.10.0-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              1/2
Thank you for installing Chef!

  Cleanup          : chef-14.15.6-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              2/2
  Running scriptlet: chef-14.15.6-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              2/2
  Verifying        : chef-13.10.0-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              1/2
  Verifying        : chef-14.15.6-1.el7.x86_64                                                                                                                                                                                                                                                                                                                                                                              2/2

Downgraded:
  chef-13.10.0-1.el7.x86_64

Complete!

##########  Run go2chef again - this time it parses the right version and honors it! ########## 


[centos@ip-172-16-1-244 ~]$ sudo /usr/local/bin/go2chef --local-config /etc/go2chef.config-broken.json

          ___    _         __
 __ _ ___|_  )__| |_  ___ / _|
/ _` / _ \/ // _| ' \/ -_)  _|
\__, \___/___\__|_||_\___|_|
|___/

GO2CHEF 2022/06/08 21:30:30 loading config from source go2chef.config_source.local
GO2CHEF 2022/06/08 21:30:30 EVENT: LOGGING_INITIALIZED in go2chef.cli -
GO2CHEF 2022/06/08 21:30:30 EVENT: STEP_0_START go2chef.step.install.linux.dnf:'install chef-13' in go2chef.cli -
GO2CHEF 2022/06/08 21:30:30 INFO: third-party-source/go/github.com/facebookincubator/go2chef/plugin/step/install/linux/dnf/dnf.go:204::Package is already installed: chef-13.10.0-1.el7.x86_64, requested ^chef-13.10.0-1.el7.*
GO2CHEF 2022/06/08 21:30:30 INFO: third-party-source/go/github.com/facebookincubator/go2chef/plugin/step/install/linux/dnf/dnf.go:132::chef specified is already installed, not reinstalling
GO2CHEF 2022/06/08 21:30:30 EVENT: STEP_0_COMPLETE go2chef.step.install.linux.dnf:'install chef-13' in go2chef.cli - completed successfully in 0 second(s)
GO2CHEF 2022/06/08 21:30:30 EVENT: ALL_STEPS_COMPLETE in go2chef.cli - 1 step(s) completed successfully in 0 second(s)
2022/06/08 21:30:30 temp dirs cleanup completed

Support caching for HTTP source

Add caching support for the HTTP source. This allows us to:

  1. Avoid redownloading potentially very large files repeatedly on failures
  2. Preserve bootstrap files locally for re-bootstrapping in limited-access scenarios (i.e. roaming client machines)

Support aws sources defaulting to derived local region

AWS-specific sources will often be used from an aws instance with access to the aws metadata service. This is already used by the sdk to provide credentials if they aren't in the config. However the sdk doesn't provide a way to automatically derive the current region before creating a session even though the metadata service will tell you where you are.

Instead, when using a source like s3 or secretsmanager, if region isn't specified, manually query the metadata to find our current region and use it if available. To do this, basically:
curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone
Then strip off the last character and that will be your region.

[Feature Request] Support specifying client key/cert when installing package using `go2chef.step.install.linux.dnf`

There is no way to provide sslclientcert/sslclientkey inside go2chef.step.install.linux.dnf - hence you can't authenticate with a yum endpoint while downloading a package.

GO2CHEF 2022/06/08 19:40:32 EVENT: STEP_3_FAILURE go2chef.step.install.linux.dnf:'install chef-13' in go2chef.cli - Get "https://<redacted_uri>/yum/centos/8.x/site-packages/x86_64/Packages/chef-13.10.0-1.el7.x86_64.rpm": remote error: tls: certificate required

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.