cloudscripts's Introduction

CloudScripts

This is a collections of different scripts used for software and server provisioning in the context of cloud, usually docker, some networking configurations and so forth.

I used to use ansible in a push strategy approach (from client or bastion to servers), however with the introduction of different HashiCorp tools, this strategy started to be cumbersome. Instead, eeach server will execute their related scripts using environment knowledge provided by the cloud platform like Metadata server, labels & tags, etcd servers, in a pull strategy approach.

For the time being, this runs over GCE cloud, debian like S.O., and systemd, but it should be easy to adapt for other providers and S.O, besides some scripts are agnostics like docker installation script. Let me know if you are interest in other providers, or tools.

A terraform sandbox is included for testing purposes.

Tools used:

Shell scripts
Gomplate for templating in services
Maybe python3 in the future using only the standard library

Starting

curl -Ls https://raw.githubusercontent.com/nuxion/cloudscripts/main/install.sh | bash

or if you want to pin a version:

https://raw.githubusercontent.com/nuxion/cloudscripts/<version>/install.sh

Where <version> should match with the tags available in this repo.

or if you want to check the install code first:

curl -Ls https://raw.githubusercontent.com/nuxion/cloudscripts/main/install.sh -o install.sh
chmod +x install.sh
./install.sh

From VERSION<=0.5.0, a version could be pined from the installation:

curl -Ls https://raw.githubusercontent.com/nuxion/cloudscripts/main/install.sh | VERSION=0.7.0 sh

⚠️ Note: If you are using it in a ci/cd workflow, some system requires an apt-get update first.

after installation you can use standalone scripts or the command line:

cscli -i nvidia-driver

Example:

Installations avalaible could be found in scripts/commands dir.

Output example:

nuxion@gce-small-cpu-j0z8dr:~/cloudscripts$ sudo cscli -i nvidia-docker
=> cuda found!
=> docker found!
=> nvidia-docker2 not found!
=> nvidia-docker2 installing... be patient it will take some minutes
....
....
=> nvidia-docker2 installed!

For command available check commands

Stability

It is being used in production for quick provisiong using packer as image builder: mostly for docker and docker-compose.

I try to follow semver, it's mean cscli will have a Stable API during minor versions, for now it is very simple:

cscli -i <pkg_name>

Options could be added between minor releases during 0.x.z releases, but never a new option added, will be deleted without a mayor version change.

Meanwhile internal structure inside scripts/ could change, how commands are defined internally could change, but the command dir and the way in that "source" import other scripts as dependencies will be maintained through minor releases.

Finally, the install.sh script only downloads the last tag from github and never deletes old installations (usually in /opt/cloudscripts-${VERSION} folder) but it replaces /usr/local/bin/cscli pointing to the new version installed.

The observant people will notice a GAP between version 0.2.0 and 0.5.0. The versions in between were buggy and they were removed.

We follow semantic version. Until we reach version 1.0.0, Minor version is used as breaking changes between the API and the structure of the command scripts.
New commands (software available) added will modify the patch part of the version: MAJOR.MINOR.patch

Concepts

commads: are mostly one shot installer of some dependency needed for running some service, for instance docker, a nvidia-cuda driver. A command could have dependecies between them (that is the case of nvdidia-docker which it has: nvdia-driver and docker as dependecies). This strategy resolve in a quick way the problem but could be error prone and generate circular dependecies because all of this depends on the order that others scripts are imported as sources in the header of a command definition. So, with a simple solution comes a big responsability ;)
services: ideally, more complex installations like a nginx web server that will require allso more dynamically configurations from others services, and complex initializacions like a db, should come here. They will be using gomplate to render configurations files needed based on the environment and external services like redis/etcd/consul or zookeeper.

In the future other special folders could be added like secrets and certs. Or special hooks for user creations, dns changes, routes add and so forth, maybe those actions will be performed by a serf agent.

Customizing

I believe that the interface is very simple and easy to respect, so changes shouln't be the norm .

Custom commands could be added inside the "scripts/commands", the only thing that they need to do is to validate if it's is installed or not, perfom the action required, verify the installation, and execute a exit 1 if the installation failed for some reason to avoid that other scripts run.

Also all the scripts are using variables and relative paths, base path and variables can be changed in cscli, env.sh and install.sh in a forked project.

Release

Update versions files:

make release VERSION=${NEW_VERSION}

git add . && git commit && git tag -a ${NEW_VERSION} && git push

Inspirations

Cloud provider detection

check: https://github.com/dgzlopes/cloud-detect/blob/master/cloud_detect/providers/azure_provider.py /sys/class/dmi/id/sys_vendor

cloudscripts's People

Contributors

Stargazers

Watchers

cloudscripts's Issues

nvidia card not found - lspci

When i try to install nvidia driver, a warning is raised because nvidia card is not found, but the problem is lspci command:

/opt/cloudscripts-0.2.0/scripts/commands/nvidia-driver.sh: line 58: lspci: command not found
WARNING: NVIDIA card not found, the installation could fail

Add it as dependency or change the error message output

add nginx util for site enabling/disabling

https://serverfault.com/questions/424452/nginx-enable-site-command

#!/bin/bash

##
#  File:
#    nginx_modsite
#  Description:
#    Provides a basic script to automate enabling and disabling websites found
#    in the default configuration directories:
#      /etc/nginx/sites-available and /etc/nginx/sites-enabled
#    For easy access to this script, copy it into the directory:
#      /usr/local/sbin
#    Run this script without any arguments or with -h or --help to see a basic
#    help dialog displaying all options.
##

# Copyright (C) 2010 Michael Lustfield <[email protected]>

# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

##
# Default Settings
##

NGINX_CONF_FILE="$(awk -F= -v RS=' ' '/conf-path/ {print $2}' <<< $(nginx -V 2>&1))"
NGINX_CONF_DIR="${NGINX_CONF_FILE%/*}"
NGINX_SITES_AVAILABLE="$NGINX_CONF_DIR/sites-available"
NGINX_SITES_ENABLED="$NGINX_CONF_DIR/sites-enabled"
SELECTED_SITE="$2"

##
# Script Functions
##

ngx_enable_site() {
    [[ ! "$SELECTED_SITE" ]] &&
        ngx_select_site "not_enabled"

    [[ ! -e "$NGINX_SITES_AVAILABLE/$SELECTED_SITE" ]] && 
        ngx_error "Site does not appear to exist."
    [[ -e "$NGINX_SITES_ENABLED/$SELECTED_SITE" ]] &&
        ngx_error "Site appears to already be enabled"

    ln -sf "$NGINX_SITES_AVAILABLE/$SELECTED_SITE" -T "$NGINX_SITES_ENABLED/$SELECTED_SITE"
    ngx_reload
}

ngx_disable_site() {
    [[ ! "$SELECTED_SITE" ]] &&
        ngx_select_site "is_enabled"

    [[ ! -e "$NGINX_SITES_AVAILABLE/$SELECTED_SITE" ]] &&
        ngx_error "Site does not appear to be \'available\'. - Not Removing"
    [[ ! -e "$NGINX_SITES_ENABLED/$SELECTED_SITE" ]] &&
        ngx_error "Site does not appear to be enabled."

    rm -f "$NGINX_SITES_ENABLED/$SELECTED_SITE"
    ngx_reload
}

ngx_list_site() {
    echo "Available sites:"
    ngx_sites "available"
    echo "Enabled Sites"
    ngx_sites "enabled"
}

##
# Helper Functions
##

ngx_select_site() {
    sites_avail=($NGINX_SITES_AVAILABLE/*)
    sa="${sites_avail[@]##*/}"
    sites_en=($NGINX_SITES_ENABLED/*)
    se="${sites_en[@]##*/}"

    case "$1" in
        not_enabled) sites=$(comm -13 <(printf "%s\n" $se) <(printf "%s\n" $sa));;
        is_enabled) sites=$(comm -12 <(printf "%s\n" $se) <(printf "%s\n" $sa));;
    esac

    ngx_prompt "$sites"
}

ngx_prompt() {
    sites=($1)
    i=0

    echo "SELECT A WEBSITE:"
    for site in ${sites[@]}; do
        echo -e "$i:\t${sites[$i]}"
        ((i++))
    done

    read -p "Enter number for website: " i
    SELECTED_SITE="${sites[$i]}"
}

ngx_sites() {
    case "$1" in
        available) dir="$NGINX_SITES_AVAILABLE";;
        enabled) dir="$NGINX_SITES_ENABLED";;
    esac

    for file in $dir/*; do
        echo -e "\t${file#*$dir/}"
    done
}

ngx_reload() {
    read -p "Would you like to reload the Nginx configuration now? (Y/n) " reload
    [[ "$reload" != "n" && "$reload" != "N" ]] && invoke-rc.d nginx reload
}

ngx_error() {
    echo -e "${0##*/}: ERROR: $1"
    [[ "$2" ]] && ngx_help
    exit 1
}

ngx_help() {
    echo "Usage: ${0##*/} [options]"
    echo "Options:"
    echo -e "\t<-e|--enable> <site>\tEnable site"
    echo -e "\t<-d|--disable> <site>\tDisable site"
    echo -e "\t<-l|--list>\t\tList sites"
    echo -e "\t<-h|--help>\t\tDisplay help"
    echo -e "\n\tIf <site> is left out a selection of options will be presented."
    echo -e "\tIt is assumed you are using the default sites-enabled and"
    echo -e "\tsites-disabled located at $NGINX_CONF_DIR."
}

##
# Core Piece
##

case "$1" in
    -e|--enable)    ngx_enable_site;;
    -d|--disable)   ngx_disable_site;;
    -l|--list)  ngx_list_site;;
    -h|--help)  ngx_help;;
    *)      ngx_error "No Options Selected" 1; ngx_help;;
esac

Recommend Projects

nuxion / cloudscripts Goto Github PK