rawiron / devops-my-ubuntu

clone my ubuntu setup using Ansible playbook

My Ubuntu Playbooks

Rebuild my Ubuntu laptop using Ansible playbooks. Just for the fun of it and to learn more about Ansible. Also automate a bunch of setup steps, eg add a new git repo to my virtualenvwrapper and bind to a the project home. A Vagrant box is used to test the Ansible playbooks.

Restore 1st steps

• create a HOME directory for the user
• install rclone
• get the rclone config from the Password Manager
• place the rclone config in ~/.config/rclone/rclone.conf
• restore secret files from encrypted cloud storage
• install git
• clone this repo from github
• create a python virtual environment
• install the requirements: wheel, ansible, mitogen, ..
• deploy

Test on the vagrant box

First create a vagrant box

$ vagrant up

Run defined tests on the vagrant box.

laptop$ vagrant ssh --command "cd /vagrant; ANSIBLE_INVENTORY=hosts ./test.sh"

Then test the ansible playbooks on the vagrant box

• all tasks

$ vagrant ssh
vagrant$ cd /vagrant
vagrant$ ansible-playbook --inventory-file=hosts deploy.yml

• one role at a time using the tags defined in deploy.yml

vagrant$ ansible-playbook --inventory-file=hosts deploy.yml --tags setup

Deploy from laptop onto the vagrant box

• deploy onto the vagrant box via ssh

laptop$ ansible-playbook --inventory-file=.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory deploy.yml --tags terminator

Run on laptop

In my usual workflow I change the ansible script, for example add a new alias aliases.zsh, and then deploy to my laptop with

laptop$ ansible-playbook --inventory-file=hosts deploy.yml --tags=alias --extra-vars 'as_root=no'

Use Ansible container instead of local Ansible install

docker pull cytopia/ansible:latest

docker run --rm -v $(pwd):/data cytopia/ansible ansible-playbook --inventory-file=hosts deploy.yml

Categories and the major software that I use

devops

automate everything and so everything is reproducible. it can be run again and again. makes testing, load testing, building and deploying easy. also great for scaling.

• virtualbox
• vagrant
• ansible
• docker

aws

tools to access, manage and deploy.

• jungledisk
• s3cmd
• awscli
• rclone

jvm development

• scala
• java runtime
• maven and sbt

python development

great python packages to help writing great code. use functional programming, TDD, dependency injection, database abstraction, database migration.

• virtualenv
• nose, pyvows, hypothesis
• toolz, pyrsistent, effect
• inject
• sqlalchemy, agnostic

system programming

programming languages for high performance and system level access.

• dmd (d compiler)
• C, C++

data analysis

wrangle, plot, predict, test hypothesis, estimate.

py-data

• anaconda
• numpy, scipy, sympy, Jupyter, Seaborn, scikit-learn

development tools

write code with fully fledged IDE or lightweight editor. plugins can be used to extend the functionality.

• intellij idea
• vscode
• vim
• git
• meld

intellij plugins

• maven helper
• scala
• python
• elixir
• markdown support

vim plugins

• YouCompleteMe
• syntastic
• UltiSnips
• NerdTree
• tagbar
• vim-fugitive
• vim-airline

system tools

look under the hood.

performance

show and measure what is happening in the Linux kernel.

• SystemTap, perf_events, dtrace4linux, ktap, lttng
• strace, ftrace, ltrace, ss, sysdig, tiptop
• mpstat, vmstat, netstat, iostat

user account

github repos

• create workspace(s) in HOME
• git clone projects into the workspace(s)

office tools

communicate. create and manage documents, files.

• firefox
• libre office
• FocusWriter
• artha thesaurus
• Kazam
• Screenkey
• 7zip, bzip2
• Midnight Commander

firefox plugins

• FileZilla

media

view and process photos, e-books, audio and video.

• rawtherapee
• vlc media player
• ripperX
• dvdrip
• OpenShot
• Calibre

games

• pychess

eye candy

give Linux the look-and-feel I like. unfortunately haven`t figured out how to configure a "glass look"

• all backgrounds are transparent, blurred
• drop-down menu are transparent, blurred
• ...

tools for UI tweaks. did a few tweaks.

• Compiz config manager (CCSM)

sudo apt-get install ccsm

Compiz config settings manager, Unity plugin

• set opacity for launcher
• set opacity for panel (or use Unity Tweak Tool)

browse and download a theme

gnome-look.org
sudo apt-get install arc-theme

set theme in Unity Tweak Tool

• current theme is Arc-dark

change desktop background

• ghost-in-the-shell

terminator

• transparency of background

devops-my-ubuntu's Issues

install R commander

open the R interpreter (from the command line: R) and run install.package("Rcmdr") in the interpreter. Once this finishes, then run library(Rcmdr).
Allow it to install missing/updated dependencies.

build on travis-ci

Start as simple as running a syntax check only for the playbooks.

Travis-Ci supports 3 different build environments. This is nicely explained in this document.
Usage of any other build environment requires containers.
To test the ansible scripts against many different distributions or to test against the latest distributions (Ubuntu 16.04 for example) containers are required.

There should be a test procedure which runs locally first. In case the tests pass the code can be pushed to remote.
Pushing to remote and test if things are working on Travis-Ci is a bad idea.

install jekyll

• install jekyll
  sudo apt-get install jekyll
• configure github pages
• read blog from github pages

fix broken build

the pre-installed erlang and elixir versions done with kerl and kiex causing the issue

kerl list installations
17.0
17.1
17.3
17.4
17.5
18.0
18.1
18.2
18.2.1
R14B02
R14B03
R14B04
R15B
R15B01
R15B02
R15B03
R16B
R16B01
R16B02
R16B03
R16B03-1

kiex list

kiex elixirs
   elixir-1.0.3
=* elixir-1.0.4

# => - current
# =* - current && default
#  * - default

deploy AWS S3 keys

copy access and secret key to config files
.s3cfg

kernel config and parameters

kernel config

build a custom kernel from source:
http://www.linuxtopia.org/online_books/linux_kernel/kernel_configuration/ch08.html

/boot/config-4.8.0-32-generic

kernel parameters

Listing all available kernel parameters

Using "sysctl -a", the full list of kernel parameters that can be controlled through the sysctl mechanism is displayed, together with their current setting.

Manual setting of kernel parameters

The "sysctl" command can be used to change a kernel setting, e.g.

sysctl -w net.ipv4.conf.eth0.rp_filter=1

These changes affect the currently running kernel. They will be lost on reboot.
To change settings permanently, add an appropriate line to the /etc/sysctl.conf file which is processed during system startup through the /etc/init.d/boot.sysctl script.

install libfcgi0ldbl - Shared library of FastCGI

sudo apt-get install libfcgi0ldbl

the lib includes the executable cgi-fcgi

which is handy for talking to fcgi directly:
http://stackoverflow.com/questions/30238602/how-to-execute-a-php-file-using-a-php5-fpm-pool-socket

install and test SystemTap

install instructions on systemtap and Ubuntu Kernel

sudo apt-get install gcc systemtap

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622

codename=$(lsb_release -c | awk  '{print $2}')
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename}      main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

sudo apt-get update
sudo apt-get install linux-image-$(uname -r)-dbgsym

git - configure global

email
name
difftool

install Java Runtime

java open sdk
oracle java 7 runtime
oracle java 7 webstart
oracle java 8 runtime
oracle java 8 webstart

git clone core-utils in workspace_open

git clone git://git.sv.gnu.org/coreutils

install inotify-tools

short intro into inotify on infoq article

apt-get install inotify-tools

add in system-tools

docker - install tool docker-enter

https://github.com/Pithikos/docker-enter

install docker

docker install and config

• Docker package
  https://docs.docker.com/engine/installation/linux/ubuntulinux/
• Ubuntu package

sudo apt-get install -y docker.io

sudo usermod -aG docker <your username>
sudo service docker start
systemctl enable docker

install composer for php

curl -sS https://getcomposer.org/installer | php
sudo mv composer.phar /usr/local/bin/composer

install sublime text 3

sudo add-apt-repository ppa:webupd8team/sublime-text-3
sudo apt-get update
sudo apt-get install sublime-text-installer

zsh - docker plugin and aliases

• Delete all containers
  docker rm $(docker ps -a -q)
• Delete all images
  docker rmi $(docker images -q)

for cases an image has more than one tag
docker images <repository-name> | tail -n +2 | awk '{ print $1 ":" $2}' | xargs docker rmi

configure maven 3.1 for ubuntu 14.04

if ubuntu 14.04:
maven 3.1, append to .profile

export M2_HOME=/usr/local/apache-maven-3.1
export M2=$M2_HOME/bin
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$M2:$PATH

install php dev

interpreter, cli, debugger, ...

xdebug, nginx for docker?

for trusty64

sudo apt-get install php5 # 5..5.x
sudo apt-get install php5-cli
sudo apt-get install phpunit # 3.7.x

for wily64

• php 5.6.x
• phpunit 4.7.x

install ack and vim-ack

ack

sudo apt-get install ack-grep 

add to .vimrc

Plugin 'mileszs/ack.vim'

vim - configure python ide

code completion (IntelliSense) and navigation

for any code completion in vim to work the omnicomplete feature is needed.
omnicomplete is available since vim 7.

filetype plugin on
set omnifunc=syntaxcomplete#Complete

Python

different options for python are available.
all of them need vim to be compiled with the pythoninterp.
check vim for +python or +python3:

vim --version

• pythoncomplete plugin
• jedi-vim
• YouCompleteMe

sudo apt-get install python3-pip
pip3 install future frozendict

git clone https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim
vim +PluginInstall +qall
cd ~/.vim/bundle/YouCompleteMe
git submodule update --init --recursive

sudo apt-get install cmake
cd ~/.vim/bundle/YouCompleteMe
./install.py --clang-completer

ctags

install exuberant-ctags

apt-get install exuberant-ctags

navigate ctags with tagbar

enable exuberant-ctags

let g:tagbar_ctags_bin = '/usr/bin/ctags-exuberant'

project view

source-code browser NerdTree

snippets

for code snippets UltiSnips is great.

syntax highlighting

syntax on

check syntax and inspect

use syntastic

run and test

:make