jorymorrison / hands-on-devops Goto Github PK

1. DevOps

A hands-on DevOps course covering the culture, methods and repeated practices of modern software development involving Vagrant, VirtualBox, Ansible, Docker-Compose, Docker, Taiga, GitLab, Drone CI, SonarQube, Selenium, InSpec...

A reveal.js presention written to accompany this course can found at https://nemonik.github.io/hands-on-DevOps/.

This course will

Discuss DevOps,
Have you spin up a DevOps toolchain and development environment, and then
Author two applications and their accompanying pipelines, the first a continuous integration (CI) and the second a continuous delivery (CD) pipeline.

After this course you will

Be able to describe and have hands on experience the DevOps methods and repeated practices (e.g., use of Agile methods, configuration management, build automations, test automation and deployment automation orchestrated under continuous integration and delivery orchestrator), and why it matters;
Identify what diverse tools and resources that exist in the DevOps ecosystem;
Be able to address challenges sponsors face by transitioning to DevOps methods and repeated practices;
Have had hands-on experience with Infrastructure as Code( Vagrant and Ansible ) to provision and configure an entire DevOps toolchain and development environment on VirtualBox including Docker Registry, GitLab, Drone CI, and SonarQube;
Have had hands-on experience authoring code to include authoring and running automated tests in a CI/CD pipeline all under Configuration Management to insure an application follows style, adheres to good coding practices, builds, identify security issues, and functions as expected;
Have had hands-on experience with (a) using Infrastructure as Code (IaC) in Vagrant and Ansible; (b) creating and using Kanban board in Taiga; (c) code configuration in git and GitLab; (d) authoring code in Go; (e) using style checkers and linters; (f) authoring a Makefile; (g) various commands in Docker (e.g., building a container image, pushing a container into a registry, creating and running a container); (h) authoring a pipeline for Drone CI; (i) using Sonar Scanner CLI to perform static analysis; (j) authoring security test in InSpec; (k) author an automated functional test in Selenium; (l) authoring a dynamic security test in OWASP Zap; an (m) using container platform to author and scale services;
Have had hands-on experience authoring code to include authoring and running automated tests in a CI/CD pipeline all under Configuration Management to insure an application follows style, adheres to good coding practices, builds, identify security issues, and functions as expected.

We will be spending most of the course hands-on working with the tools and in the Unix command line making methods and repeated practices of DevOps happen, so as to grow an understanding of how DevOps actually works.

Don't fixate on the tools used, nor the apps we develop in the course of learning how and why. How and why is far more important. This course like DevOps is not about tools although we'll be using them. You'll spend far more time writing code. (Or at the very least cutting-and-pasting code.)

2. Author

Michael Joseph Walsh [email protected], [email protected]

3. Thank you to

Walter Hiranpat for running through the course material for my first class prior to me teaching it.
Eric McCann for assisting Walter and I work through Vagrant install issues on Microsoft Windows.
My first class participants for "beta" testing my class.
Jonathan Thomas for TA'ing my second class.
David Trang for TA'ing my class countless times.
Walter Hiranpat, David Trang, and Rony Xavier for prepping the MITRE Institute classroom for my class countless times.
Rony Xavier and Aaron Lippold for assisting me mature the InSpec section.

4. Copyright and license

See the License file at the root of this project.

5. Prerequisites

The following skills would be useful in following along, but aren't strictly necessary.

What you should bring:

Managing Linux or Unix-like systems would be tremendously helpful, but not necessary, as we will living largely within the terminal.
A basic understanding of Vagrant, Docker, and Ansible would also be helpful, but not necessary.
Mostly being a software engineer, but not necessary.

6. Table of contents

1. DevOps
2. Author
3. Thank you to
4. Copyright and license
5. Prerequisites
6. Table of contents
7. DevOps unpacked
8. Reading list
9. Now the hands-on part

7. DevOps unpacked

7.1. What is DevOps?

DevOps (a clipped compound of the words "development" and "operations") is a software development methodology with an emphasis on a more reliable release pipeline, automation, and stronger collaboration across all stakeholders with the goal of deploying working functionality into the hands of users (i.e., production) faster.

Yeah, that's the formal definition. I've grown to prefer the axiom:

You are what you code.

For example,

If you're a developer or software engineer, you're at least coding the application, its build automation, unit tests, and CI/CD (the combined practices of Continuouse Integrarion and Continuous Delivery) automation.
If a tester, you're coding typically the functional test automation.
If a security engineer, you're coding the compliance and policy test automation.
If ops, you're coding the deployment and infrastructure configuration management automation.

And since all these disciplines are coding, they're essentially using the same methods and repeated practices to ensure they're producing good code.

And they're all collectively doing all this coding left in the delivery pipeline, collaborating and from this springs forth culture.

7.2. What DevOps is not

DevOps is not entirely about tools.

Or as I like to put it, "Its not about the tools."

DevOps will also not entirely stop all bugs nor all vulnerabilities from making it into production, but that's not really the point.

_{There are countless vendors out there, who want to sell you their crummy tool.}

7.3. DevOps is really about

Providing the culture and delivery/release pipeline that once a bug or vulnerability is discovered, the concern can to be quickly remediated and functionality returned back to the user.

7.4. How is it related to the Agile?

Agile Software Development is an umbrella term for a set of methods and practices based on the values and principles expressed in the Agile Manifesto.

Agile Software Development shares the same goal, but DevOps extends Agile methods and practices by adding communication and collaboration between

development,
quality assurance, and
technology operations
to ensure software systems are delivered in a rapid, reliable, low-risk manner.

For Agile, solutions evolve through collaboration between self-organizing, cross-functional teams utilizing the appropriate practices for their context.

Again, in DevOps everyone is developing software, so it is my view DevOps builds on Agile.

7.5. How do they differ?

While Agile Software Development encourages collaboration between cross-functional teams, the focus in DevOps is on the

inclusion of analysis,
design,
development, and
quality assurance functionaries as stakeholders into the development effort.

7.6. Why?

In Agile Software Development, there is rarely an integration of these individuals outside the immediate application development team with members of technology operations (e.g., network engineers administrators, testers, cyber security engineers.)

7.7. What are the principles of DevOps?

As DevOps matures, several principles have emerged, namely the necessity for product teams to:

Apply holistic thinking to solve problems,
Develop and test against production-like environments,
Deploy with repeatable, and reliable processes,
Remove the drudgery through automation,
Validate and monitor operational quality, and
Provide rapid, automated feedback to the stakeholders

7.8. How is this achieved?

Achieved through the repeated practices of Continuous Integration (CI) and Continuous Delivery (CD) often conflated into simply "CI/CD".

After tools, this is what is commonly (albeit mistakenly) thought to be the totality of DevOps.

7.9. What is Continuous Integration (CI)?

It is a repeated Agile software development practice, lifted specifically from Extreme programming, where members of a development team frequently integrate their work in order to detect integration issues as quickly as possible thereby shifting discovery of issues "left" (i.e., early) in the software release.

7.10. How?

Each integration is orchestrated through a CI service/orchestrator (e.g., Jenkins CI, Drone CI, GitLab Runners, Concourse CI) that essentially assembles a build, runs unit and integration tests every time a predetermined trigger has been met; and then reports with immediate feedback.

7.11. CI best practices

7.11.1. Utilize a Configuration Management System

For the software's source code, where the mainline (i.e., master branch) is the most the most recent working version, past releases held in branches, and new features not yet merged into the mainline branch being worked on their own branches.

7.11.2. Automate the build

By accompanying build automation (e.g., Gradle, Apache Maven, Make) alongside the source code.

7.11.3. Employ one or more CI services/orchestrators

Perform source code analysis via automate formal code inspection and assessment.

7.11.4. Make builds self-testing

In other words, ingrain testing by including unit and integration tests (e.g., Spock, JUnit, Mockito, SOAPUI, go package Testing) with the source code so as to be executed by the build automation to be execute by the CI service.

7.11.5. Never commit broken

Or untested source code to the CMS mainline or otherwise risk breaking a build.

7.11.6. Developers are expected to pre-flight new code

Prior to committing source code in their own workspace.

7.11.7. The CI service/orchestrator provides feedback

On the success or fail of a build integration to all its stakeholders.

7.12. What is Continuous Delivery?

It is a repeated software development practice of providing a rapid, reliable, low-risk product delivery achieved through automating all facets of building, testing, and deploying software.

7.12.1. Extending Continuous Integration (CI)

With additional stages/steps aimed to provide ongoing validation that a newly assembled software build meets all desired requirements and thereby is releasable.

7.12.2. Consistency

Is achieved through delivering applications into production via individual repeatable pipelines of ingrained system configuration management and testing

7.13. But wait. What's a pipeline?

A pipeline automates the various stages/steps (e.g., Static Application Security Testing (SAST), build, unit testing, Dynamic Application Security Testing (DAST), secure configuration acceptance compliance, integration, function and non-functional testing, delivery, and deployment) to enforce quality conformance.

7.14. How is a pipeline manifested?

Each delivery pipeline is manifested as Pipeline as Code (i.e., software automation) accompanying software's source code in its version control repository.

7.15. What underlines all of this?

I'd argue, a ubiquitous access to shared pools of configurable system resources and higher-level services that can be rapidly provisioned with minimal management effort without the repeated practices of DevOps will struggle. Although, it is possible to DevOps on mainframes.

7.16. But really why do we automate err. code?

In 2001, I think Larry Wall in his 1st edition of Programming Perl book put it best with "We will encourage you to develop the three great virtues of a programmer:

laziness,

impatience, and

hubris."

The second edition of the same book provided definitions for these terms

7.16.1. Why do I mention Larry Wall?

Well...

Once you have established yourself as an icon in your field it is important that you pay tribute to some of the great legends that came before you. This kind of gesture will create the illusion that you’re still humble and serve as a preemptive strike against anyone who has noticed what a callus and delusional ass you have become.

The opening monolog to the Blue Man Group’s I Feel Love https://www.youtube.com/watch?v=8vBKI3ya-l0

I kid, but in all serious the sentimanet of this seminal book still holds true.

Let me explain.

7.16.2. Laziness

The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don't have to answer so many questions about it. Hence, the first great virtue of a programmer. (p.609)

7.16.3. Impatience

The anger you feel when the computer is being lazy. This makes you write programs that don't just react to your needs, but actually anticipate them. Or at least pretend to. Hence, the second great virtue of a programmer. (p.608)

7.16.4. Hubris

Excessive pride, the sort of thing Zeus zaps you for. Also, the quality that makes you write (and maintain) programs that other people won't want to say bad things about. Hence, the third great virtue of a programmer. (p.607)

7.16.5. We automate for

Faster, coordinated, repeatable, and therefore more reliable deployments.
Discover bugs sooner. Shifting their discovery left in the process.
To accelerates the feedback loop between Dev and Ops.
Reduce tribal knowledge, where one group or person holds the keys to how things get done. Yep, this is about making us all replaceable.
Reduce shadow IT (i.e., hardware or software within an enterprise that is not supported by IT. Just waiting for its day to explode.)

7.17. Monitoring

Once deployed, the work is done, right?

A development team's work is not complete once a product leaves CI/CD and enters production; especially, under DevOps where the development teams include members of operations.

7.17.1. The most important metric

Is working software is the primary, but not the only, measure of progress. The key to successful DevOps is knowing how well the methodology and the software it produces are performing.

7.17.2. An understanding of performance

Is achieved by collecting and analyzing data produced by environments used for CI/CD and production.

7.17.3. Establish a baseline performance

So, that improvements can be gauged and anomalies detected.

7.17.4. Set reaction thresholds

To formulate and prioritize reactions weighting factors, such as, the frequency at which an anomaly arises and who is impacted.

7.17.5. Reacting

A reaction could be as simple as operations instructing users through training to not do something that triggers the anomaly, or more ideally, result in an issue being entered into the product's backlog culminating in the development team delivering a fix into production.

7.17.6. Gaps in CI/CD

Monitoring will also inform development teams of gaps in CI/CD resulting in additional testing for the issue that triggered the necessity for the improvement.

7.17.7. Eliminating waste

Further, monitoring may result in the re-scoping of requirements, re-prioritizing of a backlog, or the deprecation of un-used features.

7.18. What is DevOps culture?

culture noun \ ˈkəl-chər
the set of shared attitudes, values, goals, and practices that characterizes an institution or organization

Now that everyone is a coder using the same tools, methods and repeated practices for their particular discipline you have the foundations of a culture.

7.18.1. We were taught the requisite skills as children

7.18.2. Maintaining relationships is your most important skill

7.18.3. Be quick... Be slow to...

7.18.4. The pressures of social media

7.19. Crawl, walk, run

7.19.1. Ultimately, DevOps is Goal

With DevOps one does not simply hit the ground running.
One must first crawl, walk and then ultimately run as you embrace the necessary culture change, methods and repeated practices.
Collaboration and automation are expected continually improve so to achieve more frequent and more reliable releases.

8. Reading list

AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis William J. Brown, Raphael C. Malveau, Hays W. "Skip" McCormick, and Thomas J. Mowbray ISBN: 978-0-471-19713-3 Apr 1998

Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) David Farley and Jez Humble ISBN-13: 978-0321601919 August 2010

The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations Gene Kim Jez Humble, Patrick Debois, and John Willis ISBN-13: 978-1942788003 October 2016

Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations Nicole Forsgren PhD, Jez Humble, and Gene Kim ISBN-13: 978-1942788331 March 27, 2018

Site Reliability Engineering: How Google Runs Production Systems 1st Edition Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy ISBN-13: 978-1491929124 April 16, 2016 Also, available online at https://landing.google.com/sre/book/index.html

Release It!: Design and Deploy Production-Ready Software 2nd Edition Michael T. Nygard ISBN-13: 978-1680502398 January 18, 2018

The SPEED of TRUST: The One Thing That Changes Everything Stephen M .R. Covey ISBN-13: 978-1416549000 February 5, 2008 The gist of the book can be found at SlideShare https://www.slideshare.net/nileshchamoli/the-speed-of-trust-13205957

How to Deal With Difficult People Ujjwal Sinha Oct 25, 2014 The SlideShare can be found here https://www.slideshare.net/abhiujjwal/how-2-deal-wid-diiclt-ppl

9. Now the hands-on part

In this class you will spin up the following developent and toolchain environment.

NOTE

If your web-based Git-repository manager is paired with a PlantUML server you will see a diagram rendered below otherwise you will only the PlantUML source code for the diagram. Don't worry the course's Ansible automation will spin up both a GitLab and PlantUML.
This class makes use of NOTE sections to call out things that are important to know or will drop a few tidbits. Reading these notes may save you some aggrevation.

@startuml
skinparam shadowing false

skinparam actor {
        BorderColor #0B5C92
        BackgroundColor none
        FontName "Yanone Kaffeesatz"
        FontStyle "Thin"
        FontSize 18
}

skinparam node {
        BorderColor #0B5C92
        BackgroundColor #ffffff
        FontName "Yanone Kaffeesatz"
        FontStyle "Thin"
        FontSize 15
}

skinparam rectangle {
        roundCorner 25
        BorderColor #0B5C92
        BackgroundColor #ffffff
        FontName "Yanone Kaffeesatz"
        FontStyle "Thin"
        FontSize 15
}

skinparam component {
        BorderColor #0B5C92
        BackgroundColor #e0e59a
        FontName "Yanone Kaffeesatz"
        FontStyle "Thin"
        FontSize 15
}


skinparam note {
        BorderColor #0B5C92
        BackgroundColor #FEFECE
        FontName "Yanone Kaffeesatz"
        FontStyle "Thin"
        FontSize 15
}

skinparam database {
        BorderColor #0B5C92
        BackgroundColor #e0e59a
        FontName "Yanone Kaffeesatz"
        FontStyle "Thin"
        FontSize 15
}

actor "You" as you

rectangle "Host"{
    
	rectangle "Toolchain Vagrant" {

		rectangle "Docker CE" as docker1 {

			rectangle "Taiga" {
				component "nemonik/taiga:4.0.4" as taiga
        component "sameersbn/postgresql:9.6-4" as taiga_postgresql
			}

			rectangle "GitLab" {
				component "nemonik/gitlab:11.8.3"" as gitlab
        component "sameersbn/postgresql:10" as gitlab_postgresql
        component "redis:4.0.9-1" as gitlab_redis
			}

			rectangle "Drone CI" {
        component "drone/server:1" as drone_server
        component "drone/agent:1" as drone_agent
        component "sameersbn/postgresql:9.6-4" as drone_postgresql
			}

			rectangle "SonarQube" {
				component "sonarqube:7.1" as sonarqube
			}
        
			rectangle "PlantUML Server" {
				component "plantuml/plantuml-server:latest" as plantuml_server
			}

			rectangle "Registry" {
				component "registry:2.7.1" as registry
			}      
    }
	}

	rectangle "Development Vagrant" {
    rectangle "Docker CE" as docker2 {
      component "helloworld" as helloworld
      component "helloworld-web" as helloworld_web
    }
	}
}

taiga -[#0B5C92]-> taiga_postgresql

gitlab -[#0B5C92]-> gitlab_postgresql
gitlab -[#0B5C92]-> gitlab_redis

drone_server -[#0B5C92]-> drone_agent
drone_server -[#0B5C92]-> drone_postgresql

you .[#0B5C92].> sonarqube
you .[#0B5C92].> gitlab
you .[#0B5C92].> taiga
you .[#0B5C92].> plantuml_server
you .[#0B5C92].> drone_server
you .[#0B5C92].> registry

you .[#0B5C92].> helloworld
you .[#0B5C92].> helloworld_web
@enduml

9.1. Configuring environmental variables

If your environment makes use of an HTTP proxy or SSL inspection, you will need to configure environment variables for this class.

On Mac OS X or *NIX environments

The following set_env.sh BASH script is included in the root of the project and can be used to configure these variable for UNIX environments, but must be adjusted for your specific environment.

#!/usr/bin/env bash

# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>

# run in shell via
#
# ```
# . ./set_env.sh
# ```

export PROXY=http://gatekeeper.mitre.org:80
export proxy=$PROXY
export https_proxy=$PROXY
export http_proxy=$PROXY
export HTTP_PROXY=$PROXY
export ALL_PROXY=$PROXY
export NO_PROXY="127.0.0.1,localhost,.mitre.org,.local,$(echo 192.168.0.{1..255} | sed 's/ /,/g')"
#,172.30.1.1"
export no_proxy=${NO_PROXY}

export CA_CERTIFICATES=https://raw.githubusercontent.com/nemonik/hands-on-DevOps/master/certs/MITRE%20BA%20NPE%20CA-3(1).crt,https://raw.githubusercontent.com/nemonik/hands-on-DevOps/master/certs/MITRE%20BA%20ROOT.crt

export VAGRANT_ALLOW_PLUGIN_SOURCE_ERRORS=0

When in the root of the project, the script can be execute in terminal session via

. ./set_env.sh

If you have no HTTP proxy and no SSL inspection to be concerned about, the alternative is to execute unset.sh BASH script to unset all these values:

#!/usr/bin/env bash

# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>

# run in shell via
#
# ```
# . ./unset.sh
# ```

unset no_proxy
unset NO_PROXY
unset ALL_PROXY
unset PROXY
unset proxy
unset https_proxy
unset http_proxy
unset HTTP_PROXY
unset HTTPS_PROXY
unset ftp_proxy
unset FTP_PROXY
unset ca_certificates
unset CA_CERTIFICATES

Execute in terminal session via

. ./unset.sh

On Windows

Now if, if you are on Windows you cam perform the following to set the same environmental variable adjusting for your environment:

In the Windows taskbar, enter edit the system environment variables into Search Windows and select the icon with the corresponding name.
The Systems Property window will likely open in the background, so you will likely need to go find it and bring it forward.
In the Systems Property's Advanced tab select Environment Variables... button.
In Environment Variables windows that opens, under User variables for...press New ... to open a New User Variable window, enter each Variable Name and and its respective Value for each pair in the table below

Variable Name	Value
proxy	http://gatekeeper.mitre.org:80
http_proxy	http://gatekeeper.mitre.org:80
https_proxy	http://gatekeeper.mitre.org:80
no_proxy	127.0.0.1,localhost,.mitre.org,.local,192.168.0.10,192.168.0.11
CA_CERTIFICATES	http://employeeshare.mitre.org/m/mjwalsh/transfer/MITRE%20BA%20ROOT.crt,http://employeeshare.mitre.org/m/mjwalsh/transfer/MITRE%20BA%20NPE%20CA-3%281%29.crt
VAGRANT_ALLOW_PLUGIN_SOURCE_ERRORS	0

NOTE

The certificate URLs need to be encoded for parentheses to work.
On Windows, you may inadvertantly cut-and-paste blank space characters (e.g., tabs, spaces) and the subsequent Ansible automation may fail.
Make sure if you are using the above setting not to drop the port from the respective URLs (e.g.,http://gatekeeper.mitre.org:80)

9.2. VirtualBox

You will need to install VirtualBox, a general-purpose full virtualizer for x86 hardware.

9.2.1. Installing VirtualBox

For the class when I teach it, it is assumed VirtualBox is installed, but below are the instructions for installing it on Windows 10.

Open your browser to https://www.virtualbox.org/wiki/Downloads
Click Windows hosts link under VirtualBox 6.0.4 platform packages.
Find and click the installer to install.

Then turn for Windows 10 turn off Hyper-V

Click Windows Start and then type turn Windows features on or off into the search bar.
Select the icon with the corresponding name.
This will open the Windows Features page and then unselect the Hyper-V checkbox if it is enabled and then click Okay.

The same site has the Mac OS X download. The install is less involved.

9.3. Git Bash

Git Bash is git packaged for Windows with bash (a command-line shell) and a collection of other, separate *NIX utilities, such as, ssh, scp, cat, find and others compiled for Windows.

9.3.1. Installing Git Bash

If you are on Windows, you'll need to install git.

Download from https://git-scm.com/download/win
Click the installer.
Click next until you reach the Configuring the line ending conversions page select Checkout as, commit Unix-style line endings.
Then next, next, next...
Don't open git-bash from the final window as it will not have the environmental variables set. Go onto step-6.
On the Windows task bar, enter git into Search Windows then select Git Bash. Use Git Bash instead of Command or Powershell.

9.4. Retrieve the course material

If you are reading this on paper and have nothing else, you only have a small portion of the class material. You will need to download the class project containing all the automation to spin up a DevOps toolchain and development, etc.

In a shell, for the purposes of the class, this means in Git Bash, clone the project from https://github.com/nemonik/hands-on-DevOps.git via git like so:

git -c http.sslVerify=false clone https://github.com/nemonik/hands-on-DevOps.git

Output will resemble (i.e., not being precisely the same):

$ git -c http.sslVerify=false clone https://github.com/nemonik/hands-on-DevOps.git
Cloning into 'hands-on-DevOps'...
remote: Counting objects: 1184, done.
remote: Compressing objects: 100% (203/203), done.
remote: Total 1184 (delta 207), reused 411 (delta 178)
Receiving objects: 100% (1184/1184), 235.51 MiB | 21.53 MiB/s, done.
Resolving deltas: 100% (480/480), done.

9.5. Infrastructure as code (IaC)

This class uses Infrastructure as code (IaC) to set up the class environment (i.e., two virtual machines that will later be referred to as "vagrants".) IaC is the process of provisioning, and configuring (i.e., managing) computr systems through code, rather than directly manipulating the systems by hand (i.e., manual processes).

This class uses Vagrant and Ansible IaC frameworks and the following sections will unpack each.

9.5.1. Vagrant

This class uses Vagrant, a command line utility for managing the lifecycle of virtual machines as a vagrant in that the VMs are not meant to hang around in the same place for long.

Unless you want to pollute your machine with every imaginable programming language, framework and library version you'll find yourself often creating a virtual machine (VM) for each software project. Sometimes more than one. And if you're like me of the past you'll end up with a VirtualBox full of VMs. If you haven't gone about this the right way, you'll end up wondering what VM went with which project and now how did I create it? The anti-pattern around this problem is to write documentation. A better way that aligns with DevOps repeatable practices is to create automation to provision and configure your development VMs. This is where Vagrant comes in as it is "a command line utility for managing the lifecycle of virtual machines."

9.5.1.1. Vagrant documentation and source

Vagrant's documentation can be found at

https://www.vagrantup.com/docs/index.html

It's canonical (i.e., authoritative) source can be found at

https://github.com/hashicorp/vagrant/

Vagrant is written in Ruby. In fact, a Vagrantfile is written in a Ruby DSL.

9.5.1.2. Installing Vagrant

Download Vagrant

https://releases.hashicorp.com/vagrant/2.2.3/

As of April 2nd 2014, vagrant-ca-certificates plugin has an open issue (#34), where the plugin fails to run on the recent 2.2.4 release of Vagrant, but runs fine on prior 2.2.3 release. Please, install the 2.2.3 release of Vagrant.
Click on the installer once downloaded and follow along. The installer may stall calculating for a bit on Windows and for this same OS may bury modals you'll need to respond to in the Windows Task bar, so keep an eye out for that. The installer will automatically add the vagrant command to your system path so that it is available on the command line. If it is not found, the documentation advises to try logging out and logging back into your system. This is particularly necessary sometimes for Windows. Windows will require a reboot, so remember to come back and complete step-3.
If you're not on the MITRE corporate network skip this step. On Windows, use the File Explorer to replace the existing C:\Hashicorp\vagrant\embedded\cacert.pem file with the project's vagrant_files/cacert.pem by using the File Explorer.

Or when on Mac OS X copy it to /opt/vagrant/embedded as root using:
```
sudo cp vagrant_files/cacert.pem /opt/vagrant/embedded/.
```

9.5.1.3. Installing Vagrant plugins

Vagrant plugins extend the functionality of Vagrant, and you'll need a few of them for this course.

In the command line (in Git Bash, if on Windows) on the host

vagrant plugin install vagrant-ca-certificates
vagrant plugin install vagrant-cachier
vagrant plugin install vagrant-disksize
vagrant plugin install vagrant-proxyconf
vagrant plugin install vagrant-vbguest

Verify the version installed with

vagrant plugin list

Whose output resembles this

vagrant-ca-certificates (1.3.0, global)
vagrant-cachier (1.2.1, global)
vagrant-disksize (0.1.3, global)
vagrant-proxyconf (2.0.1, global)
vagrant-vbguest (0.17.2, global)

The versions should be these or newer. Note difference in the event there ar problems later, because these versions have been verified to work.

9.5.1.4. The Vagrantfile explained

The Vagrantfile found at the root of the project describes how to provision and configure one or more virtual machines.

Vagrant's own documentation puts it best:

Vagrant is meant to run with one Vagrantfile per project, and the Vagrantfile is supposed to be committed to version control. This allows other developers involved in the project to check out the code, run vagrant up, and be on their way. Vagrantfiles are portable across every platform Vagrant supports.

If we were instead provisioning production Virtual Machines we'd alternatively use Terraform, a tool for building, changing, and versioning infrastructure

The following sub sections enumerate the various sections of the Vagrantfile broken apart in order to discuss.

9.5.1.4.1. Modelines

# -*- mode: ruby -*-
# vi: set ft=ruby :

When authoring, tells your text editor (e.g. emacs or vim) to choose a specific editing mode for the Vagrantfile. Line one is a modeline for emacs and line two is a modeline for vim.

9.5.1.4.2. Setting Software version for Ansible roles

# Holds the version of software for the Ansible roles to install
software_versions = {
  drone_version: "1.2.1",
  drone_cli_version: "1.0.7",
  golang_version: "1.12.6",
  gitlab_version: "11.11.2",
  inspec_version: "3.9.0",
  inspec_rpm_version: "3.9.0/el/7/inspec-3.9.0-1.el7.x86_64",
  python_version: "2.7.16",
  registry_version: "2.7.1",
  plantuml_server_version: "latest",
  sonarqube_version: "7.1", # 7.6-community is different enough i will need to look into in order to patch
  sonar_scanner_cli_version: "3.3.0.1492",
  selenium_standalone_chrome_version: "3.14.0",
  selenium_standalone_firefox_version: "3.14.0",
  taiga_version: "4.0.4",
  zap2docker_stable_version: "2.7.0"
}

Contains all the version numbers for the software installed by the Ansible roles found in ansible/roles.

9.5.1.4.3. Inserting Proxy setting via host environmental variables

The proxy setting section above uses the vagrant-proxyconf plugin to set vagrants (i.e., managed VMs) to use specified proxies by using http_proxy and no_proxy environment variable.

Vagrant.configure('2') do |config|

  # Set proxy settings for all vagrants
  # 
  # Depends on install of vagrant-proxyconf plugin.
  #
  # To use:
  #
  # 1.  Install `vagrant plugin install vagrant-proxyconf`
  # 2.  Set environmental variables for `http_proxy`, `https_proxy`, `ftp_proxy`, and `no_proxy`
  # 
  #     For example:
  #
  #     ```
  #     export http_proxy=
  #     export https_proxy=
  #     export ftp_proxy=
  #     export no_proxy=
  #     ```  
  if (ENV['http_proxy'] || ENV['https_proxy'])
    if Vagrant.has_plugin?('vagrant-proxyconf')
      config.proxy.http = ENV['http_proxy']
      config.proxy.https = ENV['https_proxy']
      config.proxy.ftp = ENV['ftp_proxy']
      config.proxy.no_proxy = ENV['no_proxy']
      config.proxy.enabled = true
      puts "HTTP Proxy variables set. http_proxy = #{ config.proxy.http },  https_proxy = #{ config.proxy.https },  ftp_proxy = #{  config.proxy.ftp  }, no_proxy = #{ config.proxy.no_proxy }"
    else
      raise "Missing vagrant-proxyconf plugin.  Install via: vagrant plugin install vagrant-proxyconf"
    end
  else
    puts "No http_proxy or https_proxy environment variables are set."
    config.proxy.http = nil
    config.proxy.https = nil
    config.proxy.ftp = nil
    config.proxy.no_proxy = nil
    config.proxy.enabled = false
  end

9.5.1.4.4. Inserting enterprise CA certificates

  # To add Enterprise CA Certificates to all vagrants
  #
  # Depends on the install of the vagrant-ca-certificates plugin
  #
  # To use:
  #
  # 1.  Install `vagrant plugin install vagrant-ca-certificates`.
  # 2.  Set environement variable for `CA_CERTIFICATES` containing a comma separated list of certificate URLs.
  #
  #     For example:
  #
  #     ```
  #     export CA_CERTIFICATES=http://employeeshare.mitre.org/m/mjwalsh/transfer/MITRE%20BA%20ROOT.crt,http://employeeshare.mitre.org/m/mjwalsh/transfer/MITRE%20BA%20NPE%20CA-3%281%29.crt
  #     ```
  #
  #     The Root certificate *must* be denotes as the root certificat like so:
  #
  #     http://employeeshare.mitre.org/m/mjwalsh/transfer/MITRE%20BA%20ROOT.crt
  #
  if ENV['CA_CERTIFICATES']
    if Vagrant.has_plugin?('vagrant-ca-certificates')
      puts "CA Certificates set to #{ ENV['CA_CERTIFICATES'] }"
      config.ca_certificates.enabled = true
      config.ca_certificates.certs = ENV['CA_CERTIFICATES'].split(',')
    else
      raise "Missing vagrant-ca-certificates plugin.  Install via: vagrant plugin install vagrant-ca-certificates"
    end
  else
    puts "No CA_CERTIFICATES environment variable set."
    config.ca_certificates.certs = nil
    config.ca_certificates.enabled = false
  end

This section uses vagrant-ca-certificates to as the plugin's documentation puts it configures the virtual machine to inject the specified certificates into the guest's root bundle. This is useful, for example, if your enterprise network has a firewall (or appliance) which utilizes SSL interception. So, we aren't the only ones that have to deal with the havoc SSL interception brings to development.

9.5.1.4.5. Configuring the cache plugin to speed things along

  if Vagrant.has_plugin?("vagrant-cachier")
    # Configure cached packages to be shared between instances of the same base box.
    # More info on http://fgrehm.viewdocs.io/vagrant-cachier/usage
    config.cache.scope = :box
  else
    raise "Missing vagrant-cachier plugin.  Install via: vagrant plugin install vagrant-cachier"
  end

9.5.1.4.6. Configuring the disksize plugin to increase the disk size

  if !Vagrant.has_plugin?("vagrant-disksize")
    raise "Missing vagrant-disksize plugin.  Install via: vagrant plugin install vagrant-disksize"
  end

This section uses the vagrant-disksize plugin to to resize disks in VirtualBox.

9.5.1.4.7. Requiring vagrant-vbquest for Windows

  if Vagrant::Util::Platform.windows? and !Vagrant.has_plugin?('vagrant-vbguest') 
     raise "Missing vagrant-vbguest plugin.  Install via: vagrant plugin install vagrant-vbguest"
  end

vagrant-vbguest automatically installs the host's VirtualBox Guest Additions on the guest system. Not needed on Mac OS X, but needed on Windows.

9.5.1.4.8. Configuring the development vagrant

  ## Provision development vagrant
  config.vm.define "development", primary: true do |development|
    development.vm.box = "centos/7"
#    development.disksize.size = "80GB"
    development.vm.network "private_network", ip: "192.168.0.10"
    development.vm.network :forwarded_port, guest: 22, host: 2222, id: 'ssh'
    development.vm.hostname = "development"
    development.vm.synced_folder ".", "/vagrant", type: "virtualbox"
    development.vm.provider :virtualbox do |virtualbox|
      virtualbox.name = "DevOps Class - development"
      virtualbox.customize ["guestproperty", "set", :id, "/VirtualBox/GuestAdd/VBoxService/--timesync-set-threshold", 10]
      virtualbox.memory = 2048
      virtualbox.cpus = 2
      virtualbox.gui = false

      development_docker_disk = './development_docker.vdi'

      unless File.exist?(development_docker_disk)
        virtualbox.customize ['createmedium', '--filename', development_docker_disk, '--size', 40 * 1024]
      end

      # the value of storage_system_bus depends on your platform
      storage_system_bus = "IDE"

      # Provisions a drive for Docker storage
      virtualbox.customize ['storageattach', :id, '--storagectl', storage_system_bus, '--port', 1, '--device', 0, '--type', 'hdd', '--medium', development_docker_disk]
    end

    config.vm.provision "ansible_local" do |ansible|
      ansible.playbook = "ansible/development-playbook.yml"
      ansible.inventory_path = "hosts"
      ansible.limit = "development"
      ansible.install_mode = :default
      ansible.compatibility_mode = "2.0"
      ansible.verbose = true # true (equivalent to v), vvv, vvvv

      # Pass top-level variables into Ansible from Vagrant
      ansible.extra_vars = software_versions

      if development.proxy.enabled
        ansible.extra_vars[:http_proxy] = (!config.proxy.http ? "" : config.proxy.http)
        ansible.extra_vars[:https_proxy] = (!config.proxy.https ? "" : config.proxy.https)
        ansible.extra_vars[:ftp_proxy] = (!config.proxy.ftp ? "" : config.proxy.ftp)
        ansible.extra_vars[:no_proxy] = (!config.proxy.no_proxy ? "" : config.proxy.no_proxy)
      end

      if development.ca_certificates.enabled
        ansible.extra_vars[:CA_CERTIFICATES] = config.ca_certificates.certs
      end
    end
  end

This section provisions and configures a development vagrant used for development

The config.vm.provision block uses ansible_local to configure the development vagrant. I've written Ansible roles under ansible/roles to automate the configuration of the vagrants.
- development.vm.box loads the vagrant with this centos/7 vagrant base box. Vagrant curates a listing of base boxes here https://app.vagrantup.com/boxes/search
- development.vm.synced_folder mounts the class's project folder to /vagrant path in the vagrant.
- The development.vm.provider section tells the hypervisor how to configure the vagrant (i.e., how much memory, how many procesors)
ansible_local does not require one to have Ansible installed on the host. Vagrant handles installing it on the vagrant and executing the specified playbook.
- In this instance, I have ansible.verbose jacked to its highest setting providing loads of development logging. If too much information about what is going on unnerves you changing this to false will disable verbose logging.

9.5.1.4.9. Configuring the toolchain vagrant

  ## Provision the pipeline vagrant
  config.vm.define "toolchain", autostart: false do |toolchain|
    toolchain.vm.box = "centos/7"
#    toolchain.disksize.size = "80GB"
    toolchain.vm.network "private_network", ip: "192.168.0.11"
    toolchain.vm.network :forwarded_port, guest: 22, host: 2223, id: 'ssh'
    toolchain.vm.hostname = "toolchain"
    toolchain.vm.synced_folder ".", "/vagrant", type: "virtualbox"
    toolchain.vm.provider :virtualbox do |virtualbox|
      virtualbox.name = "DevOps Class - toolchain"
      virtualbox.customize ["guestproperty", "set", :id, "/VirtualBox/GuestAdd/VBoxService/--timesync-set-threshold", 10]
      virtualbox.memory = 6144 #4096
      virtualbox.cpus = 4
      virtualbox.gui = false

      toolchain_docker_disk = './toolchain_docker.vdi'

      unless File.exist?(toolchain_docker_disk)
        virtualbox.customize ['createmedium', '--filename', toolchain_docker_disk, '--size', 40 * 1024]
      end

      # the value of storage_system_bus depends on your platform
      storage_system_bus = "IDE"

      # Provisions a drive for Docker storage
      virtualbox.customize ['storageattach', :id, '--storagectl', storage_system_bus, '--port', 1, '--device', 0, '--type', 'hdd', '--medium', toolchain_docker_disk]
    end

    config.vm.provision "ansible_local" do |ansible|
      ansible.playbook = "ansible/toolchain-playbook.yml"
      ansible.inventory_path = "hosts"
      ansible.limit = "toolchain"
      ansible.install_mode = :default
      ansible.compatibility_mode = "2.0"
      ansible.verbose = 'vvvv' # true (equivalent to v), vvv, vvvv

      # Pass top-level variables into Ansible from Vagrant
      ansible.extra_vars = software_versions

      if toolchain.proxy.enabled
        ansible.extra_vars[:http_proxy] = (!config.proxy.http ? "" : config.proxy.http)
        ansible.extra_vars[:https_proxy] = (!config.proxy.https ? "" : config.proxy.https)
        ansible.extra_vars[:ftp_proxy] = (!config.proxy.ftp ? "" : config.proxy.ftp)
        ansible.extra_vars[:no_proxy] = (!config.proxy.no_proxy ? "" : config.proxy.no_proxy)
      end

      if toolchain.ca_certificates.enabled
        ansible.extra_vars[:CA_CERTIFICATES] = config.ca_certificates.certs
      end
    end
  end

This section provisions and configures the toolchain vagrant. This is the beefy vagrant running GitLab. Drone CI, the private Docker registry, SonarQube, Selenium, Taiga...

9.5.2. Ansible

Ansible is a "configuration management" tool that automates software provisioning, configuration management and application deployment, two core repeated practices in DevOps, so for the purposes of the class Ansible addresses this concern in configuring of two vagrants.

Ansible was open source and then subsumed by Red Hat.

There are other "configuration management" tools, such as Chef and Puppet. There are of course even more, but they hold little or no community practice or market share (e.g., BOSH, Salt.)

9.5.2.1. Inventory file

Since, Ansible will work against multiple systems in the infrastructure at the same time it does this via what it refers to as an inventory.

The project's inventory, hosts is located at the root of the project and contains:

controller  ansible_connection=local
development ansible_connection=local
toolchain   ansible_connection=local

[nodes]
development ansible_host=192.168.0.10
toolchain ansible_host=192.168.0.11

[developments]
development ansible_host=192.168.0.10

[toolchains]
toolchain ansible_host=192.168.0.11

# Used by Ansible roles to configure Docker DNS server settings
[dns]
# My home network DNS
#ns1 ansible_host=10.0.1.1

# MITRE DNS
#ns1 ansible_host=10.20.100.53
#ns2 ansible_host=10.20.200.53

# Google / Starbux DNS
#ns1 ansible_host=8.8.8.8
#ns2 ansible_host=8.8.4.4

# HAMP LAB DNS
#ns1 ansible_host=192.168.1.1

# OuterNET
ns1 ansible_host=192.52.194.138
ns2 ansible_host=198.49.146.138

The inventory collects all the vagrants under a [nodes] group, and then defines each vagrant under their respective group either [developments] or [toolchains].

NOTE

The inventory also defines a [dns] group. DNS (domain name service) hosts will vary between environment. Please configure, for your environment.

9.5.2.2. Playbooks

In Ansible one defines playbooks to manage configurations of and deployments to remote machines. The playbooks I'm using to configure the vagrants exist in ansible/.

The development vagrant Ansible playbook (ansible/development-playbook.yml) contains:

---
# Development Ansible playbook

# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>

- hosts: [developments]
  remote_user: vagrant
  roles:
    - common
    - docker
    - docker-compose
    - golang
    - golint
    - inspec
    - drone-cli
    - sonar-scanner-cli
#    - openshift-cli

The file is written in a YAML-based DSL (domain specific language.)

9.5.2.3. Roles

The roles exist in ansible/roles and permit the sharing of bits of configuration content with other users. Roles can also be found in the Ansible Galaxy, retrieved and placed into the ansible/roles folder to be used, but I wrote all the roles for the class.

A role, such as a Taiga role is comprised of many components (e.g., files, templates, tasks) with the core being the main task. For example, the main task for the taiga roles contains

---
# tasks file for taiga

# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>

- name: "copy to taiga files to /home/{{ ansible_user_id }}/taiga"
  become: yes
  copy:
    src: files/
    dest: "/home/{{ ansible_user_id }}/taiga"
    owner: "{{ ansible_user_id }}"
    group: "{{ ansible_user_id }}"

- name: "template files into /home/{{ ansible_user_id }}/taiga"
  become: yes
  template:
    src: "{{ item.src }}"
    dest: "{{ item.dest }}"
    owner: "{{ ansible_user_id }}"
    group: "{{ ansible_user_id }}"   
  with_items:
    - { src: "templates/README.j2", dest: "/home/{{ ansible_user_id }}/taiga/README.MD" }
    - { src: "templates/Dockerfile.j2", dest: "/home/{{ ansible_user_id }}/taiga/Dockerfile" }
    - { src: "templates/dockerfile_build.j2", dest: "/home/{{ ansible_user_id }}/taiga/dockerfile_build.sh" }
    - { src: "templates/docker-compose.j2", dest: "/home/{{ ansible_user_id }}/taiga/docker-compose.yml" }

- name: "make /home/{{ ansible_user_id }}/taiga/dockerfile_build.sh executable"
  become: yes
  file: 
    path: "/home/{{ ansible_user_id }}/taiga/dockerfile_build.sh"
    mode: "u=rwx,g=r,o=r"  
    owner: "{{ ansible_user_id }}"
    group: "{{ ansible_user_id }}"

- name: ensure user 'default' with a 1001 uid and a primary group of 'ROOT' exists
  become: yes
  user:
    name: default
    uid: 1001
    group: root

- name: ensure ./volumes exist
  become: yes
  file:
    path: "{{ item.path }}"
    state: directory
    owner: "{{ item.owner }}"
    group: "{{ item.group }}"    
  with_items:
    - { path: "/home/{{ ansible_user_id }}/taiga/volumes/static/" , owner: root, group: root }
    - { path: "/home/{{ ansible_user_id }}/taiga/volumes/static/admin", owner: default, group: root }
    - { path: "/home/{{ ansible_user_id }}/taiga/volumes/media/", owner: root, group: root }    
    - { path: "/home/{{ ansible_user_id }}/taiga/volumes/media/user", owner: default, group: root }

- name: spin up taiga via docker-compose
  become: yes
  docker_service:
    build: yes
    debug: yes
    project_src: "/home/{{ ansible_user_id }}/taiga"

The Taiga role dependes on docker-compose, docker, and common having configured the vagrant.

9.6. Docker image and containers

So, what is Docker? What are Containers?

Whats a container?

Containers are a form of lightweight virtualization first existing in 1979’s Version 7 UNIX operating system when the chroot command was developed.
Since chroot, containers have continued to evolve within the Linux kernel to decouple applications from the operating system and run them in an isolated manner.
A container is essentially an isolated processes running in user space.
The benefit over Virtual Machines in that multiple containers can run on the same machine (in case of this class, a vagrant) sharing the OS kernel with other containers. Whereas a VM would require a full copy of an operating system in order to run the application. This makes containers insanely light-weight.
Container support has thrived and seen popular adoption across various operating systems outside of the Linux including Windows Containers and Window’s direct ability to run Linux containers from the Hyper-V isolation work done by Microsoft.
The Open Container Initiative (OCI), formed in 2015 maintains open industry standards for containers that focus on Runtime Specification (runtime-spec) and its partner project, the Image Specification (image-spec).

What are container images?

A container image is a lightweight, self-contained, executable package that includes everything needed to run your application including runtime, system tools, system libraries, and settings.

9.7. Docker-compose

And what is docker-compose?

Doceker-compose is a tool and domain specific language based on YAML used to define and run multi-container Docker applications.

What is YAML? YAML bills itself as a human-friendly data serialization standard for all programming languages. YAML also follows in in the computing tradition of being a recursive acronym, YAML Ain't Markup Language. Many of the tools used in this course make use of YAML, so you will see plenty examples of it.

You will edit a docker-compose.yml file in this class in order to complete the configuration of one the key CI/CD tools.

9.8. Spinning up the toolchain vagrant

In the command line of the host in the root of the class project, open ansible/tool-chain-playbook.yml and make sure the roles are like so:

---
# Toolchain Ansible playbook

# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>

- hosts: toolchains
  remote_user: vagrant
  roles:
    - common
    - docker
    - docker-compose 
    - golang
    - golint
    - docker-registry

    # CI/CD Tools
    - taiga
    - gitlab
    - plantuml-server
    - drone
    - drone-cli
    - sonarqube
    - golang-container-image
    - inspec
    - python-container-image
    - golang-sonarqube-scanner-image
    - standalone-firefox-container-image
    - owasp-zap2docker-stable-image

#    - inspec-container-image
#    - standalone-chrome-container-image
#    - gnome-desktop
#    - firefox
#    - chrome
#    - openshift

Then enter into the command-line of the host at the root of the class project and enter into shell (I'll drop from time to time stating "into the shell" as it should understood.)

vagrant up toolchain

You will see a good deal of output and on the Windows OS, it will pester you to approve certain things. As a trust exercise blindly approve everything.

Once complete, open a secure shell (ssh) to the toolchain vagrant

vagrant ssh toolchain

The command line will open a prompt a bash shell on the vagrant

[vagrant@toolchain ~]$

On this vagrant will be running a number of DevOps tools. This will take a while, so let's discuss what is being installed.
TODO: Insert the tail end of succesful output here.

NOTE

It is very possible a network anomaly may result in Ansible failing, if you can determine the role the automation failed in, you could try re-running the role the failure occured in and the subsequent roles like so:
```
ANSIBLE_ARGS='--tags "python-container-image,golang-sonarqube-scanner-image, standalone-firefox-container-image,owasp-zap2docker-stable-image"' vagrant provision toolchain
```
Otherwise, re-run the automation from the start:
```
vagrant up toolchain --provision
```
Caution, it is easy to forget DevOps is as much about culture as it is about a methodology and repeated practices (often further mistakenly thought as "tools and automation"), so keep this in mind.
The tools, methodology and repeated practices exist to support the culture.
Again, I'll drop from time to time stating "into the shell" when instruction you to enter things in the CLI as it should understood.

9.8.1. Taiga, an example of Agile project management software

Taiga is an Open Source project management platform for agile development.

There are many project management platforms for Agile.

Typically, Agile teams work using a visual task management tool such as a project board, task board or Kanban or Scrum visual management board. These boards can be implemented using a whiteboard or open space on a wall or in software. The board is at a minimum segmented into a few columns To do, In process, and Done, but the board can be tailored. I've personally seen boards for very large projects consume every bit of wallspace of a very large cavernous room, but as Lean-Agile has matured, teams have grown larger and more disparate, tools have emerged to provide a clear view into a project's management to all levels of concern (e.g., developers, managers, product owner, and the customer) answering:

Are deadlines being achieved?
Are team members overloaded?
How much is complete?
What's next?

Further, the Lean-Agile Software tools should provide the following capabilities:

Dividing integration and development effort into multiple projects.
Defining, allocating, and viewing resources and their workload across each product.
Defining, maintaining, and prioritizing the accumulation of an individual product's requirements, features or technical tasks which, at a given moment, are known to be necessary and sufficient to complete a project's release.
Facilitating the selection and assignment of individual requirements to resources, and the tracking of progress for a release.
Permit collaboration with external third parties.

The 800lb Gorilla in this market segment is JIRA Software. Some of my co-workers hate it. It is part of the Atlassian suite providing provides collaboration software for teams with products including JIRA Software, Confluence, Bitbucket, and Stash.

NOTE

Lean-Agile Project Management software's primary purpose is to integrare people and really not much else.

9.8.1.1. Documentation, source, container image

Taiga's documentation can be found at

It's canonical source can be found at

https://github.com/taigaio/taiga-front-dist/

dedicated to the front-end, and

https://github.com/taigaio/taiga-back/

dedicated to the back-end.

Taiga doesn't directly offer a Docker container image for, but I've authored an image that collapses both taiga-front-dist and -back onto one container behind an NGINX reverse proxy.

9.8.1.2. URL, Username and password

Once, stood up your instance of Taiga will reachable at

http://192.168.0.11:8080/

The default admin accont username and password are

admin
123123

9.8.2. GitLab CE, an example of configuration management software

GitLab is installed on the toolchain vagrant, where it will be accessible at http://192.168.0.11:10080/.

GitLab Community Edition (CE) is an open source end-to-end software development platform with built-in version control, issue tracking, code review, and CI/CD.

For Agile teams to collaborate, a Configuration management (CM) is necessary to coordinate the development of new feature, changes, and experimentation. Also, a CM system (CMS) provides a history of changes, and thereby, the ability to roll back to a version known to be acceptable.

At a minimum, the following items will be placed under revision control in CM:

Source code,
If a database is needed, schema initialization and the migration between versions,
Text documentation containing
- a synopsis (i.e., project name, overview, etc.),
- version description,
- guidance covering
  - build,
  - unit testing, and
  - installation
- a contributor enumeration,
- license and/or ownership declaration with contacts, etc.,

A single CMS and the associated workflow (e.g., GitHub Workflow) can serve as the focal point for the entire enterprise thereby provide centralized version control if all documentation is authored in a lightweight markup language with plain text formatting syntax (e.g., Markdown, PlantUML).

A CMS must facilitate best practices, not limited to:

A means for developers to copy and work off a complete repository thereby permitting
- Private individual work to later be synchronized via exchanging sets of changes (i.e., patches) through a means described as "distributed version control", and
- Pre-flight build and test of their source code in their own private workspace, so as to minimize the chance of committing broken or untested source code thereby encouraging
- The committing of completed source code only.
Granular commits that communicate the motivation for the commit (i.e., the what and why). For example, for a change these could be:
- the inclusion of a new feature,
- a bug fix,
- the removal of dead code
Reducing the risk breaking a build by
- Utilizing branching to separate different lines of development, and
- Standardizing on CMS workflows (e.g., GitHub Workflow),
Make builds be self-testing (i.e., ingrain testing) by including unit and integration test with the source code so that it can be executed by
- the build automation, and
- the Continuous Integration service.
Trigger follow-on activities orchestrated by the Continuous Integration Service.

9.8.2.1. Documentation, source, container image

GitLab's documentation can be found at

https://docs.gitlab.com/ce/

It's canonical source can be found at

https://hub.docker.com/r/sameersbn/gitlab/

https://gitlab.com/gitlab-org/gitlab-ce

I'm using Sameer Naik's Docker container image for GitLab. The image can be found at

https://hub.docker.com/r/sameersbn/gitlab/

And its canonical source is located at

https://github.com/sameersbn/docker-gitlab

9.8.2.2. URL, Username and password

Once, stood up your instance of GitLab will reachable at

http://192.168.0.11:10080/

GitLab requires you to enter a password for its root account. You will be using the root account to host your repositories vice creating your own, but if you want you can. There is nothing stopping you.

I would suggest for the purposes of the class choosing something simple, but at least the same number of characters as "password".

9.8.2.3. Allow requests to the local network from hooks and services

Perform the following steps as the GitLab adminstrator:

Log into GitLab as root with the password you entered in the prior step
Click the wrench at the top of the page to enter the Admin area
In the side bar running the length of the left-side of the page, click the gear-icon, select Network, and expand Outbound requests
Under Outbound requests, check Allow requests to the local network from hooks and services and then click Save Changes

9.8.2.4. Do not default to Auto DevOps pipeline for all projects

Log into GitLab as root with the password you entered in the prior step
Click the wrench at the top of the page to enter the Admin area
In the side bar running the length of the left-side of the page, click the gear-icon, select CI/CD, and expand Continuous Integration and Deployment
Uncheck Default to Auto DevOps pipeline for all projects and then click Save Changes

9.8.3. Drone CI, an example of CI/CD orchestrator

Drone CI often referred to simply as "Drone" is installed on the toolchain vagrant, where it will be accessible at http://192.168.0.11/ after some user configuration that will be explained later in the Integrate Drone CI with GitLab section.

Drone is essentially a Continuous Delivery system built on container technology.

Drone is distributed as a Docker image. Drone CI can be run with an internal SQLite database, but it is advisable to run with an external database and this the configuration the class uses. It also integrates with multiple version control providers (i.e., GitHub, GitLab, BitBucket, Stash, and Gogs). Both CMS and database are configured using environment variables passed when the Drone CI container is first to run. Drone plugins (really just containers) can be used to deploy code, publish artifacts, send a notification, etc. Drone's approach to plugins is novel as plugins are really just Docker containers distributed in a typical manner. Each plugin is designed to perform pre-defined tasks and is configured as steps in your pipeline. Plugins are executed with read/write/execution access at the root of the source branch, therefore, permitting the pipeline to interact with the specific branch of source to build, test, bundle, deliver, and deploy. Developers create a Drone CI pipeline by placing a .drone.yml file in the root of the repository. The .drone.yml is authored in a domain specific language (DSL) that is a superset of the docker-compose DSL to describe the build with multiple named steps executed in a separate Docker container having shared disk access to the specific branch of the source repository.

Drone and its brethren (e.g., Jenkins CI, GitLab CI/CD) is used to facilitate Continuous Integration (CI), a software development practice where members of an Agile team frequently integrate their work in order to detect integration issues as soon as possible. Each integration is orchestrated through a service that essentially assembles a build and runs tests every time a predetermined trigger has been met; and then reports with immediate feedback.

I don't use Jenkins unless I have to. Why? I'm simply not a fan. Initially, because its plugin architecture is painful to manage and with prior version your pipelines existed entirely in the Jenkins CI tool itself. Later, Jenkins CI introduced Groovy-based Jenkin Pipelines that are CMed with your project's source. Every orchestrator has based their DSL on YAML and although I love the Groovy language for its power, I don't find it makes for a good orchestration language. Your opinion may differ. I'm okay with that. Really. I am.

There are also SaaS CI/CD tools, such as Travis CI and Circle CI. These are great, free CI/CD orchestrators.

My java-stix project hosted on GitHub.com uses both Travis CI and Circle CI as part of its continuous integration.

The Travis CI orchestration contains

language: java
dist: precise
jdk:
  - oraclejdk7

before_install:
  - chmod +x gradlew

env: GRADLE_OPTS=-Dorg.gradle.daemon=true
env: CI_OPTS=--stacktrace

install: /bin/true
script: "./gradlew -x signArchives"

Whereas, the Circle CI orchestration contains

machine:
  java:
    version: oraclejdk7
  environment:
    GRADLE_OPTS: -Dorg.gradle.daemon=true
    CI_OPTS: --stacktrace --debug

test:
  override:
    - ./gradlew -x signArchives

They are essentially similar with both requiring the Oracle JDK and use Gradle to build and unit test the code. Both of these services under the covers use containers to run the builds.

In another GitHub hosted project, the java-stix-validator the Travis CI orchestration contains

language: java
jdk:
- oraclejdk8
before_install:
- chmod +x gradlew
env: CI_OPTS=--stacktrace
install: "/bin/true"
script: "./gradlew build -d"
deploy:
  provider: heroku
  api_key:
    secure: m9Gbt0Oyqtjwyu4Y8CVobNNnj1q5mFt+Ygi2wiDWlf/RunLOj2CE8YAYuRyEAbpCOd1lrmrhmQb8uQAfiydauYBcQE5yyOlyIhNkrLi2m1+we0VeWWr6gxIVz57VuAhfbzoMtvkhmxl/Ey0U+1vI7tYurK0thzFUyQFqZh1wNq6EldIfHxVNxDZbEVtkDzFtK5cmVnPE8HM9xaQmuV7k3NhrvRS4pzN87uvndfFVb0vDhLmg5DulF+PLkdpP9UC5jsAE1HXMBL0cTtsvSHUkIyO7qhLb0RFAzVdRMvn7kEW2Q0ekoK09sPR13VwmfjewzHSNrWIf+rjJx7EzoBzbq5/VmC9nxH1oiGpXxoAG08pJjcQYMSxsa2JZLH8dSIEaMgOFNOxkrAhcqP59xXWZ9WVLCYPSN4atmg4L6etJOzFqfz3jAp40AB4Eu2QU49c60r6BH31Xj8ymjKMKqnlL199qCoqfZtv7FYqOFG3keLeWvL/F7JhmtV+JdvuqVPEvNq2D1b3kdCKk2cw4lmRCwC9hdT2oXTCwhjQvYwSm0sHQ98aeV55FkE7DuH4B+CuzYw4N9K78j3eQtW2Oas1lLCoHSDXgA/4O79RlM8p0nLa3MjdVq5OSIjbcCqhDLBe8nc5ucSpMMjnjNvhAKvcyrc5AbXdIVaLVvE2azMuZJLo=
  app: agile-journey-9583
  on:
    repo: STIXProject/java-stix-validator

To deploy the web application to Heroku, one of the first PaaS (Platforms-as-Service). The code was last committed in 2015 and the code is still running free on Heroku at http://agile-journey-9583.herokuapp.com/#/

More details on Drone is sprinkled across the class. As you can see I favor being a polyglot when it comes to software development to include CI/CD.

9.8.3.1. Documentation, source, container image

Drone's main site is at

https://drone.io/

Its documentation is at

http://docs.drone.io/

Its plugin market is at

http://plugins.drone.io/

Drone's canonical source can be found at

https://github.com/drone/drone

Drone is distributed as a container image and can be found respectfully at

https://hub.docker.com/r/drone/drone/

and

https://hub.docker.com/r/drone/agent/

9.8.3.2. URL, Username and password

Once, stood up your instance of Drone CI will reachable at

http://192.168.0.11/

Drone will authenticate you off GitLab once integrated.

9.8.4. Once the toolchain vagrant is up

Perform the follwing tasks.

9.8.4.1. Integrate GitLab with the PlantUML-Server

This README.MD has embedded PlantUML diagrams for you visual learners without the integration the diagrams will remain in the source code.

You'll need to enable the integration between GitLab and the PlantUML-Server by accomplishing the following:

As root log into GitLab with the password you entered in the prior step,
Click on the wrench to enter the Admin Area,
Then click Settings or the gear-icon and then select Integrations.
Expand PlantUML and then check Enable PlantUML checkbox, set the PlantUML URL to http://192.168.0.11:8081
Then click Save changes.

9.8.4.2. Integrate Drone CI with GitLab

The Ansible automation will install patched Docker containers for Drone and template the docker-compose file used to spin it up, but the automation will not spin up Drone as it requires human intervention in the absence of automation. I was too lazy to write this automation.

So, you will need to accomplish the following steps, while at the same time becoming familiar with GitLab.

As root log into GitLab with the password you entered in the prior step,
Click on the wrench to enter the Admin Area,
Click Applications,
Click New Application,
Enter Drone CI for Name,
Enter http://192.168.0.11/login,
Check off api and read_user,
Click submit,
Open a secure shell to the toolchain vagrant at the root of the project with

vagrant ssh toolchain

The command line will open a prompt to the vagrant

[vagrant@toolchain ~]$

Enter the drone directory with

cd ~/drone

Open docker-compose.yml in a text editor.

            - DRONE_GITLAB_CLIENT=Your Application Id
            - DRONE_GITLAB_SECRET=Your Secret

Around line 50, replace Your Application Id and Your Secret with Application Id and Secret provided by GitLab, respectively. Save your changes and exit the editor.

Then enter into the command line

docker-compose up -d

Command line output will resemble

[vagrant@toolchain drone]$ docker-compose up -d
drone_drone-postgresql_1 is up-to-date
Recreating drone_drone-server_1 ... done
Recreating drone_drone-agent_1  ... done

You can drop -d on the docker-compose so as to not daemonize the drone, so you can monitor its logging for issues.

Exit the root session.

exit

Open http://192.168.0.11 in the browser and click Authorize to permit Drone to use your GitLab account.

You will only have to do these step in regards to Drone once. Drone should always restart in the toolchain vagrant even if you halt the vagrant and later bring it up.

9.8.4.3. Optionally, add the hands-on-DevOps repository to the `toolchain`'s GitLab

In GitLab (http://192.168.0.11:10080/) click on Projects in the upper left. a. Select Create Project.
b. Or click http://192.168.0.11:10080/projects/new
Leave the Project path defaulted to http://192.168.0.11:10080/root/.
Enter hands-on-DevOps (the form field the page defaults to) for the Project name.
Provide an optional Project description. Something descriptive, such as, "An awesome DevOps class".
Make the application public to save yourself from entering your username and password when cloning.
Click the green Create Project button on the lower left.

The UI will refresh to show you a landing page for the project that should be accessible from http://192.168.0.11:10080/root/hands-on-DevOps

On your host in the DevOps class project:

Enter into the following into the command line

git config --global user.name "Administrator"
git config --global user.email "[email protected]"

Then enter the following

git remote add toolchain http://192.168.0.11:10080/root/hands-on-DevOps.git
git push -u toolchain --all
git push -u toolchain --tags

You now have a clone of the project hosted in the GitLab running on the toolchainvagrant. This way you can open the README.MD and follow along with rendered PlanUML diagrams.

9.8.5. SonarQube, an example of a platform for the inspection of code quality

SonarQube provides the capability to show the health of an application's source code, highlighting issues as they are introduced. SonarQube can be extended by language-specific extensions/plugins to report on duplicated code, coding standards, unit tests, code coverage, code complexity, comments, bugs, and security vulnerabilities.

9.8.5.1. Documentation, source, container image

SonarQube's main site is at

https://www.sonarqube.org/

Its documentation is at

https://docs.sonarqube.org/display/SONAR/Documentation

SonarQube's canonical source can be found here

https://github.com/SonarSource/sonarqube

I'm using the container image provided at

https://hub.docker.com/_/sonarqube/

9.8.5.2. URL, Username and password

Once, stood up your instance of SonarQube will reachable at

http://192.168.0.11:9000/

The default admin account username and password is

admin

9.9. Spin up the development vagrant

In another command line of the host in the root of the class project

Open ansible/development-playbook.yml and make sure these roles are uncommented:

---
# Development Ansible playbook

# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>

- hosts: [developments]
  remote_user: vagrant
  roles:
    - common
    - docker
    - docker-compose 
    - golang
    - golint
    - drone-cli
    - sonar-scanner-cli
#    - openshift-cli

Then in the command-line of the host (not while ssh'ed into the toolchain vagrant) at the root of the class project type

vagrant up development

You will see a good deal of output.

TODO: Insert the tail end of succesful output here.

Once complete open a secure shell to the development vagrant

vagrant ssh development

Command line will open a prompt to the vagrant

[vagrant@development ~]$

NOTE

Again, it is very possible a network anomaly may result in Ansible failing, if you can determine the role automation failed in you could try re-running the role the failure occured in and the subsequent roles like so:
```
ANSIBLE_ARGS='--tags "inspec,drone-cli,sonar-scanner-cli"' vagrant provision development
```
Otherwise, re-run the automation from the start:
```
vagrant up development --provision
```

9.10. Golang helloworld project

The prior toolchain and development vagrants are required to be up and running for the following sections.

In this next part, we will create a simple helloworld GoLang project to demonstrate Continuous Integration. GoLang lends itself well to DevOps and underlines almost every new tool you can think of related to DevOps and cloud (e.g., golang / go, docker / docker-ce, kubernetes / kubernetes, openshift / origin ,hashicorp / terraform, coreos / etcd, hashicorp / vault, hashicorp / packer, hashicorp / consul, gogits / gogs, drone / drone.)

9.10.1. Create the project's backlog

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->“Create\nthe project’s\nbacklog” #FFFFFF
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based Continuous Integration”
-left-> (*)

A backlog is essentially your to-do list, a prioritized list of work derived from the roadmap (e.g., the outline for future product functionality and when new features will be released) and its requirements.

Open Taiga in your web browser

http://192.168.0.11:8080

The default admin account username and password are

admin
123123

Complete the follow to track your progress in completing the Golang helloworld project

Click Create Project.
Select Kanban. In a Kanban board work moves from left to right with each column represents a stage within the value stream.
Give your project a name. For example, Helloworld and a description, such as, My Kanban board for this awesome helloworld app and then click CREATE PROJECT.
You can skip this step and opt to chose to click >< to fold READY, USER STORY STATUS and ARCHIVED only after completing step 6. Otherwise, you can edit your Kanban board to just NEW, IN PROGRESS, and DONE by a. On the bottom left, click the ADMIN gear.
b. Click ATTRIBUTES.
c. Scroll down to USER STORY STATUS.
d. Hover over Ready, click the trash icon to delete and click ACCEPT.
c. Do the same for Ready for test and Archived.
d. Click the KANBAN icon on the far left. It Looks like columns. And then reload the browser to get the changes to take.

In the NEW column select Add New bulk icon that looks like a list and when the page updates cut-and-paste the lines below into the text box and click SAVE.

Create the project’s backlog
Create the project in GitLab
Setup your project on the development Vagrant
Author the application
Align source code with Go coding standards
Lint the code
Build the application
Run your application
Author the unit tests
Write the Makefile
Author Drone-based Continuous Integration

Track your progress in Taiga as you work through each section.

9.10.2. Create the project in GitLab

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab” #FFFFFF
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)

In GitLab (http://192.168.0.11:10080/) click on Projects in the upper left. a. Select Create Project.
b. Or click http://192.168.0.11:10080/projects/new
Leave the Project path defaulted to http://192.168.0.11:10080/root/.
Enter helloworld (the form field the page defaults to) for the Project name.
Provide an optional Project description. Something descriptive, such as, "GoLang helloworld application for the hands-on DevOps class.".
Make the application public to save yourself from entering your username and password when cloning.
Click the green Create Project button on the lower left.

The UI will refresh to show you a landing page for the project that should be accessible from http://192.168.0.11:10080/root/helloworld

NOTE

Sometimes GitLab with release 11.5.1 will throw an error on the step 6 above. Simply click the green Create Project button again.

9.10.3. Setup the project on the development Vagrant

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant” #FFFFFF
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)

On your host open a shell to development vagrant, and configure your user name and email:

git config --global user.name "Administrator"
git config --global user.email "[email protected]"

The Ansible golang role (ansible/roles/golang) in the class project on your host will already have configured:

Your $GOPATH and $GOBIN environmental variable.
Added $GOBIN to your PATH environmental variable.
Created go workspace in vagrant user's home directory (i.e., /home/vagrant/go directory containing bin, pkg, and src.)

Go source code is placed in the src directory under a namespace (i.e., a unique base path to avoid naming collisions under which all your go code will reside.) In open source software development, it is typical to use github.com account path. Mine is github.com/nemonik, so I would create this base path via (You do the same., so we are all on the same page.)

mkdir -p $GOPATH/src/github.com/nemonik

Now lets create the GoLang helloworld project to demonstrate Continuous Integration via

cd $GOPATH/src/github.com/nemonik
git clone http://192.168.0.11:10080/root/helloworld.git
cd helloworld

NOTE

Ignore the warning: You appear to have cloned an empty repository. warning. This is perfectly normal. Then move into the clone of your repository via

So that you do not commit certian files to GitLab when you push, create a .gitignore file with your editor with the following contents

# OS-specific
.DS_Store

.scannerwork
*.out
*.json

helloworld

NOTE

Make sure you pre-pend that dot (.) at the start of .gitignore. In *NIX Dot-files are hidden files.
.gitignore will not show up if you simply list the file system via the ls command, but they do if you use ls -a or ls --all. Either arguments configures ls to not ignore entries starting with ..

9.10.4. Author the application

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application” #FFFFFF
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)

In the project follder (i.e., /home/vagrant/go/src/github.com/nemonik/helloworld), create main.go in emacs, nano, vi, or vim with this content:

package main

import "fmt"

func main() {
  fmt.Println(HelloWorld())
}

func HelloWorld() string {
  return "hello world"
}

9.10.5. Align source code with Go coding standards

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards” #FFFFFF
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)

Format source code according to Go coding standards using

go fmt

Results in the code being formatted to

package main

import "fmt"

func main() {
    fmt.Println(HelloWorld())
}

func HelloWorld() string {
    return "hello world"
}

You'll see the difference if you cat your source

cat main.go

9.10.6. Lint your code

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code” #FFFFFF
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)

Already installed on your development vagrant is golint. Where go fmt reformatted the code to GoLang standards, golint prints style mistakes.

To run golint, in the root of the helloworld project execute

golint

Command line out will be

[vagrant@development helloworld]$ golint
hello.go:9:1: exported function HelloWorld should have comment or be unexported

Fix the error by editing main.go to

package main

import "fmt"

func main() {
	fmt.Println(HelloWorld())
}

// HelloWorld returns "hello world"
func HelloWorld() string {
	return "hello world"
}

Run golint again and it should return no output indicating it sees nothing wrong.

9.10.7. Build the application

Build the project by executing

go build -o helloworld .

Success returns no command line output. What? Did you want a cookie? No cookie for you. This is GoLangs way of doing things. Silence is golden and means things went fine. Otherwise, go back and fix the mistakes in your code.

9.10.8. Run your application

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application” #FFFFFF
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)

Running your application

./helloworld

The command line output will be

[vagrant@development hello]$ ./helloworld
hello world

9.10.9. Author the unit tests

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests” #FFFFFF
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)

GoLang ships with a built-in testing package

https://golang.org/pkg/testing/

for automated unit testing of Go packages. Unit testing is a software development process where the smallest testable components of an application are individually tested for proper operation. Unit testing offers the biggest return for dollars spent in comparison to integration and functional testing.

For more on this topic read Martin Fowler's

https://martinfowler.com/bliki/TestPyramid.html

In your editor create main_test.go with this content:

package main

import (
  "os"
  "testing"
)

func TestMain(m *testing.M) {
  os.Exit(m.Run())
}

func TestHelloWorld(t *testing.T) {
  if HelloWorld() != "hello world" {
    t.Errorf("got %s expected %s", HelloWorld(), "hello world")
  }
}

Execute the unit test by entering

go test -v -cover

The command line returns

=== RUN   TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
coverage: 50.0% of statements
ok  	github.com/nemonik/helloworld	0.002s

This step and all the proceeding follows of a DevOps tenant of Devops, "Developers are expected to pre-flight new code."

9.10.10. Automated the build (i.e., write the Makefile)

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile” #FFFFFF
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)

Build automation is a key practice of CI. So, let's make the build reproducible by automating everything we've done this far via authoring a Makefile.

In the root of the project create Makefile and add the following contents

GOCMD=go
GOFMT=$(GOCMD) fmt
GOBUILD=$(GOCMD) build
GOCLEAN=$(GOCMD) clean
GOTEST=$(GOCMD) test
GOLINT=$(GOCMD)lint
GOGET=$(GOCMD) get
BINARY_NAME=helloworld

all: build
clean:
	$(GOCLEAN)
	rm -f $(BINARY_NAME)
fmt:
	$(GOFMT)
lint: fmt
	$(GOGET) golang.org/x/lint/golint
	golint
test: lint
	$(GOTEST) -v -cover ./...
build: test
	$(GOBUILD) -o $(BINARY_NAME) -v
run:
	$(GOBUILD) -o $(BINARY_NAME) -v ./...
	./$(BINARY_NAME)

NOTE

Each line indentation is a tab and not a series of space characters. Make will fail to execute if these tabs are converted to a series of space characters.

Save the file and exit your editor.

Okay, let's try it out in the development terminal

make all

The output will resemble

go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN   TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
coverage: 50.0% of statements
ok  	github.com/nemonik/helloworld	(cached)	coverage: 50.0% of statements
go build -o helloworld -v

9.10.11. Author Drone-based Continuous Integration

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration” #FFFFFF
-left-> (*)

CI integrates all of the steps we have worked to ensure a high quality build into a pipeline, so let's do that.

We're going to author a continuous integration pipeline for our application and execute it on Drone. Drone expects a .drone.yml to exist at the root of the project and will execute the pipeline it contains when the project is committed to GitLab.

A pipeline is broken up into multiple named steps, where each step executes in an ephemeral (i.e., does its job and then poof it is gone) Docker container with shared disk access to the project's workspace. The benefit of this approach is that it relieves you from having to create and maintain slaves to execute your pipelines.

Drone automatically clones your project's repo (Short for "repository.") into a volume (referred to as the workspace) shared by each Docker container (including plugins service containers).

In your editor create .drone.yml file at the root of the helloworld project:

kind: pipeline
name: default

steps:
- name: build
  image: 192.168.0.11:5000/nemonik/golang:1.12.6
  commands:
  - make build

- name: run
  image: 192.168.0.11:5000/nemonik/golang:1.12.6
  commands:
  - make run

NOTE

Make sure you pre-pend that dot (.) at the start of .drone.yml.
Make sure to leave two consecutive returns to indicate the end of the .drone.yml file or a parse error when executed in Drone will result.

The pipeline is authored in YAML like almost all the CI orchestrators out there except for Jenkin's Pipelines, whose you author in Groovy-based DSL.

steps: - defines a list of steps to build, test and deploy your code.
build and run - are the names of the step. These are yours to name. Name steps something meaningful as to what the step is orchestrating. Each step is executed serially, in the order defined.
image: 192.168.0.11:5000/nemonik/golang:1.12.6 - The build step is using the nemonik/golang container tagged 1.12.1 retrieved from private Docker registry located at 192.168.0.11:5000. Drone uses Docker images for the build environment, plugins and service containers. Drone spins them up for the execution of the pipeline and when no longer needed they go poof.
commands - Is a collection of terminal commands to be executed. These are all the same commands we executed previously in the command line. If anyone of these commands were to fail returning a non-zero exit code, the pipeline will immediately end resulting in a failed build.

9.10.11.1. Configure Drone to execute your pipeline

Open http://192.168.0.11/ in your browser and authenticate through GitLab on into Drone, if you need to.
Then select SYNC. The arrows will chase each other for a bit.
Then click root/helloworld repo and ACTIVATE REPOSITORY, then SAVE under the Main section to enable Drone orchestration for the project.
Then click the Drone logo in the upper left of the page to return home.

You won't have any builds to start, but when you do the builds will increment starting from 1. Red is failed the build. Yellow-orange is a presently executing build. Green is a build that passed. When a build does start, click on its row to open and monitor it. The UI will update as the build proceeds informing you as to its progress.

9.10.11.2. Trigger the build

To trigger the build, in your ssh connection to development simply commit your code:

git add .
git commit -m "Added Drone pipeline"
git push origin master

Immediately after you enter your GitLab username/password open http://192.168.0.11/root/hello in your browser, if you re-use an existing tab to this page refresh the page.

The execution of this pipeline will follow as so:

A new build will appear. Click on it.
Drone will clone your project's repository in a clone step.
And then will execute a build and run steps in order each spinning up nemonik/golang:1.12.6 container, whose container image was patched if proxy environmental variable are set to work behind MITRE's http proxy and SSL introspection. (Later, you may want examine the contents of ansible/roles/golang-container-image/files/Dockerfile for more details as to how this was accomplished.)
These steps execute the commands in the same way you executed them yourself: a. make lint b. make test c. make build d. make run

The output of the build (An arbitrary name. You could use "skippy".) step will resemble:

+ make lint
go fmt
go get golang.org/x/lint/golint
golint
+ make test
go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
coverage: 50.0% of statements
ok _/drone/src	0.004s	coverage: 50.0% of statements
+ make build
go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
coverage: 50.0% of statements
ok _/drone/src	(cached)	coverage: 50.0% of statements
go build -o helloworld -v
_/drone/src

The output of the run step will resemble:

+ make run
go build -o helloworld -v ./...
./helloworld
hello world

Our build was successful. Drone uses a container's exit code to determine success or failure. A container's non-zero exit code will cause the pipeline to exit immediately.

That's it. This is essentially CI. Remember, CI stands for Continuous Integration. Scintillating isn't it?

9.10.12. The completed source for helloworld

The helloworld project can be viewed completed at

https://github.com/nemonik/helloworld

9.11. Golang helloworld-web project

Like helloworld, the helloworld-web project is a very simple application that we will use to explore Continuous Deliver. Remember, Continuous Delivery builds upon Continuous Integration.

9.11.1. Create the project's backlog

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog” #FFFFFF
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

Open Taiga in your web browser

http://192.168.0.11:8080

Complete the follow to track your progress in completing the helloworld-web project

Click Create Project.
Select Kanban. A Kanban board shows how work moves from left to right, each column represents a stage within the value stream.
Give your project a name. For example, Helloworld-web and a description, such as, My Kanban board for this awesome helloworld-web app and then click CREATE PROJECT.
You can skip this step and opt to chose to click >< to fold READY, USER STORY STATUS and ARCHIVED only after completing step 6. Otherwise, you can edit your Kanban board to just NEW, IN PROGRESS, and DONE by a. On the bottom left, click the ADMIN gear. b. Click ATTRIBUTES. c. Scroll down to USER STORY STATUS. d. Hover over Ready, click the trash icon to delete and click ACCEPT. c. Do the same for Ready for test and Archived. d. Click the KANBAN icon on the far left. It Looks like columns. And then reload the browser to get the changes to take

In the NEW column select Add New bulk icon that looks like a list and when the page updates cut-and-paste the lines below into the text box and click SAVE.

Create the project's backlog
Create the project in GitLab
Setup the project in the development Vagrant
Author the application
Build and run the application
Run gometalinter.v2 on application
Fix the application
Author unit tests
Perform static analysis on the CLI
Write the Makefile
Dockerize the application
Run the Docker container
Push the container image to the private registry
Configure Drone to execute the CI/CD pipeline
Add Static Analysis (SonarQube) step to pipeline
Add build step to pipeline
Add container image publish step to pipeline
Add container deploy step to pipeline
Add compliance automation test (InSpec) step to pipeline
Add automated functional test (Selenium) to pipeline
Add DAST step (OWASP ZAP) to the pipeline

Track your progress in Taiga as you work through each section.

9.11.2. Create the project in GitLab

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab" #FFFFFF
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

In GitLab (http://192.168.0.11:10080/) click on Projects in the upper left. a. Select Your projects from the dropdown. b. Click the green New Project on the far right under Projects, or c. Click http://192.168.0.11:10080/projects/new
Leave the Project path defaulted to http://192.168.0.11:10080/root/.
Enter helloworld-web for the Project name. Be careful with the spelling.
Provide an optional Project description. Something descriptive, such as, "GoLang helloworld application for the hands-on DevOps class.".
Save yourself a headache, and make the application public.
Click the green Create Project button on the lower left.

The UI will refresh to show you landing page for the project that should be accessible from

http://192.168.0.11:10080/root/helloworld-web

9.11.3. Setup the project on the development Vagrant

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant" #FFFFFF
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

Create a helloworld-web GitLab hosted repo as you did prior for the helloworld project.

If you do not already have a shell open on development vagrant, on your host in a shell at the root of the class project enter the following

vagrant ssh development

Once connected to your development vagrant enter

cd ~/go/src/github.com/nemonik
git clone http://192.168.0.11:10080/root/helloworld-web.git
cd helloworld-web

NOTE

Again, ignore the "warning: You appear to have cloned an empty repository." warning.
The git clone will fail if you did not name your project correctly while in GitLab.

So, that you do not commit certian files to GitLab when you push, create a .gitignore file with your editor with this content

# OS-specific
.DS_Store

.scannerwork
*.out
*.json

helloworld-web

9.11.4. Author the application

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application" #FFFFFF
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

Create main.go in emacs, nano, vi, or vim with this content:

package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", handler)
	fmt.Print("listening on :3000\n")
	http.ListenAndServe(":3000", logRequest(http.DefaultServeMux))
}

func handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello world!\n")
}

func logRequest(handler http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Printf("%s  %s  %s\n", r.RemoteAddr, r.Method, r.URL)
		handler.ServeHTTP(w, r)
	})
}

Format and lint the code like you did previously.

go fmt
golint

9.11.5. Build and run the application

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application" #FFFFFF
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

Run the application

go build -o helloworld-web .

No output means things are peachy. Otherwise fix your mistakes.

Now run

./helloworld-web

Command line output will be

[vagrant@development helloworld-web]$ ./helloworld-web
listening on :3000

To run the application, either

Open http://192.168.0.10:3000 in a web browser, or
Enter curl http://192.168.0.10:3000 into the command-line.

Both will return:

Hello world!

The command line will output

192.168.0.1:63201  GET  /
192.168.0.1:63201  GET  /favicon.ico

crl-c will stop your application.

9.11.6. Run gometalinter.v2 on application

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application" #FFFFFF
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

The following command line steps are optional as you have installed these dependencies for the helloworld application

go get -u gopkg.in/alecthomas/gometalinter.v2
gometalinter.v2 --install

Output will resemble

Installing:
  deadcode
  dupl
  errcheck
  gochecknoglobals
  gochecknoinits
  goconst
  gocyclo
  goimports
  golint
  gosec
  gosimple
  gotype
  gotypex
  ineffassign
  interfacer
  lll
  maligned
  megacheck
  misspell
  nakedret
  safesql
  staticcheck
  structcheck
  unconvert
  unparam
  unused
  varcheck

Run our linters

gometalinter.v2

Gometalinter then runs all these Go linters:

go vet - Reports potential errors that otherwise compile.
go tool vet --shadow - Reports variables that may have been unintentionally shadowed.
gotype - Syntactic and semantic analysis similar to the Go compiler.
gotype -x - Syntactic and semantic analysis in external test packages (similar to the Go compiler).
deadcode - Finds unused code.
gocyclo - Computes the cyclomatic complexity of functions.
golint - Google's (mostly stylistic) linter.
varcheck - Find unused global variables and constants.
structcheck - Find unused struct fields.
maligned - Detect structs that would take less memory if their fields were sorted.
errcheck - Check that error return values are used.
megacheck - Run staticcheck, gosimple and unused, sharing work.
dupl - Reports potentially duplicated code.
ineffassign - Detect when assignments to existing variables are not used.
interfacer - Suggest narrower interfaces that can be used.
unconvert - Detect redundant type conversions.
goconst - Finds repeated strings that could be replaced by a constant.
gosec - Inspects source code for security problems by scanning the Go AST.

And the output like the follwing outout will be returned after some time

main.go:11:21:warning: error return value not checked (http.ListenAndServe(":3000", logRequest(http.DefaultServeMux))) (errcheck)
main.go:15:13:warning: error return value not checked (fmt.Fprintf(w, "Hello world!\n")) (errcheck)
main.go:11::warning: Errors unhandled.,LOW,HIGH (gosec)

Oops. Line 11 and line 15 have problems:

http.ListenAndServe(":3000", logRequest(http.DefaultServeMux)) on line 11 returns an err if it runs into problems. We need to handle the problem by logging the err and exit.
fmt.Fprintf on line 17 returns the number of bytes written and any write error encountered. We need to handle that too. We can swallow the number of bytes return via _, collect the err, log it, and exit.

9.11.7. Fix the application

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application" #FFFFFF
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

Let's fix the problems by importing fmt and logging the errs:

package main

import (
	"log"
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", handler)
	fmt.Print("listening on :3000\n")
	log.Fatal(http.ListenAndServe(":3000", logRequest(http.DefaultServeMux)))
}

func handler(w http.ResponseWriter, r *http.Request) {
	_, err := fmt.Fprintf(w, "Hello world!\n")
	if err != nil {
       		log.Fatal(err)
	}
}

func logRequest(handler http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Printf("%s  %s  %s\n", r.RemoteAddr, r.Method, r.URL)
		handler.ServeHTTP(w, r)
	})
}

Run our linters again

gometalinter.v2

And after some time, nothing is returned. Problem solved.

9.11.8. Author unit tests

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests" #FFFFFF
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

We don't have any unit tests, so let's fix that.

Create the text file main_test.go in emacs, nano, vi, or vim with this content:

package main

import (
	"io/ioutil"
	"net/http"
	"net/http/httptest"
	"os"
	"testing"
)

func TestLogRequest(t *testing.T) {

	mux := http.NewServeMux()
	mux.Handle("/", http.HandlerFunc(handler))

	l := logRequest(mux)

	// Create an http request
	req, _ := http.NewRequest("GET", "/", nil)

	// Create  http.ResponseWriter for test inspection
	recorder := httptest.NewRecorder()

	// Capture Stdout
	rescueStdout := os.Stdout
	r, w, _ := os.Pipe()
	os.Stdout = w

	l.ServeHTTP(recorder, req)

	// Stop capturing Stdout
	w.Close()
	out, _ := ioutil.ReadAll(r)
	os.Stdout = rescueStdout

	// Compare

	if string(out) != "  GET  /\n" {
		t.Errorf("logRequest didn't log the expected \"GET  /\"")
	}
}

func TestHandler(t *testing.T) {
	// Create an http request
	req, _ := http.NewRequest("GET", "/", nil)

	// Create  http.ResponseWriter for test inspection
	recorder := httptest.NewRecorder()

	// Call the handler
	handler(recorder, req)

	// Inspect the http.ResponseWriter
	if recorder.Code != http.StatusOK {
		t.Errorf("Server did not return %v", http.StatusOK)
	}

	if recorder.Body.String() != "Hello world!\n" {
		t.Errorf("Body contain \"%v\" instead of expected \"Hello world!\"", recorder.Body.String())
	}
}

func TestMain(m *testing.M) {
	os.Exit(m.Run())
}

Execute the unit test by entering

go test -v -cover

The command line will return something like

=== RUN   TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN   TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok  	github.com/nemonik/helloworld-web	0.005s

Notice we only scored 55.6% coverage, but had unit tests for all our methods? This where dicernment comes in.

9.11.9. Perform static analysis (i.e., sonar-scanner) on the command line

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI" #FFFFFF
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

SonarQube provides the capability to show the health of an application's source code, highlighting issues as they are introduced.

9.11.9.1. Optionally, register your app in SonarQube

This section is optional as the sonar-scannercommand line tool will do all this automagically for you.

Open in your browser

http://192.168.0.11:9000/admin/projects_management

If you have to authenticate, your username and password is

admin

Click Create project.

Provide the name of

helloworld-web

Provide the key of

helloworld-web

9.11.9.2. Install the SonarGo plugin

Open you browser to http://192.168.0.11:9000/admin/marketplace
If you need to login as admin with the pasword admin.
Search for SonarGo.
Click Install.
Click Restart.
Yes, your sure. Click Restart in the modal.

9.11.9.3. Perform static analysis (run sonar-scanner) on the command line

Generate the reports and submit to SonarQube

gometalinter.v2 > gometalinter-report.out
golint > golint-report.out
go test -v ./... -coverprofile=coverage.out
go test -v ./... -json > report.json
sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go

Output will look like

[vagrant@development helloworld-web]$ gometalinter.v2 > gometalinter-report.out
[vagrant@development helloworld-web]$ golint > golint-report.out
[vagrant@development helloworld-web]$ go test -v ./... -coverprofile=coverage.out
=== RUN   TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN   TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok  	github.com/nemonik/helloworld-web	0.004s	coverage: 55.6% of statements
[vagrant@development helloworld-web]$ go test -v ./... -json > report.json
[vagrant@development helloworld-web]$ sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go
INFO: Scanner configuration file: /usr/local/sonar-scanner-3.3.0.1492-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-957.5.1.el7.x86_64 amd64
INFO: User cache: /home/vagrant/.sonar/cache
INFO: SonarQube server 7.1.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=97ms
INFO: Server id: AWngQ8AWiugn-Qnmvhkr
INFO: User cache: /home/vagrant/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=41ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=92ms
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=22ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=51ms
INFO: Load active rules
INFO: Load active rules (done) | time=275ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=37ms
WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.
INFO: Project key: helloworld-web
INFO: Project base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: -------------  Scan helloworld-web
INFO: Load server rules
INFO: Load server rules (done) | time=22ms
INFO: Base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: Working dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web/.scannerwork
INFO: Source paths: .
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Index files
INFO: Excluded sources:
INFO:   **/*test.go
INFO: 6 files indexed
INFO: 1 file ignored because of inclusion/exclusion patterns
INFO: Quality profile for go: Sonar way
INFO: Sensor SonarGo [go]
INFO: Load coverage report from '/home/vagrant/go/src/github.com/nemonik/helloworld-web/coverage.out'
INFO: Sensor SonarGo [go] (done) | time=178ms
INFO: Sensor Go Unit Test Report [go]
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestLogRequest
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestHandler
INFO: Sensor Go Unit Test Report [go] (done) | time=10ms
INFO: Sensor Import of Golint issues [go]
ERROR: GoLintReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of Golint issues [go] (done) | time=0ms
INFO: Sensor Import of GoMetaLinter issues [go]
ERROR: GoMetaLinterReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of GoMetaLinter issues [go] (done) | time=0ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=14ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: Calculating CPD for 1 file
INFO: CPD calculation finished
INFO: Analysis report generated in 74ms, dir size=3 KB
INFO: Analysis reports compressed in 7ms, zip size=3 KB
INFO: Analysis report uploaded in 360ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://192.168.0.11:9000/dashboard/index/helloworld-web
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.0.11:9000/api/ce/task?id=AWngSRM9HEJ11KtPiWR1
INFO: Task total time: 1.771 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 3.839s
INFO: Final Memory: 7M/108M
INFO: ------------------------------------------------------------------------

NOTE

If you see this outputed

ERROR: Error during SonarQube Scanner execution
ERROR: No quality profiles have been found, you probably don't have any language plugin installed.

you didn't enable the SonarGo plugin in SonarQube.

Let me unpack what the above commands are doing

The first line go get github.com/alecthomas/gometalinter installs the gometalinter dependency for concurrently run Go lint tools and normalise their output

gometalinter.v2 > gometalinter-report.out executes all the above linters and consolidates their results into a single gometalinter-report.out report. Go is a very quiet language. As you may have noticed. So, when gometalinter.v2 runs and everything passes the output of report will contain nothing if there are no problems. If you want to be assured it called the linters it claims to call, you run gometalinter.v2 if debug like so gometalinter.v2 --debug.

go test -coverprofile=coverage.out executes your unit tests and generate the coverage.out report.

go test -json > report.json executes your unit tests and generate a unit tests execution report, report.json

sonar-scanner through -D parameters is configured to submit the reports. If you skipped the prior section, SonarQube will automatically create the project for you.

Open in your host's web browser

http://192.168.0.11:9000/dashboard?id=helloworld-web

to view your SonarQube report.

9.11.10. Automated the build (i.e., write the Makefile)

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile" #FFFFFF
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

In your editor create a Makefile to ensure the build and the steps leading to are repeatable.

GOCMD=go
GOFMT=$(GOCMD) fmt
GOBUILD=$(GOCMD) build
GOCLEAN=$(GOCMD) clean
GOTEST=$(GOCMD) test
GOLINT=$(GOCMD)lint
GOGET=$(GOCMD) get
BINARY_NAME=helloworld-web

SONAR_DEPENDENCIES := \
	gopkg.in/alecthomas/gometalinter.v2

all: sonar build
clean:
	$(GOCLEAN)
	rm -f $(BINARY_NAME)
fmt:
	$(GOFMT)
lint: fmt
	$(GOGET) golang.org/x/lint/golint
	golint
test: lint
	$(GOTEST) -v -cover ./...
sonar: test
	$(GOGET) -u $(SONAR_DEPENDENCIES)
	-gometalinter.v2 --install
	-gometalinter.v2 > gometalinter-report.out
	-$(GOLINT) > golint-report.out
	-$(GOTEST) -v ./... -coverprofile=coverage.out
	-$(GOTEST) -v ./... -json > report.json
	-sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go
build:
	$(GOBUILD) -o $(BINARY_NAME) -v
run:
	./$(BINARY_NAME)

NOTE

The indents are tab characters and not spaces characters. otherwise you make will fail to execute.
And cut-and-paste may split the single line beginning with sonar-scanner into multiple lines. You want this to be single line of text.

Test out your Makefile

make all

And so, you have build and test automation whose output resembles

[vagrant@development helloworld-web]$ make all
go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN   TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN   TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok  	github.com/nemonik/helloworld-web	(cached)	coverage: 55.6% of statements
go get -u gopkg.in/alecthomas/gometalinter.v2
gometalinter.v2 --install
Installing:
  deadcode
  dupl
  errcheck
  gochecknoglobals
  gochecknoinits
  goconst
  gocyclo
  goimports
  golint
  gosec
  gosimple
  gotype
  gotypex
  ineffassign
  interfacer
  lll
  maligned
  megacheck
  misspell
  nakedret
  safesql
  staticcheck
  structcheck
  unconvert
  unparam
  unused
  varcheck
gometalinter.v2 > gometalinter-report.out
golint > golint-report.out
go test -v ./... -coverprofile=coverage.out
=== RUN   TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN   TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok  	github.com/nemonik/helloworld-web	0.004s	coverage: 55.6% of statements
go test -v ./... -json > report.json
sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go
INFO: Scanner configuration file: /usr/local/sonar-scanner-3.3.0.1492-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-957.5.1.el7.x86_64 amd64
INFO: User cache: /home/vagrant/.sonar/cache
INFO: SonarQube server 7.1.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=76ms
INFO: Server id: AWngQ8AWiugn-Qnmvhkr
INFO: User cache: /home/vagrant/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=46ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=10ms
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=69ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=29ms
INFO: Load active rules
INFO: Load active rules (done) | time=88ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=38ms
WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.
INFO: Project key: helloworld-web
INFO: Project base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: -------------  Scan helloworld-web
INFO: Load server rules
INFO: Load server rules (done) | time=15ms
INFO: Base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: Working dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web/.scannerwork
INFO: Source paths: .
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Index files
INFO: Excluded sources:
INFO:   **/*test.go
INFO: 7 files indexed
INFO: 1 file ignored because of inclusion/exclusion patterns
INFO: Quality profile for go: Sonar way
INFO: Sensor SonarGo [go]
INFO: Load coverage report from '/home/vagrant/go/src/github.com/nemonik/helloworld-web/coverage.out'
INFO: Sensor SonarGo [go] (done) | time=158ms
INFO: Sensor Go Unit Test Report [go]
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestLogRequest
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestHandler
INFO: Sensor Go Unit Test Report [go] (done) | time=14ms
INFO: Sensor Import of Golint issues [go]
ERROR: GoLintReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of Golint issues [go] (done) | time=0ms
INFO: Sensor Import of GoMetaLinter issues [go]
ERROR: GoMetaLinterReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of GoMetaLinter issues [go] (done) | time=1ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=25ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: Calculating CPD for 1 file
INFO: CPD calculation finished
INFO: Analysis report generated in 67ms, dir size=3 KB
INFO: Analysis reports compressed in 7ms, zip size=3 KB
INFO: Analysis report uploaded in 32ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://192.168.0.11:9000/dashboard/index/helloworld-web
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.0.11:9000/api/ce/task?id=AWngTpT4HEJ11KtPiWR3
INFO: Task total time: 1.271 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 2.371s
INFO: Final Memory: 7M/112M
INFO: ------------------------------------------------------------------------
go build -o helloworld-web -v

9.11.11. Dockerize the application

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application" #FFFFFF
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

Build a Docker image named nemonik/helloworld-web for the application and all its dependencies to ensure a repeatable deployment on Docker by opening the file Dockerfile in an editor at the root of the project and copy the is content into it

FROM 192.168.0.11:5000/nemonik/golang:1.12.6
MAINTAINER Michael Joseph Walsh <[email protected]>

RUN mkdir /app
ADD main.go /app/

WORKDIR /app

RUN go build -o helloworld-web .

CMD ["/app/helloworld-web"]

EXPOSE 3000

Then create the Docker image

docker build -t nemonik/helloworld-web .

NOTE

Don't miss that last period (.) at the end the line above.

Command line output will resemble

[vagrant@development helloworld-web]$ docker build -t nemonik/helloworld-web .
Sending build context to Docker daemon  10.64MB
Step 1/8 : FROM 192.168.0.11:5000/nemonik/golang:1.12.6
1.12.1: Pulling from nemonik/golang
e79bb959ec00: Pull complete
d4b7902036fe: Pull complete
1b2a72d4e030: Pull complete
d54db43011fd: Pull complete
963c818ebafc: Pull complete
2c6333e9b74a: Pull complete
3b0c71504fac: Pull complete
eb7d59297445: Pull complete
6335f01326ba: Pull complete
bfcbc32359e9: Pull complete
76c51a60b271: Pull complete
Digest: sha256:e5f086026391553344abba61bcf184618f1ff6e6c32dc392f645bbd8971ab403
Status: Downloaded newer image for 192.168.0.11:5000/nemonik/6
 ---> 8bd76ee17c41
Step 2/8 : MAINTAINER Michael Joseph Walsh <[email protected]>
 ---> Running in f68f90915e36
Removing intermediate container f68f90915e36
 ---> 096e6501ab10
Step 3/8 : RUN mkdir /app
 ---> Running in b0ded1b2fead
Removing intermediate container b0ded1b2fead
 ---> d4752c51206f
Step 4/8 : ADD main.go /app/
 ---> ecfbd5db128d
Step 5/8 : WORKDIR /app
 ---> Running in 022f248d47f0
Removing intermediate container 022f248d47f0
 ---> 8bfa4710fdc1
Step 6/8 : RUN go build -o helloworld-web .
 ---> Running in 622b81b17191
Removing intermediate container 622b81b17191
 ---> 269d7f9c41b0
Step 7/8 : CMD ["/app/helloworld-web"]
 ---> Running in 78e6b23e9fc3
Removing intermediate container 78e6b23e9fc3
 ---> 37c8c1ed4ced
Step 8/8 : EXPOSE 3000
 ---> Running in fc138b7e2b5b
Removing intermediate container fc138b7e2b5b
 ---> d279e0618ec7
Successfully built d279e0618ec7
Successfully tagged nemonik/helloworld-web:latest

What just happened?

The FROM line instructs Docker to retreive the nemonik/golang:1.12.6 from the private Docker registry running on our Toolchain vagrant, which it did.
Then the rest of the commands in the Dockerfile are executed building a new docker image.
docker build then places the image with the name nemonik/helloworld-web in the Development's local Docker registry so that containers can be created off this image locally.

Check the local Docker registry via

docker images

The command line output will resemble

[vagrant@development helloworld-web]$ docker images
REPOSITORY                         TAG                 IMAGE ID            CREATED             SIZE
nemonik/helloworld-web             latest              facd0af56fe5        2 minutes ago       1.01GB
192.168.0.11:5000/nemonik/golang   1.11.2              32c900b4b4da        2 hours ago         1GB

NOTE:

The registy will also contain the 192.168.0.11:5000/nemonik/golang:1.12.6 on which nemonik/helloworld-web:latest is based, so the next time re-build image you won't have to wait for 192.168.0.11:5000/nemonik/golang:1.12.6 to be retrieved.

We can update the project's Makefile adding a docker-build target

docker-build: build
	docker build --no-cache -t nemonik/helloworld-web .

Also, add the docker-build to the all target like so:

all: sonar docker-build

Remember to us tab characters vice space characters.

And then run in the command lin

make all

Whose output will be

[vagrant@development helloworld-web]$ make all
go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN   TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN   TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok  	github.com/nemonik/helloworld-web	(cached)	coverage: 55.6% of statements
go get -u gopkg.in/alecthomas/gometalinter.v2
gometalinter.v2 --install
Installing:
  deadcode
  dupl
  errcheck
  gochecknoglobals
  gochecknoinits
  goconst
  gocyclo
  goimports
  golint
  gosec
  gosimple
  gotype
  gotypex
  ineffassign
  interfacer
  lll
  maligned
  megacheck
  misspell
  nakedret
  safesql
  staticcheck
  structcheck
  unconvert
  unparam
  unused
  varcheck
gometalinter.v2 > gometalinter-report.out
golint > golint-report.out
go test -v ./... -coverprofile=coverage.out
=== RUN   TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN   TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok  	github.com/nemonik/helloworld-web	0.004s	coverage: 55.6% of statements
go test -v ./... -json > report.json
sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go
INFO: Scanner configuration file: /usr/local/sonar-scanner-3.3.0.1492-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-957.5.1.el7.x86_64 amd64
INFO: User cache: /home/vagrant/.sonar/cache
INFO: SonarQube server 7.1.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=97ms
INFO: Server id: AWngQ8AWiugn-Qnmvhkr
INFO: User cache: /home/vagrant/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=56ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=13ms
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=79ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=30ms
INFO: Load active rules
INFO: Load active rules (done) | time=82ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=32ms
WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.
INFO: Project key: helloworld-web
INFO: Project base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: -------------  Scan helloworld-web
INFO: Load server rules
INFO: Load server rules (done) | time=15ms
INFO: Base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: Working dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web/.scannerwork
INFO: Source paths: .
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Index files
INFO: Excluded sources:
INFO:   **/*test.go
INFO: 8 files indexed
INFO: 1 file ignored because of inclusion/exclusion patterns
INFO: Quality profile for go: Sonar way
INFO: Sensor SonarGo [go]
INFO: Load coverage report from '/home/vagrant/go/src/github.com/nemonik/helloworld-web/coverage.out'
INFO: Sensor SonarGo [go] (done) | time=166ms
INFO: Sensor Go Unit Test Report [go]
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestLogRequest
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestHandler
INFO: Sensor Go Unit Test Report [go] (done) | time=25ms
INFO: Sensor Import of Golint issues [go]
ERROR: GoLintReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of Golint issues [go] (done) | time=1ms
INFO: Sensor Import of GoMetaLinter issues [go]
ERROR: GoMetaLinterReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of GoMetaLinter issues [go] (done) | time=0ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=16ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: Calculating CPD for 1 file
INFO: CPD calculation finished
INFO: Analysis report generated in 61ms, dir size=3 KB
INFO: Analysis reports compressed in 8ms, zip size=3 KB
INFO: Analysis report uploaded in 26ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://192.168.0.11:9000/dashboard/index/helloworld-web
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.0.11:9000/api/ce/task?id=AWngWOBIHEJ11KtPiWR4
INFO: Task total time: 1.280 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 2.733s
INFO: Final Memory: 7M/114M
INFO: ------------------------------------------------------------------------
go build -o helloworld-web -v
docker build --no-cache -t nemonik/helloworld-web .
Sending build context to Docker daemon  10.64MB
Step 1/8 : FROM 192.168.0.11:5000/nemonik/golang:1.12.6
 ---> 8bd76ee17c41
Step 2/8 : MAINTAINER Michael Joseph Walsh <[email protected]>
 ---> Running in f1da2f04877b
Removing intermediate container f1da2f04877b
 ---> 0eaaae49e98c
Step 3/8 : RUN mkdir /app
 ---> Running in 1a248ef09da9
Removing intermediate container 1a248ef09da9
 ---> 3ebe65b2cb75
Step 4/8 : ADD main.go /app/
 ---> 7de0547ee25e
Step 5/8 : WORKDIR /app
 ---> Running in b38d14f60f18
Removing intermediate container b38d14f60f18
 ---> 5c662ba39229
Step 6/8 : RUN go build -o helloworld-web .
 ---> Running in 28966ff6604d
Removing intermediate container 28966ff6604d
 ---> a2c445d6bb55
Step 7/8 : CMD ["/app/helloworld-web"]
 ---> Running in 90b64a9fd5e7
Removing intermediate container 90b64a9fd5e7
 ---> 241184d5dc62
Step 8/8 : EXPOSE 3000
 ---> Running in d5172565bd5d
Removing intermediate container d5172565bd5d
 ---> 3c1cdf32fbfd
Successfully built 3c1cdf32fbfd
Successfully tagged nemonik/helloworld-web:latest

NOTE

As noted before the base image already existed in the local Docker registry thereby saving us time.

9.11.12. Run the Docker container

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container" #FFFFFF
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

Spin up a new nemonik/helloworld-web container by either entering

docker run -p 3000:3000 --name helloworld-web nemonik/helloworld-web and hitting http://192.168.0.10:3000 in your web browser, or
docker run -d -p 3000:3000 --name helloworld-web nemonik/helloworld-web and hitting the same URL via curl http://192.168.0.10:3000.

Where

run messages Docker you are running a new container
-d in the second option, runs the container in background and prints container ID.
-p 3000:3000 published the container's port as 3000 to the host.
--name helloworld-web names the running container
nemonik/helloworld-web states what container image to use.

9.11.12.1. Option 1

The command line output for the first option will be

[vagrant@development helloworld-web]$ docker run -p 3000:3000 --name helloworld-web nemonik/helloworld-web
listening on :3000
192.168.0.1:54740  GET  /
192.168.0.1:54740  GET  /favicon.ico
^C

9.11.12.2. Option 2

For the second option there will be no output written to the screen, but you can see the same output if you run

docker logs <the container id output when started the container> -f

or

docker logs helloworld-web  -f

returning

[vagrant@development helloworld-web]$ docker logs 31257f287d26bdb8ae02d9d444f8ebb1478da183f7d6f908767e006c212fcbd8 -f
listening on :3000
192.168.0.1:55159  GET  /
192.168.0.1:55159  GET  /favicon.ico

ctrl-c (i.e., press and hold the ctrl key while pressing c) to exit the logs.

To kill the container

docker rm -f <the container id output when started the container>

and the container id returned when you ran the container.

9.11.13. Push the container image to the private Docker registry

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry" #FFFFFF
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

In the development vagrant, push the nemonik/hellowrold-web container image into the private Docker registry running on the Toolchain vagrant, so that both vagrants can create containers from the image with the commands

docker tag nemonik/helloworld-web 192.168.0.11:5000/nemonik/helloworld-web
docker push 192.168.0.11:5000/nemonik/helloworld-web

Command line output will be

[vagrant@development helloworld-web]$ docker tag nemonik/helloworld-web 192.168.0.11:5000/nemonik/helloworld-web
[vagrant@development helloworld-web]$ docker push 192.168.0.11:5000/nemonik/helloworld-web
The push refers to repository [192.168.0.11:5000/nemonik/helloworld-web]
f7ac261b7c4d: Pushed
67624b1e99ab: Pushed
53ce894cd0ad: Pushed
f3956851d408: Mounted from nemonik/golang
dea8f6ac2dd7: Mounted from nemonik/golang
303fb5379543: Mounted from nemonik/golang
d6ae08d86ad3: Mounted from nemonik/golang
49c92637f9e9: Mounted from nemonik/golang
9ccbb4ca29f8: Mounted from nemonik/golang
f4907c4e3f89: Mounted from nemonik/golang
b17cc31e431b: Mounted from nemonik/golang
12cb127eee44: Mounted from nemonik/golang
604829a174eb: Mounted from nemonik/golang
fbb641a8b943: Mounted from nemonik/golang
latest: digest: sha256:0d7508fadaa570ac261906c95054efc96663ab6c9c7131a13cbd225a2e77eb0c size: 3267

We can update the project's Makefile adding a docker-push target

docker-push: docker-build
	docker tag nemonik/helloworld-web 192.168.0.11:5000/nemonik/helloworld-web
	docker push 192.168.0.11:5000/nemonik/helloworld-web

Also, add the docker-push to the all target like so:

all: sonar docker-push

And then run in the command line

make all

The output will be something like

[vagrant@development helloworld-web]$ make all
go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN   TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN   TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok  	github.com/nemonik/helloworld-web	(cached)	coverage: 55.6% of statements
go get -u gopkg.in/alecthomas/gometalinter.v2
gometalinter.v2 --install
Installing:
  deadcode
  dupl
  errcheck
  gochecknoglobals
  gochecknoinits
  goconst
  gocyclo
  goimports
  golint
  gosec
  gosimple
  gotype
  gotypex
  ineffassign
  interfacer
  lll
  maligned
  megacheck
  misspell
  nakedret
  safesql
  staticcheck
  structcheck
  unconvert
  unparam
  unused
  varcheck
gometalinter.v2 > gometalinter-report.out
golint > golint-report.out
go test -v ./... -coverprofile=coverage.out
=== RUN   TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN   TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok  	github.com/nemonik/helloworld-web	0.004s	coverage: 55.6% of statements
go test -v ./... -json > report.json
sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go
INFO: Scanner configuration file: /usr/local/sonar-scanner-3.3.0.1492-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-957.5.1.el7.x86_64 amd64
INFO: User cache: /home/vagrant/.sonar/cache
INFO: SonarQube server 7.1.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=78ms
INFO: Server id: AWngQ8AWiugn-Qnmvhkr
INFO: User cache: /home/vagrant/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=67ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=12ms
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=78ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=26ms
INFO: Load active rules
INFO: Load active rules (done) | time=83ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=34ms
WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.
INFO: Project key: helloworld-web
INFO: Project base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: -------------  Scan helloworld-web
INFO: Load server rules
INFO: Load server rules (done) | time=12ms
INFO: Base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: Working dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web/.scannerwork
INFO: Source paths: .
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Index files
INFO: Excluded sources:
INFO:   **/*test.go
INFO: 8 files indexed
INFO: 1 file ignored because of inclusion/exclusion patterns
INFO: Quality profile for go: Sonar way
INFO: Sensor SonarGo [go]
INFO: Load coverage report from '/home/vagrant/go/src/github.com/nemonik/helloworld-web/coverage.out'
INFO: Sensor SonarGo [go] (done) | time=159ms
INFO: Sensor Go Unit Test Report [go]
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestLogRequest
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestHandler
INFO: Sensor Go Unit Test Report [go] (done) | time=14ms
INFO: Sensor Import of Golint issues [go]
ERROR: GoLintReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of Golint issues [go] (done) | time=16ms
INFO: Sensor Import of GoMetaLinter issues [go]
ERROR: GoMetaLinterReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of GoMetaLinter issues [go] (done) | time=0ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=15ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: Calculating CPD for 1 file
INFO: CPD calculation finished
INFO: Analysis report generated in 70ms, dir size=3 KB
INFO: Analysis reports compressed in 8ms, zip size=3 KB
INFO: Analysis report uploaded in 31ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://192.168.0.11:9000/dashboard/index/helloworld-web
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.0.11:9000/api/ce/task?id=AWngW6IWHEJ11KtPiWR5
INFO: Task total time: 1.371 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 2.879s
INFO: Final Memory: 8M/115M
INFO: ------------------------------------------------------------------------
go build -o helloworld-web -v
docker build --no-cache -t nemonik/helloworld-web .
Sending build context to Docker daemon  10.64MB
Step 1/8 : FROM 192.168.0.11:5000/nemonik/golang:1.12.6
 ---> 8bd76ee17c41
Step 2/8 : MAINTAINER Michael Joseph Walsh <[email protected]>
 ---> Running in 4de3c6d49980
Removing intermediate container 4de3c6d49980
 ---> 1f58e6e0faf3
Step 3/8 : RUN mkdir /app
 ---> Running in 7cfdef36f806
Removing intermediate container 7cfdef36f806
 ---> 29ec163e00eb
Step 4/8 : ADD main.go /app/
 ---> af9663389a72
Step 5/8 : WORKDIR /app
 ---> Running in a3fd471cf2b9
Removing intermediate container a3fd471cf2b9
 ---> ae03915db2cc
Step 6/8 : RUN go build -o helloworld-web .
 ---> Running in d49a8cc11d8a
Removing intermediate container d49a8cc11d8a
 ---> 5838172f19ba
Step 7/8 : CMD ["/app/helloworld-web"]
 ---> Running in 26f26efb5e11
Removing intermediate container 26f26efb5e11
 ---> d0390a3e0c83
Step 8/8 : EXPOSE 3000
 ---> Running in 84d2f0fa920d
Removing intermediate container 84d2f0fa920d
 ---> 79aa7a9c6392
Successfully built 79aa7a9c6392
Successfully tagged nemonik/helloworld-web:latest
docker tag nemonik/helloworld-web 192.168.0.11:5000/nemonik/helloworld-web
docker push 192.168.0.11:5000/nemonik/helloworld-web
The push refers to repository [192.168.0.11:5000/nemonik/helloworld-web]
636decb58204: Pushed
166b3d7a9755: Pushed
fa87d269f747: Pushed
f3956851d408: Layer already exists
dea8f6ac2dd7: Layer already exists
303fb5379543: Layer already exists
d6ae08d86ad3: Layer already exists
49c92637f9e9: Layer already exists
9ccbb4ca29f8: Layer already exists
f4907c4e3f89: Layer already exists
b17cc31e431b: Layer already exists
12cb127eee44: Layer already exists
604829a174eb: Layer already exists
fbb641a8b943: Layer already exists
latest: digest: sha256:0e30de168363aa683b78cc010ef6cd88427579e18ba2b757e4283bb6bec5f5a0 size: 3267

The Docker registry container image shipped by Docker does not provide a GUI, but we can verify by querying the catalog of the private registry through a web browser or Unix command line tool curl by entering into the command line

curl -X GET http://192.168.0.11:5000/v2/_catalog

Returns in the command line

[vagrant@development helloworld-web]$ curl -X GET http://192.168.0.11:5000/v2/_catalog
{"repositories":["nemonik/golang","nemonik/golang-sonarqube-scanner","nemonik/helloworld-web","nemonik/python","nemonik/standalone-firefox","nemonik/zap2docker-stable"]}

The Docker registry on our Toolchain vagrant doesn't have a human readable web UI, but you can query for the container you pushed and read the JSON:

curl -X GET http://192.168.0.11:5000/v2/nemonik/helloworld-web/tags/list

Returns in the command line

[vagrant@development helloworld-web]$ curl -X GET http://192.168.0.11:5000/v2/nemonik/helloworld-web/tags/list
{"name":"nemonik/helloworld-web","tags":["latest"]}

9.11.14. Configure Drone to execute your CI/CD pipeline

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline" #FFFFFF
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"

-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

As you did for the purpose of CI (Continuous Integration) of the prior application, you will need to configure Drone to perform CI/CD (a combination of Continuous Integration and Continuous Delivery) on the helloworld-web application.

Complete the following:

Open http://192.168.0.11/ in your browser and authenticate through GitLab on into Drone, if you need to.
Then select SYNC. The arrows will chase each other for a bit.
Then click root/helloworld-web repo and ACTIVATE REPOSITORY, then SAVE under the Main section to enable Drone orchestration for the project.
Then click the Drone logo in the upper left of the page to return home.

9.11.15. Add Static Analysis (SonarQube) step to pipeline

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline" #FFFFFF
-right->"Add build step\nto pipeline" 
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"

-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

So, lets create our pipeline starting with a sonar-scan step to update SonarQube with code quality scans automatically by opening .drone.yml in our text editor and adding the following step right after our build step, so that we run the step prior to a deploy.

kind: pipeline
name: default

steps:
- name: sonarqube
  image: 192.168.0.11:5000/nemonik/golang-sonarqube-scanner:3.3.0.1492
  commands:
  - export DRONESRC=`pwd`
  - export GOBIN=$GOPATH/bin
  - mkdir -p $GOPATH/src/github.com/nemonik
  - cd $GOPATH/src/github.com/nemonik
  - ln -s $DRONESRC helloworld-web
  - cd helloworld-web
  - go get -u golang.org/x/lint/golint
  - go get -u gopkg.in/alecthomas/gometalinter.v2
  - gometalinter.v2 --install
  - gometalinter.v2 --deadline=120s --checkstyle > gometalinter-report.out || true
  - go lint > golint-report.out || true
  - go test -v ./... -coverprofile=coverage.out || true
  - go test -v ./... -json > report.json || true
  - sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go || true

Things to note in the above

This step uses an image, nemonik/golang-sonarqube-scanner:3.3.0.1492, built ontop of the nemonik/golang:1.12.6 image to speed builds along.
Cut-and-pasting may split the last -sonar-scanner line into multiple line resulting in your build failing. Correct in the editor.
Make sure to leave two returns to indicate the end of the .drone.yml file or a parse error will result in Drone.

  - export DRONESRC=`pwd`
  - export GOBIN=$GOPATH/bin
  - mkdir -p $GOPATH/src/github.com/nemonik
  - cd $GOPATH/src/github.com/nemonik
  - ln -s $DRONESRC helloworld-web

What follows handles running the scan absorbing errors as they arise, so as to not break the build.

  - gometalinter.v2 --install
  - gometalinter.v2 --deadline=120s --checkstyle > gometalinter-report.out || true
  - go lint > golint-report.out || true
  - go test -v ./... -coverprofile=coverage.out || true
  - go test -v ./... -json > report.json || true
  - sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go || true

Commit the code to GitLab hosted remote repo

git add .drone.yml
git commit -m "added sonar step to pipeline"
git push origin master

Open on your host

http://192.168.0.11/root/helloworld-web

And monitor the progress of the build. The pipeline should execute in a few minutes.

Typical success resembles

+ export DRONESRC=/drone/src
+ export GOBIN=$GOPATH/bin
+ mkdir -p $GOPATH/src/github.com/nemonik
+ cd $GOPATH/src/github.com/nemonik
+ ln -s $DRONESRC helloworld-web
+ cd helloworld-web
+ go get -u golang.org/x/lint/golint
+ go get -u gopkg.in/alecthomas/gometalinter.v2
+ gometalinter.v2 --install
Installing:
deadcode
dupl
errcheck
gochecknoglobals
gochecknoinits
goconst
gocyclo
goimports
golint
gosec
gosimple
gotype
gotypex
ineffassign
interfacer
lll
maligned
megacheck
misspell
nakedret
safesql
staticcheck
structcheck
unconvert
unparam
unused
varcheck
+ gometalinter.v2 --deadline=120s --checkstyle > gometalinter-report.out || true
+ go lint > golint-report.out || true
go lint: unknown command
Run 'go help' for usage.
+ go test -v ./... -coverprofile=coverage.out || true
go: warning: "./..." matched no packages
no packages to test
+ go test -v ./... -json > report.json || true
go: warning: "./..." matched no packages
no packages to test
+ sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go || true
INFO: Scanner configuration file: /usr/local/sonar-scanner-3.3.0.1492-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-957.5.1.el7.x86_64 amd64
INFO: User cache: /root/.sonar/cache
INFO: SonarQube server 7.1.0
INFO: Default locale: "en_US", source code encoding: "US-ASCII" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=83ms
INFO: Server id: AWngQ8AWiugn-Qnmvhkr
INFO: User cache: /root/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=49ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=89ms
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=79ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=27ms
INFO: Load active rules
INFO: Load active rules (done) | time=84ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=33ms
WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.
INFO: Project key: helloworld-web
INFO: Project base dir: /go/src/github.com/nemonik/helloworld-web
INFO: ------------- Scan helloworld-web
INFO: Load server rules
INFO: Load server rules (done) | time=19ms
INFO: Base dir: /go/src/github.com/nemonik/helloworld-web
INFO: Working dir: /go/src/github.com/nemonik/helloworld-web/.scannerwork
INFO: Source paths: .
INFO: Source encoding: US-ASCII, default locale: en_US
INFO: Index files
INFO: Excluded sources:
INFO: **/*test.go
INFO: 3 files indexed
INFO: 0 files ignored because of inclusion/exclusion patterns
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=11ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: Calculating CPD for 0 files
INFO: CPD calculation finished
INFO: Analysis report generated in 65ms, dir size=1 KB
INFO: Analysis reports compressed in 10ms, zip size=1 KB
INFO: Analysis report uploaded in 29ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://192.168.0.11:9000/dashboard/index/helloworld-web
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.0.11:9000/api/ce/task?id=AWngY_tOHEJ11KtPiWR6
INFO: Task total time: 1.169 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 3.359s
INFO: Final Memory: 11M/132M
INFO: ------------------------------------------------------------------------

9.11.16. Add the build step to the pipeline

Add a build step to our .drone.yml

- name: build
  image: nemonik/golang:1.12.6
  commands:
  - make build

NOTE

Make sure to leave two consecutive returns to indicate the end of the .drone.yml file or a parse error will result in Drone.

Then commit to GitLab

git add .
git commit -m "added build step to pipeline"
git push origin master

Since we have not registered an SSH key with GitLab, we will need to enter a Username and Password when prompted.

In the browser open Drone at http://192.168.0.11/root/helloworld-web and click on the build being executed to monitor progress.

Output for build resembles

+ make build
go build -o helloworld-web -v
_/drone/src/192.168.0.11/root/helloworld-web

Mirroring what you saw in development in your local environment.

9.11.17. Add container image publish step to pipeline

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" #FFFFFF
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"

-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

Add the publish step to your .drone.yml so that the container image is publish to the private registry via the pipeline. The publish: step must be indented the same as the prior build: step.

- name: publish
  image: plugins/docker
  settings:
    storage_driver: overlay
    insecure: true
    registry: 192.168.0.11:5000
    repo: 192.168.0.11:5000/nemonik/helloworld-web
    force_tag: true
    tags:
    - latest
    custom_dns:
    - "10.20.100.53"
    - "10.20.200.53"

Note

Your DNS server entries for custom_dns may vary for your environment or this line may not be needed. In this case, the DNS servers were needed to be pass otherwise the build will fail to resolve the enterprise proxy when it attempts to retrieve external dependency to include in the container image.
Make sure to leave two consecutive returns to indicate the end of the .drone.yml file or a parse error will result in Drone.

Push your code into GitLab

git add .
git commit -m "added publish step to pipeline"
git push origin master

The publish step takes quite a bit of time and resembles:

+ /usr/local/bin/dockerd -g /var/lib/docker -s overlay --insecure-registry 192.168.0.11:5000 --dns 10.20.100.53 --dns 10.20.200.53
Registry credentials not provided. Guest mode enabled.
+ /usr/local/bin/docker version
Client:
 Version:	17.12.0-ce
 API version:	1.35
 Go version:	go1.9.2
 Git commit:	c97c6d6
 Built:	Wed Dec 27 20:05:38 2017
 OS/Arch:	linux/amd64

Server:
 Engine:
  Version:	17.12.0-ce
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.2
  Git commit:	c97c6d6
  Built:	Wed Dec 27 20:12:29 2017
  OS/Arch:	linux/amd64
  Experimental:	false
+ /usr/local/bin/docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.12.0-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.14.4.el7.x86_64
Operating System: Alpine Linux v3.7 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 5.67GiB
Name: 5f313f5c5c13
ID: 5H5L:Z6X5:IPNQ:YTB3:GBVC:RV2O:3BP3:KAWS:JQZA:MKIT:HI3K:AWGR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 192.168.0.11:5000
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
+ /usr/local/bin/docker build --rm=true -f Dockerfile -t 05cfa544b6fc3082f61155a9797d8ae5628a8d74 . --pull=true --label org.label-schema.schema-version=1.0 --label org.label-schema.build-date=2018-12-01T06:59:05Z --label org.label-schema.vcs-ref=05cfa544b6fc3082f61155a9797d8ae5628a8d74 --label org.label-schema.vcs-url=http://192.168.0.11:10080/root/helloworld-web.git
Sending build context to Docker daemon  6.653MB

Step 1/9 : FROM 192.168.0.11:5000/nemonik/golang:1.12.6
1.11.2: Pulling from nemonik/golang
54f7e8ac135a: Pulling fs layer
d6341e30912f: Pulling fs layer
087a57faf949: Pulling fs layer
5d71636fb824: Pulling fs layer
f368dba6a331: Pulling fs layer
f2e97ec220f9: Pulling fs layer
e4c39f2ed8c2: Pulling fs layer
a2992ac95eb6: Pulling fs layer
b6347f061623: Pulling fs layer
624cedf99a02: Pulling fs layer
b3caf54db51f: Pulling fs layer
e4c39f2ed8c2: Waiting
a2992ac95eb6: Waiting
b6347f061623: Waiting
624cedf99a02: Waiting
b3caf54db51f: Waiting
5d71636fb824: Waiting
f368dba6a331: Waiting
f2e97ec220f9: Waiting
087a57faf949: Verifying Checksum
087a57faf949: Download complete
d6341e30912f: Verifying Checksum
d6341e30912f: Download complete
54f7e8ac135a: Verifying Checksum
54f7e8ac135a: Download complete
5d71636fb824: Verifying Checksum
5d71636fb824: Download complete
e4c39f2ed8c2: Verifying Checksum
e4c39f2ed8c2: Download complete
f368dba6a331: Verifying Checksum
f368dba6a331: Download complete
a2992ac95eb6: Verifying Checksum
a2992ac95eb6: Download complete
b6347f061623: Verifying Checksum
b6347f061623: Download complete
624cedf99a02: Verifying Checksum
624cedf99a02: Download complete
f2e97ec220f9: Verifying Checksum
f2e97ec220f9: Download complete
b3caf54db51f: Verifying Checksum
b3caf54db51f: Download complete
54f7e8ac135a: Pull complete
d6341e30912f: Pull complete
087a57faf949: Pull complete
5d71636fb824: Pull complete
f368dba6a331: Pull complete
f2e97ec220f9: Pull complete
e4c39f2ed8c2: Pull complete
a2992ac95eb6: Pull complete
b6347f061623: Pull complete
624cedf99a02: Pull complete
b3caf54db51f: Pull complete
Digest: sha256:5f5b7b244da69c56a95bb1c924c2557e08b60b703c7f3131884f73d02f907bab
Status: Downloaded newer image for 192.168.0.11:5000/nemonik/golang:1.12.6
 ---> 32c900b4b4da
Step 2/9 : MAINTAINER Michael Joseph Walsh <[email protected]>
 ---> Running in ac61aba08f1c
Removing intermediate container ac61aba08f1c
 ---> 33a70df94816
Step 3/9 : RUN mkdir /app
 ---> Running in 52c411171688
Removing intermediate container 52c411171688
 ---> ce4ef1288d28
Step 4/9 : ADD main.go /app/
 ---> 8adeb9f66464
Step 5/9 : WORKDIR /app
Removing intermediate container 8e92749cfe8d
 ---> 8f164d7c66c9
Step 6/9 : RUN go build -o helloworld-web .
 ---> Running in dbc17c5c9e1c
Removing intermediate container dbc17c5c9e1c
 ---> 807609ef0468
Step 7/9 : CMD ["/app/helloworld-web"]
 ---> Running in a01670b6a0aa
Removing intermediate container a01670b6a0aa
 ---> 549af300f148
Step 8/9 : EXPOSE 3000
 ---> Running in 2455b914519f
Removing intermediate container 2455b914519f
 ---> fa6831463c27
Step 9/9 : LABEL "org.label-schema.build-date"='2018-12-01T06:59:05Z' "org.label-schema.schema-version"='1.0' "org.label-schema.vcs-ref"='05cfa544b6fc3082f61155a9797d8ae5628a8d74' "org.label-schema.vcs-url"='http://192.168.0.11:10080/root/helloworld-web.git'
 ---> Running in cff7286ce8c2
Removing intermediate container cff7286ce8c2
 ---> 495506e61a7a
Successfully built 495506e61a7a
Successfully tagged 05cfa544b6fc3082f61155a9797d8ae5628a8d74:latest
+ /usr/local/bin/docker tag 05cfa544b6fc3082f61155a9797d8ae5628a8d74 192.168.0.11:5000/nemonik/helloworld-web:latest
+ /usr/local/bin/docker push 192.168.0.11:5000/nemonik/helloworld-web:latest
The push refers to repository [192.168.0.11:5000/nemonik/helloworld-web]
b30658e0f8ca: Preparing
ec0ffa96b9d6: Preparing
65083988c970: Preparing
382408b8ce47: Preparing
48302eac2a97: Preparing
274449f3d60f: Preparing
b65d59e413b9: Preparing
652fefcd9ea8: Preparing
8019683f918e: Preparing
e7b9eaeca217: Preparing
f75e64f96dbc: Preparing
8f7ee6d76fd9: Preparing
c23711a84ad4: Preparing
90d1009ce6fe: Preparing
8019683f918e: Waiting
e7b9eaeca217: Waiting
f75e64f96dbc: Waiting
8f7ee6d76fd9: Waiting
274449f3d60f: Waiting
c23711a84ad4: Waiting
b65d59e413b9: Waiting
90d1009ce6fe: Waiting
652fefcd9ea8: Waiting
48302eac2a97: Layer already exists
382408b8ce47: Layer already exists
274449f3d60f: Layer already exists
652fefcd9ea8: Layer already exists
b65d59e413b9: Layer already exists
e7b9eaeca217: Layer already exists
8019683f918e: Layer already exists
65083988c970: Pushed
ec0ffa96b9d6: Pushed
f75e64f96dbc: Layer already exists
8f7ee6d76fd9: Layer already exists
90d1009ce6fe: Layer already exists
c23711a84ad4: Layer already exists
b30658e0f8ca: Pushed
latest: digest: sha256:e692b9f876d463755ab58cc07c124139d2d99fb4b8da4a55832a8b9ff9665b3a size: 3267
+ /usr/local/bin/docker rmi 05cfa544b6fc3082f61155a9797d8ae5628a8d74
Untagged: 05cfa544b6fc3082f61155a9797d8ae5628a8d74:latest
+ /usr/local/bin/docker system prune -f
Total reclaimed space: 0B

Indicating the publish step executed, successfully.

9.11.18. Add container deploy step to pipeline

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline" #FFFFFF
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"

-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

Usually, one adds a Drone secret and the appleboy/ssh plugin, but the last I looked there appeared to be a bug in the plugin, but one can accomplish the same goal using the plugin slightly differently.

Enable the Trusted setting for the repository in Drone by opening

http://192.168.0.11/root/helloworld-web/settings
Under Main and then Project Settings check off Trusted.
Leave the others to their defaults and click SAVE and look for Drone to float a modal at the bottom left denoting
```
Successfully saved
```
indicating success.
Then click the Drone icon in the upper left of the page to return home.

Since we let Vagrant manage the creation and configuration of each Vagrant's SSH keys for each of our variants (e.g., dev, toolchain) vagrant will have to create a unique key pair for each vagrant. Vagrant store each vagrant's private inside the .vagrant path created in the class project.

The toolchain vagrant's private keys exists at

/vagrant/.vagrant/machines/toolchain/virtualbox/private_key

We can access each private key though from inside each vagrant, because we configured each to mount the class project to the path /vagrant with the line

    dev.vm.synced_folder ".", "/vagrant", type: "virtualbox"

placed inside of each vagrant's config.vm.define block.

For example, while on the development vagrant enter the following command

ls /vagrant

It will show you class project.

Open root/helloworld-web's .drone.yml in your editor and add an additional deploy: step below the ones you already have with the deploy: being indented the same as the prior build: and publish: steps.

- name: deploy
  image: appleboy/drone-ssh
  volumes:
  - name: private_key
    path: /root/ssh/drone_rsa
  settings:
    host: 192.168.0.11
    port: 22
    username: vagrant
    key_path: /root/ssh/drone_rsa
    command_timeout: 5m
    script:
      - docker stop helloworld-web 2>/dev/null
      - docker rm helloworld-web 2>/dev/null
      - docker rmi 192.168.0.11:5000/nemonik/helloworld-web 2>/dev/null
      - docker run -d --restart=always --name helloworld-web --publish 3000:3000 192.168.0.11:5000/nemonik/helloworld-web

volumes:
- name: private_key
  host:
    path: /vagrant/.vagrant/machines/toolchain/virtualbox/private_key

NOTE

Make sure to leave two consecutive returns to indicate the end of the .drone.yml file or a parse error will result.

Commit the code to GitLab hosted remote repo

git add .drone.yml
git commit -m "added deploy step to pipeline"
git push origin master

The deploy step output resembles:

======CMD======
docker stop helloworld-web 2>/dev/null
docker rm helloworld-web 2>/dev/null
docker rmi 192.168.0.11:5000/nemonik/helloworld-web 2>/dev/null
docker run -d --restart=always --name helloworld-web --publish 3000:3000 192.168.0.11:5000/nemonik/helloworld-web
======END======
out: f56fa19cf749134fb8e1a53f8e4706612a7956fe0756d25e4d0a2284f3b5fa4f
err: Unable to find image '192.168.0.11:5000/nemonik/helloworld-web:latest' locally
err: latest: Pulling from nemonik/helloworld-web
err: 54f7e8ac135a: Already exists
err: d6341e30912f: Already exists
err: 087a57faf949: Already exists
err: 5d71636fb824: Already exists
err: f368dba6a331: Already exists
err: f2e97ec220f9: Already exists
err: e4c39f2ed8c2: Already exists
err: a2992ac95eb6: Already exists
err: b6347f061623: Already exists
err: 624cedf99a02: Already exists
err: b3caf54db51f: Already exists
err: a9520f945746: Pulling fs layer
err: 19de2ec0e886: Pulling fs layer
err: aa0e691fa83d: Pulling fs layer
err: 19de2ec0e886: Verifying Checksum
err: 19de2ec0e886: Download complete
err: a9520f945746: Download complete
err: aa0e691fa83d: Verifying Checksum
err: aa0e691fa83d: Download complete
err: a9520f945746: Pull complete
err: 19de2ec0e886: Pull complete
err: aa0e691fa83d: Pull complete
err: Digest: sha256:208ecc85c106a81d67bca9cd81146403568b16a4e66996c996f622c1b803c734
err: Status: Downloaded newer image for 192.168.0.11:5000/nemonik/helloworld-web:latest
==========================================
Successfully executed commands to all host.
==========================================

Open on your host

http://192.168.0.11/root/helloworld-web

And monitor the progress of the build. The pipeline should execute in a few minutes.

Once completed sucessfully, on your host open secure shell to your toolchain vagrant via

vagrant ssh toolchain

Then list the running Docker containers filtering for the one the pipeline just created

docker ps --filter "name=helloworld-web"

Command line output will resemble

[root@toolchain drone]# docker ps --filter "name=helloworld-web"
CONTAINER ID        IMAGE                                      COMMAND                 CREATED             STATUS              PORTS                    NAMES
9d08909cc2ee        192.168.0.11:5000/nemonik/helloworld-web   "/app/helloworld-web"   6 seconds ago       Up 5 seconds        0.0.0.0:3000->3000/tcp   helloworld-web

Then either on your docker host in a browser open

http://192.168.0.11:3000

or in the command-line of your development vagrant enter

curl http://192.168.0.11:3000

Either way the container will return

Hello world!

So, now we have beginnings of a real CI/CD pipeline. There are no strings on me.

9.11.19. Add complaince and policy automation (InSpec) test to the pipeline

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline" #FFFFFF
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

InSpec tests permit you to author compliance as code thereby turing compliance, security, and other policy requirements into automated tests.

InSpec has a very robust support for both the docker container and its host, see: https://lollyrock.com/posts/inspec-for-docker/ for more information for using InSpec with Docker.

More on InSpec in general can be found here

https://www.inspec.io/docs/

And these specific sectons are relevant to Docker:

https://www.inspec.io/docs/reference/resources/docker_container/

https://www.inspec.io/docs/reference/resources/docker/

https://www.inspec.io/docs/reference/resources/docker_image/

9.11.19.1. First author an InSpec test

At the root of the helloworld-web project create a directory to hold your inspec test

mkdir inspec-tests
cd inspec-tests

Then author a simple Ruby-based tests to verify the expected state of the nemonik/helloworld-web:latest Docker container by creating inspec.rb in an editor with this content:

control 'docker-checks-1.1' do
  impact 0.7 # High Impact
  tag "nist": ['CM-6', 'Rev_4']
  title 'Verify Docker Container exists and is running'

  describe docker_container(name: 'helloworld-web') do
    it { should exist }
    it { should be_running }
    its('repo') { should eq '192.168.0.11:5000/nemonik/helloworld-web' }
    its('ports') { should eq '0.0.0.0:3000->3000/tcp' }
    its('command') { should eq '/app/helloworld-web' }
  end
end

And then create the profile's metadata file, inspec.yml with the contents:

name: helloworld-tests
title: InSpec Profile
maintainer: Michael Joseph Walsh
copyright: Michael Joseph Walsh
copyright_email: [email protected]
license: New BSD License
summary: A minimal InSpec Compliance Profile for helloworld-web
version: 0.0.1
supports:
  platform: os
 attributes:
  - name: 'application_url'
    type: string
    default: 192.168.0.11
  - name: 'application_port'
    type: string

9.11.19.2. Execute your test

First, we'll execute the test against container running locally by first spinning up a 192.168.0.11:5000/nemonik/helloworld-web:latest container via

docker run -d -p 3000:3000 --name helloworld-web 192.168.0.11:5000/nemonik/helloworld-web:latest

If this command fails you likely already have a instance of the container running, you can kill the previous instance via

docker stop helloworld-web
docker rm helloworld-web

And then in the root of your helloworld-web project run the inspec test

inspec exec inspec.rb

Note

Remember to stop your helloworld-web docker container via
```
docker rm -f helloworld-web
```

9.11.19.3. The results

The result will be a comparsion of the expected state against the current state of the container and will look like

[vagrant@development inspec-tests]$ inspec exec inspec.rb

Profile: tests from inspec.rb (tests from inspec.rb)
Version: (not specified)
Target:  local://

  ✔  docker-checks-1.1: Verify Docker Container exists and is running
     ✔  Docker Container helloworld-web should exist
     ✔  Docker Container helloworld-web should be running
     ✔  Docker Container helloworld-web repo should eq "192.168.0.11:5000/nemonik/helloworld-web"
     ✔  Docker Container helloworld-web ports should eq "0.0.0.0:3000->3000/tcp"
     ✔  Docker Container helloworld-web command should eq "/app/helloworld-web"


Profile Summary: 1 successful control, 0 control failures, 0 controls skipped
Test Summary: 5 successful, 0 failures, 0 skipped

9.11.19.4. Add InSpec to the pipeline

Open in an editor .drone.yml and add the following step

- name: inspec
  image: appleboy/drone-ssh
  volumes:
  - name: private_key
    path: /root/ssh/drone_rsa
  settings:
    host: 192.168.0.11
    port: 22
    username: vagrant
    key_path: /root/ssh/drone_rsa
    command_timeout: 5m
    script:
      - cd /tmp
      - rm inspec.*
      - wget http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.rb
      - wget http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.yml
      - inspec exec inspec.rb

Then commit the your code

git add .
git commit -m "added inspec step to pipeline"
git push origin master

The output of the inspec step of the pipeline will resemble

======CMD======
cd /tmp
rm inspec.*
wget http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.rb
wget http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.yml
inspec exec inspec.rb
======END======
out: 
out: Profile: tests from inspec.rb (tests from inspec.rb)
out: Version: (not specified)
out: Target:  local://
out: 
out:   ✔  docker-checks-1.1: Verify Docker Container exists and is running
out:      ✔  Docker Container helloworld-web should exist
out:      ✔  Docker Container helloworld-web should be running
out:      ✔  Docker Container helloworld-web repo should eq "192.168.0.11:5000/nemonik/helloworld-web"
out:      ✔  Docker Container helloworld-web ports should eq "0.0.0.0:3000->3000/tcp"
out:      ✔  Docker Container helloworld-web command should eq "/app/helloworld-web"
out: 
out: 
out: Profile Summary: 1 successful control, 0 control failures, 0 controls skipped
out: Test Summary: 5 successful, 0 failures, 0 skipped
err: --2018-12-06 02:07:49--  http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.rb
err: Connecting to 192.168.0.11:10080... connected.
err: HTTP request sent, awaiting response... 200 OK
err: Length: 449 [text/plain]
err: Saving to: ‘inspec.rb’
err: 
err:      0K                                                       100% 73.3M=0s
err: 
err: 2018-12-06 02:07:50 (73.3 MB/s) - ‘inspec.rb’ saved [449/449]
err: 
err: --2018-12-06 02:07:50--  http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.yml
err: Connecting to 192.168.0.11:10080... connected.
err: HTTP request sent, awaiting response... 200 OK
err: Length: 416 [text/plain]
err: Saving to: ‘inspec.yml’
err: 
err:      0K                                                       100% 77.1M=0s
err: 
err: 2018-12-06 02:07:50 (77.1 MB/s) - ‘inspec.yml’ saved [416/416]
err: 
==========================================
Successfully executed commands to all host.
==========================================

9.11.19.5. Viewing the results in Heimdall-lite

MITRE maintains two projects for viewing of InSpec profiles and evaluations in a convenient interface.

We'll use Heimdall-lite, a single page JavaScript implementation of the MITRE Heimdall InSpec results viewer, to view:

In the Development vagrant, use inspec to report out to JSON via

inspec exec inspec.rb --reporter json > /vagrant/inspec_hellworld.json

Open Heimdall-lite https://mitre.github.io/heimdall-lite/, select Load JSON, the Browse to navigate to inspec_hellworld.json in the root of the class project and upload to view the results.

9.11.20. Add automated funtional test to pipeline

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline" #FFFFFF
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)

We're going to write an automated functional test of helloworld-web instead of relying on a manual functional test by using Selenium, a portable software-testing framework for web applications. Essentially, Selenium automates web browsers.

More can be found here

http://www.seleniumhq.org/

You'll need a couple of shells open to your development vagrant to complete this section.

9.11.20.1. Run the helloworld-web application

In the first vagrant ssh development enter

docker run -d -p 3000:3000 --name helloworld-web nemonik/helloworld-web:latest

NOTE

If you don't mind running more than a couple shells you can drop the -d argument and not daemonize the container, so you can watch its log.

9.11.20.2. Pull and run Selenium Firefox Standalone

In another vagrant ssh to development enter

curl -X GET http://192.168.0.11:5000/v2/_catalog
curl -X GET http://192.168.0.11:5000/v2/nemonik/standalone-firefox/tags/list
docker pull 192.168.0.11:5000/nemonik/standalone-firefox:3.14.0

to search for and pull then pull the nemonik/standalone-firefox:3.14.0 container image frome our private Docker repository running on our toolchain vagrant.

Then enter into the command line

docker run -p 4444:4444 192.168.0.11:5000/nemonik/standalone-firefox:3.14.0

This stands up Selenium specifically running Firefox. The container is running in the foreground so we can watch the log output.

A good start outputs to the command line

[vagrant@development ~]$ docker run -p 4444:4444 192.168.0.11:5000/nemonik/standalone-firefox:3.14.0
2018-12-01 08:15:45,146 WARN Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
2018-12-01 08:15:45,148 INFO supervisord started with pid 7
2018-12-01 08:15:46,154 INFO spawned: 'xvfb' with pid 10
2018-12-01 08:15:46,158 INFO spawned: 'selenium-standalone' with pid 11
08:15:46.576 INFO [GridLauncherV3.launch] - Selenium build info: version: '3.14.0', revision: 'aacccce0'
08:15:46.577 INFO [GridLauncherV3$1.launch] - Launching a standalone Selenium Server on port 4444
2018-12-01 08:15:46,578 INFO success: xvfb entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-12-01 08:15:46,578 INFO success: selenium-standalone entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-12-01 08:15:46.679:INFO::main: Logging initialized @487ms to org.seleniumhq.jetty9.util.log.StdErrLog
08:15:46.852 INFO [SeleniumServer.boot] - Selenium Server is up and running on port 4444

9.11.20.3. Create our test automation

In another vagrant ssh to development, we author and run our automated test.

In this other command line at the root of the helloworld-web project (e.g., /home/vagrant/go/src/github.com/nemonik/helloworld-web) enter

mkdir selenium-test
cd selenium-test

We're going to write our test in Python. Python is already installed on the development vagrant. Ansible uses it.

We'll need to install a dependency, we can install by entering into the command line

 sudo pip install 'selenium==3.14.0'

The output of looks like

[vagrant@development ~]$ sudo pip install 'selenium==3.14.0'
Collecting selenium==3.14.0
  Downloading https://files.pythonhosted.org/packages/b8/53/9cafbb616d20c7624ff31bcabd82e5cc9823206267664e68aa8acdde4629/selenium-3.14.0-py2.py3-none-any.whl (898kB)
    100% |████████████████████████████████| 901kB 3.1MB/s
Requirement already satisfied: urllib3 in /usr/lib/python2.7/site-packages (from selenium==3.14.0) (1.24.1)
Installing collected packages: selenium
Successfully installed selenium-3.14.0

But let's create a requirements.txt file to hold our dependency with our text editor containing

selenium==3.14.0

This allow the test's dependencies to be enumerated in a file including the specififc version of the dependency, so their installation can be automated.

Then test out the requirements.txt file by entering

sudo pip install -r requirements.txt

Then in an editor create test_helloworld.py Python-based unit test containing

# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>

import unittest
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import sys
import os

class HelloworldWebTest(unittest.TestCase):

    def setUp(self):

        webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
            "proxyType":"DIRECT"
#                "noProxy":"os.environ['no_proxy']," + selenium_host # Selenium cannot handle no_proxy
    }

        self.driver = webdriver.Remote(command_executor="http://" + selenium_host + ":4444/wd/hub", desired_capabilities=webdriver.DesiredCapabilities.FIREFOX)

    def test_helloworld_web(self):
        driver = self.driver
        driver.get(helloworld_web_url)
        assert "Hello world!" in driver.page_source

    def tearDown(self):
        self.driver.close()

if __name__ == "__main__":
    if len(sys.argv) == 3:
        selenium_host = sys.argv[1]
        helloworld_web_url = sys.argv[2]
    else:
        print "python test_hello_world.py selenium_host helloworld_web_url"
        sys.exit()

    del sys.argv[1:]
    unittest.main()

Save the file and exit.

You might notice the commented out section about Selenium cannot handle no_proxy. Yeah. This will later present a problem.

For now, now let us run our test by entering into the command line

python test_helloworld.py 192.168.0.10 http://192.168.0.10:3000

Successful command line output in this window will be

[vagrant@development selenium-test]$ python test_helloworld.py 192.168.0.10 http://192.168.0.10:3000
.
----------------------------------------------------------------------
Ran 1 test in 1.699s

OK

The other windows will show logging.

ctrl-c to stop the containers in your other shells.

NOTE

If you pushed the helloword-web to the background, you'll have to remove it via
```
docker rm -f helloworld-web
```

9.11.20.4. Add a selenium step to the pipeline

In one of your existing vagrant ssh to development enter into the commmand line

cd /home/vagrant/go/src/github.com/nemonik/helloworld-web

Edit the .drone.yml file and add to the bottom the following with the selenum: step having the same idention as the prior steps, and services: starting the line.

- name: selenium
  image: 192.168.0.11:5000/nemonik/python:2.7.16
  commands:
  - export NO_PROXY=$NO_PROXY,$(python selenium-test/resolve.py firefox)
  - export no_proxy=$no_proxy,$(python selenium-test/resolve.py firefox)
  - cd selenium-test
  - pip install -r requirements.txt
  - python test_helloworld.py firefox http://192.168.0.11:3000

volumes:
- name: private_key
  host:
    path: /vagrant/.vagrant/machines/toolchain/virtualbox/private_key
- name: shared_memory
  host:
    path: /dev/shm

services:
- name: firefox
  image: 192.168.0.11:5000/nemonik/standalone-firefox:3.14.0
  volumes:
  - name: shared_memory
    path: /dev/shm

NOTE

services is not part of the prior steps, because it is not a step, but enumeration of services, so be careful when you edit the pipeline.

Drone uses the services: section to spin up a patched version of selenium/standalone-firefox:3.14.0 exposed with the name firefox. This is where the problem creeps in that I mentioned earlier, where the selenium step will hit corporate proxy to resolve firefox. We don't want that to happen, so the selenium: step will makes use of another Python script to resolved firefox service to its private IP and add it the no_proxy and NO_PROXY environmental variables, so that the Python selenium test code doesn't attempt to pass the request on to the corporate proxy and result in the test failing.

Create in a text editor selenium-test/resolve.py with

#!/usr/bin/env python

# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>

import socket, sys

if __name__ == "__main__":
    if len(sys.argv) == 2:
        print socket.gethostbyname( sys.argv[1] )

Commit the code to GitLab hosted remote repo

git add .
git commit -m "added selenium step to pipeline"
git push origin master

Open on your host

http://192.168.0.11/root/helloworld-web

And monitor the progress of the build. The pipeline should execute in a few minutes.

Successful output for this stage resembles

+ export NO_PROXY=$NO_PROXY,$(python selenium-test/resolve.py firefox)
+ export no_proxy=$no_proxy,$(python selenium-test/resolve.py firefox)
+ cd selenium-test
+ pip install -r requirements.txt
Collecting selenium==3.14.0 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/b8/53/9cafbb616d20c7624ff31bcabd82e5cc9823206267664e68aa8acdde4629/selenium-3.14.0-py2.py3-none-any.whl (898kB)
Collecting urllib3 (from selenium==3.14.0->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
Installing collected packages: urllib3, selenium
Successfully installed selenium-3.14.0 urllib3-1.24.1
+ python test_helloworld.py firefox http://192.168.0.11:3000
.
----------------------------------------------------------------------
Ran 1 test in 5.283s

OK

9.11.21. Add DAST step (OWASP ZAP) to pipeline

skinparam shadowing false

skinparam title {
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 30
}

skinparam activity {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityDiamond {
  BorderColor #0B5C92
  BackgroundColor #e0e59a
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

skinparam activityStart {
  Color #0B5C92
}

skinparam activityEnd {
  Color #0B5C92
}

skinparam arrow {
  Color #0B5C92
}

skinparam note {
  BorderColor #0B5C92
  BackgroundColor #FEFECE
  FontName "Yanone Kaffeesatz"
  FontStyle "Thin"
  FontSize 15
}

(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline" #FFFFFF
-left-> (*)

Dynamic application security testing (DAST) is used to detect security vulnerabilities in an application while it is running, so as to help you remediate these concerns while in development.

The OWASP Zed Attack Proxy (ZAP) is one of the world’s most popular free DAST tools actively maintained by hundreds of international volunteers, so add a step to test the application.

First exercise ZAP locally opening vagrant ssh to development at the root of the project

docker pull 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0
docker run 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0 zap-baseline.py -t http://192.168.0.11:3000

Success in the command line resembles the following for the docker pull.

[vagrant@development helloworld-web]$ docker pull 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0
2.7.0: Pulling from nemonik/zap2docker-stable
660c48dd555d: Pull complete
4c7380416e78: Pull complete
421e436b5f80: Pull complete
e4ce6c3651b3: Pull complete
be588e74bd34: Pull complete
2e09786d5b2f: Pull complete
197722f79cca: Pull complete
bd33ded2095e: Pull complete
4b06ea4ab95b: Pull complete
5549c936d696: Pull complete
c2bfc8c8addb: Pull complete
1a4b29516a99: Pull complete
8ce1a9f9c46c: Pull complete
8808d1ac4805: Pull complete
24eed6f4cce3: Pull complete
2cd87b08c44e: Pull complete
5ae7b6620997: Pull complete
d6d847cf31e3: Pull complete
1bd5a2ca7470: Pull complete
737316e32acc: Pull complete
6379bbb7b3df: Pull complete
Digest: sha256:6787b975227b4e559fc029e48c0fbf3181ef2567abaae8c5b60b50d5948de9ec
Status: Downloaded newer image for 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0

And the following for the docker run

[vagrant@development helloworld-web]$ docker run 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0 zap-baseline.py -t http://192.168.0.11:3000

Dec 01, 2018 8:52:48 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Total of 3 URLs
PASS: Cookie No HttpOnly Flag [10010]
PASS: Cookie Without Secure Flag [10011]
PASS: Password Autocomplete in Browser [10012]
PASS: Incomplete or No Cache-control and Pragma HTTP Header Set [10015]
PASS: Web Browser XSS Protection Not Enabled [10016]
PASS: Cross-Domain JavaScript Source File Inclusion [10017]
PASS: Content-Type Header Missing [10019]
PASS: X-Frame-Options Header Scanner [10020]
PASS: Secure Pages Include Mixed Content [10040]
PASS: Private IP Disclosure [2]
PASS: Session ID in URL Rewrite [3]
PASS: Script Passive Scan Rules [50001]
PASS: Application Error Disclosure [90022]
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 4
        http://192.168.0.11:3000/
        http://192.168.0.11:3000/robots.txt
        http://192.168.0.11:3000/sitemap.xml
        http://192.168.0.11:3000
FAIL-NEW: 0     FAIL-INPROG: 0  WARN-NEW: 1     WARN-INPROG: 0  INFO: 0 IGNORE: 0       PASS: 13

NOTE

OWASP ZAP is not the fastest tool, so give it time.

Great, now lets add another step to our pipeline after the selenium step, but before services:.

- name: owasp-zaproxy
  image: 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0
  commands:
  - zap-baseline.py -t http://192.168.0.11:3000 || true

Commit the code to GitLab hosted remote repo

git add .
git commit -m "added owasp-zaproxy step to pipeline"
git push origin master

Open on your host

http://192.168.0.11/root/helloworld-web

And monitor the progress of the build. The pipeline should execute in a few minutes.

Successful output for this stage resembles

+ zap-baseline.py -t http://192.168.0.11:3000 || true
Apr 03, 2019 2:39:30 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Total of 3 URLs
PASS: Cookie No HttpOnly Flag [10010]
PASS: Cookie Without Secure Flag [10011]
PASS: Password Autocomplete in Browser [10012]
PASS: Incomplete or No Cache-control and Pragma HTTP Header Set [10015]
PASS: Web Browser XSS Protection Not Enabled [10016]
PASS: Cross-Domain JavaScript Source File Inclusion [10017]
PASS: Content-Type Header Missing [10019]
PASS: X-Frame-Options Header Scanner [10020]
PASS: Secure Pages Include Mixed Content [10040]
PASS: Private IP Disclosure [2]
PASS: Session ID in URL Rewrite [3]
PASS: Script Passive Scan Rules [50001]
PASS: Application Error Disclosure [90022]
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 4
http://192.168.0.11:3000/
http://192.168.0.11:3000/robots.txt
http://192.168.0.11:3000/sitemap.xml
http://192.168.0.11:3000
FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 1	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 13

The application is relatively simple, so it was doubtful anything would be found, but there is a warning that a X-Content-Type-Options Header is missing that could be resolved by adding addtional handlers to the helloworld-web's main.go.

9.11.22. All the source for helloworld-web

The helloworld-web project can be viewed completed at

https://github.com/nemonik/helloworld-web

9.12. Microservices

So, you may not realize it, but with the helloworld-web application you've actually written a microservice.

What's a microservice?

Microservices perform a single function (i.e., a single business capability or feature.) In our case, our application's single function is to return, "Helloworld!"
Communicate with other services via well-defined APIs.
Can be written using different frameworks and programming languages.
Can be deployed independently or as a group.

The benefit

Traditional monolithic architectures don’t lend themselves to cloud deployments.
- Nor do they scale.
- They’re simple not “cloud-native.” (More on that word later.)
In monolithic architecture, if a single feature experiences a demand, the entire architecture must be scaled.
In Microservice Architecture each feature runs as a separate service within its own container.
Thereby, permitting each service to be scaled and maintained independently.
Providing crash Isolation, so if a single piece of your service is crashing, then the rest of your application continues to work
Providing Isolation for Security, so a compromise of one service only gains the adversary access to the resources of that single service
Providing independent scaling, so each microservice’s compute utilization can be independently scaled up and down
Increasing development velocity, lowering development risks thereby permitting teams to build faster in comparison to monolithic development.

So, with the helloworld-web application, you've developed a micrsoservice, packaged it into a container (i.e., containerized it), and by doing so essentially created a cloud-native application.

9.12.1. What's cloud-native?

Cloud-native computing deploys applications as microservices, packaging each into its own container, and by doing so have opened yourself up to the capability of being able to dynamically orchestrate the container to optimize resource utilization, elastic scaling, security issolation...

Granted our microservice doesn't do much, but we can get a peak of the possibilities without even installing a full blown Container Platform, such as, Kubernetes, Docker Swarm, or OpenShift.

9.12.1.1. So, let's experiment with our little "microservice"

On the development vagrant

docker swarm init --advertise-addr=192.168.0.10

Edit the go/src/github.com/nemonik/helloworld-web/main.go

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
)

var hostname string

func main() {
	hostname, _ = os.Hostname()
	http.HandleFunc("/", handler)
	fmt.Print("listening on :3000\n")
	log.Fatal(http.ListenAndServe(":3000", logRequest(http.DefaultServeMux)))
}

func handler(w http.ResponseWriter, r *http.Request) {
	_, err := fmt.Fprintf(w, "Hello world! %s\n", hostname)
	if err != nil {
		log.Fatal(err)
	}
}

func logRequest(handler http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Printf("%s  %s  %s\n", r.RemoteAddr, r.Method, r.URL)
		handler.ServeHTTP(w, r)
	})
}

Changes made:

Added os package.
Added the hostname global variable.
Set hostname global in main().
Append hostname to output of the handler.

Why these changes were made will become apparent.

At the root of the hellworld-web application create a new file named docker-compose.yml and add to it the following content

version: "3"

services:
  helloworld:
    build: .
    image: nemonik/helloworld-web:latest
    deploy:
      replicas: 1
      resources:
        limits:
          cpus: "0.1"
          memory: 4M
      restart_policy:
        condition: on-failure
    ports:
      - "3000:3000"
    networks:
      - web
networks:
  web:

The docker-compose.yml file tells Docker to do the following:

Use the nemonik/helloworld-web:latest Docker image.
Run 1 instance of that image as a service called service, limiting each replica to at most 10% of the CPU and 4MB of RAM (the smallest amount one can in this instance).
Restart immediately if container fails.
Share port 3000 via a load-balanced network called web.
All running on the load-balanced overlay web network.

To spin up our app

docker-compose build
docker stack deploy -c docker-compose.yml helloworld

The output will resemeble

WARNING: Some services (helloworld) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
Building helloworld
Step 1/8 : FROM 192.168.0.11:5000/nemonik/golang:1.12.6
 ---> 6aa2d9de5f78
Step 2/8 : MAINTAINER Michael Joseph Walsh <[email protected]>
 ---> Using cache
 ---> 17abd315c43e
Step 3/8 : RUN mkdir /app
 ---> Using cache
 ---> 3ae5aef96fce
Step 4/8 : ADD main.go /app/
 ---> Using cache
 ---> dffac9d00543
Step 5/8 : WORKDIR /app
 ---> Using cache
 ---> b1e5d8c118fb
Step 6/8 : RUN go build -o helloworld-web .
 ---> Using cache
 ---> a8d30237b993
Step 7/8 : CMD ["/app/helloworld-web"]
 ---> Using cache
 ---> b2668738c4ea
Step 8/8 : EXPOSE 3000
 ---> Using cache
 ---> e999d0f3737d
Successfully built e999d0f3737d
Successfully tagged nemonik/helloworld-web:latest
Ignoring unsupported options: build

Creating network helloworld_web
Creating service helloworld_helloworld

Now list the services Docker is running via

docker service list

The output will look like

ID                  NAME                    MODE                REPLICAS            IMAGE                           PORTS
mgra793avso7        helloworld_helloworld   replicated          1/1                 nemonik/helloworld-web:latest   *:3000->3000/tcp

Now test your service

[vagrant@development helloworld-web]$ curl http://192.168.0.10:3000

The output will look like

Hello world! b6835470176e

That random hex number is the result of our code change. It is the hostname of the container that responded to your request.

Now lets scale from 1 to 5 replicas via

docker service scale helloworld_helloworld=5

The output will resemble

helloworld_helloworld scaled to 5
overall progress: 5 out of 5 tasks
1/5: running
2/5: running
3/5: running
4/5: running
5/5: running
verify: Service converged

Re-running

docker service list

Will also show we've scale to 5 replicas.

ID                  NAME                    MODE                REPLICAS            IMAGE                           PORTS
mgra793avso7        helloworld_helloworld   replicated          5/5                 nemonik/helloworld-web:latest   *:3000->3000/tcp

Now run curl several times via

for i in {1..10}
do
curl http://192.168.0.10:3000
done

The output will resemble

Hello world! 6a0e64e07f27
Hello world! 899d588cae34
Hello world! 3fd46049e8bb
Hello world! 743630bf2f8b
Hello world! b6835470176e
Hello world! 6a0e64e07f27
Hello world! 899d588cae34
Hello world! 3fd46049e8bb
Hello world! 743630bf2f8b
Hello world! b6835470176e

Notice how the curl requests are Round-robin load balanced across each replica container. Typically, these containers would be spread across a container platform's cluster thereby permiting you to scale your application to accommodate shifting capacity needs. In other words, to be become elastic. Further, the approach permits iterative software development and deployment (thereby benefitting from DevOps methods and repeated practices), isolation, and resiliency.

9.13. Using what you've learned

Git clone the Python+Flask Magic Eight Ball web app

https://github.com/nemonik/magiceightball

and develop a pipeline for.

9.14. Shoo away your vagrants

If you're done with your vagrants, shoo them away from the root of the project on the host

vagrant destroy -f

And they're gone.

9.15. That's it

That's a wrap.

jorymorrison / hands-on-devops Goto Github PK

hands-on-devops's Introduction

1. DevOps

2. Author

3. Thank you to

4. Copyright and license

5. Prerequisites

6. Table of contents

7. DevOps unpacked

7.1. What is DevOps?

7.2. What DevOps is not

7.3. DevOps is really about

7.4. How is it related to the Agile?

7.5. How do they differ?

7.6. Why?

7.7. What are the principles of DevOps?

7.8. How is this achieved?

7.9. What is Continuous Integration (CI)?

7.10. How?

7.11. CI best practices

7.11.1. Utilize a Configuration Management System

7.11.2. Automate the build

7.11.3. Employ one or more CI services/orchestrators

7.11.4. Make builds self-testing

7.11.5. Never commit broken

7.11.6. Developers are expected to pre-flight new code

7.11.7. The CI service/orchestrator provides feedback

7.12. What is Continuous Delivery?

7.12.1. Extending Continuous Integration (CI)

7.12.2. Consistency

7.13. But wait. What's a pipeline?

7.14. How is a pipeline manifested?

7.15. What underlines all of this?

7.16. But really why do we automate err. code?

7.16.1. Why do I mention Larry Wall?

7.16.2. Laziness

7.16.3. Impatience

7.16.4. Hubris

7.16.5. We automate for

7.17. Monitoring

7.17.1. The most important metric

7.17.2. An understanding of performance

7.17.3. Establish a baseline performance

7.17.4. Set reaction thresholds

7.17.5. Reacting

7.17.6. Gaps in CI/CD

7.17.7. Eliminating waste

7.18. What is DevOps culture?

7.18.1. We were taught the requisite skills as children

7.18.2. Maintaining relationships is your most important skill

7.18.3. Be quick... Be slow to...

7.18.4. The pressures of social media

7.19. Crawl, walk, run

7.19.1. Ultimately, DevOps is Goal

8. Reading list

9. Now the hands-on part

9.1. Configuring environmental variables

9.2. VirtualBox

9.2.1. Installing VirtualBox

9.3. Git Bash

9.3.1. Installing Git Bash

9.4. Retrieve the course material

9.5. Infrastructure as code (IaC)

9.5.1. Vagrant

9.5.1.1. Vagrant documentation and source

9.5.1.2. Installing Vagrant

9.5.1.3. Installing Vagrant plugins

9.5.1.4. The Vagrantfile explained

9.5.1.4.1. Modelines

9.5.1.4.2. Setting Software version for Ansible roles

9.5.1.4.3. Inserting Proxy setting via host environmental variables

9.5.1.4.4. Inserting enterprise CA certificates

9.5.1.4.5. Configuring the cache plugin to speed things along

9.5.1.4.6. Configuring the disksize plugin to increase the disk size

9.5.1.4.7. Requiring vagrant-vbquest for Windows

9.5.1.4.8. Configuring the development vagrant

9.5.1.4.9. Configuring the toolchain vagrant

9.5.2. Ansible

9.5.2.1. Inventory file

9.5.2.2. Playbooks

9.8.4.3. Optionally, add the hands-on-DevOps repository to the `toolchain`'s GitLab