dnaeon / gru

Orchestration made easy with Go and Lua

Gru - Orchestration made easy with Go and Lua

Gru is a fast and concurrent orchestration framework powered by Go and Lua, which allows you to manage your UNIX/Linux systems with ease.

Documentation

You can find the latest documentation here.

The API documentation is available here.

Features

• Written in fast, compiled language - Go
• Uses a fast, lightweight, embeddable, scripting language as the DSL - Lua
• Concurrent execution of idempotent operations
• Distributed - using etcd for node discovery and communication and Git for version control and data sync
• Easy to deploy - comes with a single, statically linked binary
• Suitable for orchestration and configuration management

Status

Gru is in constant development. Consider the API unstable as things may change without a notice.

Contributions

Gru is hosted on Github. Please contribute by reporting issues, suggesting features or by sending patches using pull requests.

License

Gru is Open Source and licensed under the BSD License.

References

References to articles related to this project in one way or another.

• Managing VMware vSphere environment with Go and Lua by using Gru orchestration framework
• Introducing triggers in Gru orchestration framework
• Puppet vs Gru - Benchmarking Speed & Concurrency
• Extending Lua with Go types
• Choosing Lua as the data description and configuration language
• Creating an orchestration framework in Go
• Dependency graph resolution algorithm in Go
• Orchestration made easy with Gru v0.2.0
• Membership test in Go
• Testing HTTP interactions in Go
• Concurrent map and slice types in Go
• Lua as a Configuration And Data Exchange Language

gru's Issues

Be more flexible when managing services on a FreeBSD system

@kaey

I've been checking the resource for service management on FreeBSD systems, and thought that we could perhaps make it even better.

One thing that I'd like to do is get rid of the logic for sendmail service, and have the resource be more flexible, so that we don't have to hard-code all services, which don't use YES and NO as the rcvalues.

Here's a diff I came up with.

diff --git a/resource/service_freebsd.go b/resource/service_freebsd.go
index 7bb82e8..0bbf84a 100644
--- a/resource/service_freebsd.go
+++ b/resource/service_freebsd.go
@@ -24,6 +24,16 @@ type Service struct {
    // RCVar (see rc.subr(8)), set to {svcname}_enable by default.
    // If service doesn't define rcvar, you should set svc.rcvar = "".
    RCVar string `luar:"rcvar"`
+
+   // RCValue is the value used when setting the rcvar.
+   // If no value is provided, then the default ones will be used,
+   // which are "YES" and "NO" when a service is respectively
+   // being enabled or disabled.
+   //
+   // Use this field only when the service does not use the
+   // default values for rcvar, for example the "sendmail"
+   // service for which you should use "NONE" as the value.
+   RCValue string `luar:"rcvalue"`
 }

 // NewService creates a new resource for managing services
@@ -98,18 +108,18 @@ func (s *Service) Update() error {
        return nil
    }

-   rcValue := "YES"
-   if !s.Enable {
-       rcValue = "NO"
-       if s.RCVar == "sendmail_enable" {
-           // I think sendmail is the only service, that requires NONE to be disabled.
-           rcValue = "NONE"
+   // Pick a default value, if we don't have one already
+   if s.RCValue == "" {
+       switch s.Enable {
+       case true:
+           s.RCValue = "YES"
+       case false:
+           s.RCValue = "NO"
        }
    }

    // TODO: rcvar should probably be deleted from rc.conf, when disabling service (except for sendmail).
-   // Currently we set it to NO.
-   return exec.Command("sysrc", fmt.Sprintf(`%s="%s"`, s.RCVar, rcValue)).Run()
+   return exec.Command("sysrc", fmt.Sprintf(`%s="%s"`, s.RCVar, s.RCValue)).Run()
 }

 func init() {

@kaey what do you think about this one?

Thanks,
Marin

Create base resource interface

Currently Gru is able to send tasks to remote minions, which once they receive the task simply execute a command.

We should take it further and implement a base resource type, which would give us a layer of abstraction on the commands/tasks being executed.

This would allow for creating resources such as package, file, service, etc. which would manage things in idempotent manner.

Add supervisord configuration for running Gru services

We already have systemd units for Gru. Add supervisord configuration as well.

Add support for Vault

Add support for Vault so that we can use it to retrieve sensitive information from Lua when creating resources.

Resource namespaces in Lua

Implement resource namespaces in Lua

Use lua as the configuration and data language for Gru modules

This issue is to evaluate whether lua might be a good fit for the configuration and data language for Gru modules.

Create File resource

Implement resource for managing files, e.g.

resource "file" {
  path = "$HOME/.tmux.conf"
  owner = "dnaeon"
  group = "dnaeon"
}

Docs: Create the Quickstart guide

DSL: Importing of modules

Be able to use imports in the HCL-based DSL language used to declare resources.

This would make it possible to create modular catalogs and allow for code re-use.

Support for file templates

Implement support for file templates.

gructl: add command for displaying registered resource providers

Implement meta resource for package management

Instead of having to use pacman, yum, apt, etc. directly we should implement the meta resource package, which can install the package using the correct provider.

new release with most recent changes?

Last release is from 2017, but there have been more recent changes since, which address various bugs & new functionality. Perhaps makes sense to make a new release to avoid go getting master directly?

Export HCL resources to JSON

Implement resource for managing directories

Create Shell resource

Implement a resource for executing shell commands.

Example resource declaration:

resource "shell" {
  name = "echoes foo"
  command = "echo foo"
}

Implement resource for managing packages with Yum

Resource catalog

Use a resource catalog for all resources loaded from the given input and create a DAG graph to determine the proper order of resource evaluation.

Display report of applied resources

Add summary of resources changed/created/deleted.

Implement resource for managing packages with apt-get

gructl: Add "apply" command

Implement a new sub-command "apply" for applying an HCL-based Gru module on the local system.

Shell completions

Enable shell completions and document how to activate them.

Add sub-command for validating HCL configurations for Gru

Error while building gru

Hi,

I'm getting the output below, while trying to build gru after downloading it from github. Could you please take a look ?

root@pc:/home/user/go/gru#

ls
catalog classifier client contrib docs graph gructl integration LICENSE main.go Makefile minion README.md resource site task utils version
root@pc:/home/user/go/gru# make
go get -v -t -d ./...
github.com/dnaeon/gru (download)
created GOPATH=/root/go; see 'go help gopath'
github.com/coreos/etcd (download)
github.com/coreos/go-semver (download)
github.com/ugorji/go (download)
github.com/deckarep/golang-set (download)
github.com/blang/semver (download)
github.com/coreos/go-systemd (download)
github.com/godbus/dbus (download)
github.com/coreos/pkg (download)
github.com/pborman/uuid (download)
github.com/vmware/govmomi (download)
github.com/yuin/gopher-lua (download)
Fetching https://layeh.com/gopher-luar?go-get=1
Parsing meta tags from https://layeh.com/gopher-luar?go-get=1 (status code 200)
get "layeh.com/gopher-luar": found meta tag get.metaImport{Prefix:"layeh.com/gopher-luar", VCS:"git", RepoRoot:"https://github.com/layeh/gopher-luar"} at https://layeh.com/gopher-luar?go-get=1
layeh.com/gopher-luar (download)
github.com/dnaeon/backoff (download)
github.com/gosuri/uiprogress (download)
github.com/gosuri/uilive (download)
github.com/gosuri/uitable (download)
github.com/mattn/go-runewidth (download)
github.com/urfave/cli (download)
github.com/dnaeon/go-vcr (download)
Fetching https://gopkg.in/yaml.v2?go-get=1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag get.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
gopkg.in/yaml.v2 (download)
go build -ldflags="-s -w" -o bin/gructl -v
github.com/coreos/etcd/pkg/pathutil
github.com/coreos/etcd/pkg/types
github.com/coreos/go-semver/semver
github.com/ugorji/go/codec
github.com/deckarep/golang-set
github.com/coreos/etcd/version
github.com/blang/semver
github.com/coreos/etcd/pkg/srv
github.com/godbus/dbus
github.com/dnaeon/gru/graph
github.com/coreos/pkg/dlopen
github.com/pborman/uuid
github.com/dnaeon/gru/utils
github.com/vmware/govmomi/vim25/debug
github.com/vmware/govmomi/vim25/progress
github.com/vmware/govmomi/vim25/types
github.com/coreos/go-systemd/util
github.com/coreos/go-systemd/dbus
github.com/vmware/govmomi/vim25/xml
github.com/yuin/gopher-lua/ast
github.com/yuin/gopher-lua/parse
github.com/yuin/gopher-lua/pm
github.com/yuin/gopher-lua
github.com/dnaeon/gru/classifier
github.com/dnaeon/backoff
github.com/dnaeon/gru/task
github.com/gosuri/uilive
github.com/gosuri/uiprogress/util/strutil
github.com/gosuri/uiprogress
github.com/mattn/go-runewidth
github.com/gosuri/uitable/util/strutil
github.com/gosuri/uitable/util/wordwrap
github.com/gosuri/uitable
github.com/urfave/cli
github.com/dnaeon/gru/version
layeh.com/gopher-luar
github.com/coreos/etcd/client
github.com/vmware/govmomi/vim25/soap
github.com/vmware/govmomi/vim25/methods
github.com/vmware/govmomi/vim25
github.com/vmware/govmomi/vim25/mo
github.com/vmware/govmomi/property
github.com/vmware/govmomi/session
github.com/vmware/govmomi/list
github.com/vmware/govmomi/nfc
github.com/vmware/govmomi/task
github.com/vmware/govmomi
github.com/vmware/govmomi/object
github.com/vmware/govmomi/find
github.com/dnaeon/gru/resource

github.com/dnaeon/gru/resource

/root/go/src/github.com/dnaeon/gru/resource/vsphere_cluster.go:134:18: obj.ReconfigureCluster undefined (type *object.ClusterComputeResource has no field or method ReconfigureCluster)
Makefile:4: recipe for target 'build' failed
make: *** [build] Error 2

Create Package resource

Implement a resource for managing packages, e.g.

resource "package" {
  name = "tmux"
  state = "installed"
}

Document how to use triggers

Create a documentation describing how to use triggers in resources.

Implement resource for editing configuration files

Add resource for editing configuration files using augeas

Implement resource for managing users and groups

Implement resource for managing mount points

Generate DOT graph of the resource dependency tree

Be able to generate the resources dependency tree in DOT language:

• https://en.wikipedia.org/wiki/DOT_(graph_description_language)

Resource dependencies

Implement resource dependencies and resolve the dependency graph, e.g. service X -> package X.

This would make it possible to define the order of how resources are being evaluated and processed.

Implement resource for managing cron jobs

Create Service resource

Implement resource for managing services, e.g.

resource "service" {
 name = "nginx"
 state = "running"
 enable = true
}

Syncing of modules and data to minions

We do want to have a way to sync modules and data files to remote minions.

That way we can also run minions in stand-alone mode if needed, but would allow minions to continue running in case we get disconnected.

Implement Lua REPL for Gru

Add support for Consul

Concurrent processing of resources

Investigate possibilities to process resources concurrently which have no dependencies

Resource for downloading files

Implement a resource that downloads files.

Create a Timer resource

Would be nice to have a resource which can run commands or evaluate other resources periodically.

Something like a time.Ticker.

Add support for plugins

Implement resources for managing VMware vSphere environments

Create resources for managing Datacenters, Clusters, Hosts, Datastores and VMs as a start.

Implement resource for managing virtual machines

A good start for this would be to probably use https://github.com/apcera/libretto

Implement resource for symlinks

Implement resource triggers

When a resource state changes trigger a refresh of other resources.

Incorporate HIL into Gru

https://github.com/hashicorp/hil

Implement DSL for Gru

Good candidate for a DSL language would be to use the HashiCorp HCL config language.

For example:

resource "package" {
  name = "tmux"
  state = "installed"
}

https://en.wikipedia.org/wiki/Domain-specific_language