reconquest / orgalorg

Parallel SSH commands runner and file synchronization tool

orgalorg

orgalorg can run command and upload files in parallel by SSH on many hosts

Features

• Zero-configuration. No config files. Everything is done via command line flags.

• Running SSH commands or shell scripts on any number of hosts in parallel. All output from nodes will be returned back, keeping stdout and stderr streams mapping of original commands.

• Synchronizing files and directories across cluster with prior global cluster locking. After synchronization is done, arbitrary command can be evaluated.

• Synchronizing files and directories with subsequent run of complex multi-step scenario with steps synchronization across cluster.

• User-friendly progress indication.

• Both strict or loose modes of failover to be sure that everything will either fail on any error or try to complete, no matter of what.

• Interactive password authentication as well as SSH public key authentication. Will use ssh-agent if present. On Windows, orgalorg can connect to pageant or openssh agent.

• Ability to run commands through sudo.

• Grouped mode of output, so stdout and stderr from nodes will be grouped by node name. Alternatively, output can be returned as soon as node returns something.

Installation

go get

go get github.com/reconquest/orgalorg

Alternatives

• ansible: intended to apply complex DSL-based scenarios of actions; orgalorg aimed only on running commands and synchronizing files in parallel. orgalorg can accept target hosts list on stdin and can provide realtime output from commands, which ansible can't do (like running tail -f). orgalorg also uses same argument semantic as ssh: orgalorg ... -C tail -f '/var/log/*.log' will do exactly the same.

• clusterssh / cssh: will open number of xterm terminals to all nodes. orgalorg intended to use in batch mode, no GUI is assumed. orgalorg, however, can be used in interactive mode (see example section below).

• pssh: buggy, uses binary ssh, which is not resource efficient. orgalorg uses native SSH protocol implementation, so safe and fast to use on thousand of nodes.

• dsh / gsh / pdsh: not maintained.

Example usages

-o <host>... in later examples will mean any supported combination of host-specification arguments, like -o node1.example.com -o node2.example.com.

Evaluating command on hosts in parallel

orgalorg -o <host>... -C uptime

Evaluating command on hosts given by stdin

axfr is a tool of your choice for retrieving domain information from your infrastructure DNS.

axfr | grep phpnode | orgalorg -s -C uptime

Evaluate command under root (passwordless sudo required)

orgalorg -o <host>... -x -C whoami

Tailing logs from many hosts in realtime

orgalorg -o <host>... -C tail -f /var/log/syslog

Copying SSH public key for remote authentication

orgalorg -o <host>... -p -i ~/.ssh/id_rsa.pub -C tee -a ~/.ssh/authorized_keys

Synchronizing configs and then reloading service (like nginx)

orgalorg -o <host>... -xn 'systemctl reload nginx' -S /etc/nginx.conf

Evaluating shell script

orgalorg -o <host>... -i script.bash -C bash

Install package on all nodes and get combined output from each node

orgalorg -o <host>... -lx -C pacman -Sy my-package --noconfirm

Evaluating shell oneliner

orgalorg -o <host>... -C sleep '$(($RANDOM % 10))' '&&' echo done

Running poor-man interactive parallel shell

orgalorg -o <host>... -i /dev/stdin -C bash -s

Obtaining global cluster lock

orgalorg -o <host>... -L

Next orgalorg calls will fail with message, that lock is already acquired, until first instance will be stopped.

Useful for setting cluster into maintenance state.

Obtaining global cluster lock on custom directory

orgalorg -o <host>... -L -r /etc

Description

orgalorg provides easy way of synchronizing files across cluster and running arbitrary SSH commands.

orgalorg works through SSH & tar, so no unexpected protocol errors will arise.

In default mode of operation (lately referred as sync mode) orgalorg will perform steps in the following order:

1. Acquire global cluster lock (check more detailed info above).
2. Create, upload and extract specified files in streaming mode to the specified nodes into temporary run directory.
3. Start synchronization tool on each node, that should relocate files from temporary run directory to the destination.

So, orgalorg expected to work with third-party synchronization tool, that will do actual files relocation and can be quite intricate, but orgalorg can work without that tool and perform simple files sync (more on this later).

Global Cluster Lock

Before doing anything else orgalorg will perform global cluster lock. That lock is acquired atomically, and no other orgalorg instance can acquire lock if it is already acquired.

Locking is done via flock'ing specified file or directory on each of target nodes, and will fail, if flock fails on at least one node.

Directory can be used as lock target as well as ordinary file. --lock-file can be used to specify lock target different from /.

After acquiring lock, orgalorg will run heartbeat process, which will check, that lock is still intact. By default, that check will be performed every 10 seconds. If at least one heartbeat is failed, then orgalorg will abort entire sync procedure.

User can stop there by using --lock or -L flag, effectively transform orgalorg to the distributed locking tool.

File Upload

Files will be sent from local node to the amount of specified nodes.

orgalorg will perform streaming transfer, so it's safe to synchronize large files without major memory consumption.

By default, orgalorg will upload files to the temporary run directory. That behaviour can be changed by using --root or -r flag. Then, files will be uploaded to the specified directory.

User can specify --upload or -U flag to transform orgalorg to the simple file upload tool. In that mode orgalorg will upload files to the specified directory and then exit.

orgalorg preserves all file attributes while transfer as well as user and group IDs. That behaviour can be changed by using --no-preserve-uid and --no-preseve-gid command line options. These flags are ignored when orgalorg is ran from Windows.

By default, orgalorg will keep source file paths as is, creating same directory layout on the target nodes. E.g., if orgalorg told to upload file a while current working directory is /b/c/, orgalorg will upload file to the <root>/b/c/a on the remote nodes. That behaviour can be changed by specifying --relative or -e flag. Then, orgalorg will not preserve source file base directory.

orgalorg will try to upload files under specified user (current user by default). However, if user has NOPASSWD record in the sudoers file on the remote nodes, --sudo or -x can be used to elevate to root before uploading files. It makes possible to login to the remote nodes under normal user and rewrite system files.

Synchronization Tool

After file upload orgalorg will execute synchronization tool (/usr/lib/orgalorg/sync). That tool is expected to relocate synced files from temporary directory to the target directory. However, that tool can perform arbitrary actions, like reloading system services.

To specify custom synchronization tool user can use --sync-cmd or -n flag. Full shell syntax is supported in the argument to that option.

Tool is also expected to communicate with orgalorg using sync protocol (described below), however, it's not required. If not specified, orgalorg will communicate with that tool using stdin/stdout streams. User can change that behaviour using --simple or -m flag, which will cause orgalorg to treat specified sync tool as simple shell command. User can even provide stdin to that program by using --stdin or -i flag.

Tool can accept number of arguments, which can be specified by using -g or --arg flags.

Synchronization Protocol

orgalorg will communicate with given sync tool using special sync protocol, which gives possibility to perform some actions with synchronization across entire cluster.

orgalorg will start sync tool as it specified in the command line, without any modification.

After start, orgalorg will communicate with running sync tool using stdin and stdout streams. stderr will be passed to user untouched.

All communication messages should be prefixed by special prefix, which is send by orgalorg in the hello message. All lines on stdout that are not match given prefix will be printed as is, untouched.

Communication begins from the hello message.

Protocol

HELLO

orgalorg -> sync tool

<prefix> HELLO

Start communication session. All further messages should be prefixed with given prefix.

NODE

orgalorg -> sync tool

<prefix> NODE <node> [CURRENT]

orgalorg will send node list to the sync tools on each running node.

CURRENT flag will be present next to the node which is currently receiving protocol messages.

START

orgalorg -> sync tool

<prefix> START

Start messages will be sent at the end of the nodes list and means that sync tool can start doing actions.

SYNC

sync tool -> orgalorg

<prefix> SYNC <description>

Sync tool can send sync messages after some steps are done to be sure, that every node in cluster are performing steps gradually, in order.

When orgalorg receives sync message, it will be broadcasted to every connected sync tool.

SYNC (broadcasted)

orgalorg -> sync tool

<prefix> SYNC <node> <description>

orgalorg will retransmit incoming sync message from one node to every connected node (including node, that is sending sync).

Sync tools can wait for specific number of the incoming sync messages to continue to the next step of execution process.

Example

<- are outgoing messages (from orgalorg to sync tools).

<- ORGALORG:132464327653 HELLO
<- ORGALORG:132464327653 NODE [user@node1:22]
<- ORGALORG:132464327653 NODE [user@node2:1234] CURRENT
<- ORGALORG:132464327653 START
-> (from node1) ORGALORG:132464327653 SYNC phase 1 completed
<- ORGALORG:132464327653 SYNC [user@node1:22] phase 1 completed
-> (from node2) ORGALORG:132464327653 SYNC phase 1 completed
<- ORGALORG:132464327653 SYNC [user@node2:1234] phase 1 completed

Testing

To run tests it's enough to:

./run_tests

Requirements

Testcases are run through tests.sh library.

For every testcase new set of temporary containers will be initialized through hastur, so systemd is required for running test suite.

orgalorg testcases are close to reality as possible, so orgalorg will really connect via SSH to cluster of containers in each testcase.

Coverage

Run following command to calculate total coverage (available after running testsuite):

make coverage.total

Current coverage level is something about 85%.

orgalorg's Issues

Another alternative: GoSSHa

Consired adding another alternative:
https://github.com/YuriyNasretdinov/GoSSHa

Add support for SSH config

Love the tool!

Would it be possible to honor ~/.ssh/config files? My main use case is using a bastion host to run commands on machines.

Thanks!

Project description change suggestion

Hi guys,

I really like this tool, basically helped me ditch ansible and makeshift bash scripts to accomplish the same thing.
File synchronization is a great bonus.

What I'm about to say is .. trivial and superficial, I know. I just noticed something funny in the project description: "Parallel SSH commands executioner and file synchronization tool".
Unfortunately, executioner is exclusively someone that kills... well, people :) and has no context in IT.

I would recommend Parallel SSH commands runner.

Consul inventory support

It'd be really cool for the tool to be able to operate on consul inventory groups. To explain, here's what I currently do with ansible (pretty much same functionality):

ansible -i consul_dynamic_inventory.rb wiki -s -m command -a 'uptime'

This is going to consul using the ruby file and unrolling my "wiki" server group. Then it uses ansible's command plugin to fire uptime via ssh. Consul has a cool advantage of adding/removing services ad-hoc.

Here's how to get that information from Consul API:

#!/usr/bin/env ruby

require 'net/http'
require 'uri'
require 'json'

output = {}
s_json = JSON.parse(Net::HTTP.get_response(URI.parse('http://localhost:8500/v1/catalog/services')).body)
services = s_json.keys.reject{|k| k == 'consul'}
services.each do |srv|
  res = Net::HTTP.get_response(URI.parse("http://localhost:8500/v1/catalog/service/#{srv}")).body
  json = JSON.parse(res)
  a = []
  json.each {|j| a << j['Address'] }
  output = output.merge({srv => a})
end

puts JSON.generate(output)

It should be trivial to add this using golang, unfortunately I don't speak go yet :/ Sorry :)

run_tests freeze the centos 7 OS

I have a test centos 7 VMWare VM and the test phase will hang the OS.
May I know what is the OS dist/version that works ?

[pi@centos7 orgalorg]$ make
./run_tests
[build] building test go binary... ok.
[hastur] checking and initializing hastur...   -

[macOS Catalina]: orgalorg cannot be opened because the developer cannot be verified.

In Issue #28, I asked for simpler orgalorg macOS make experience, and @kovetskiy quickly added darwin to the github workflow releases (thus I can download a darwin binary from github Releases instead of fiddling with make): 🎉

However, I get a warning when I try to run the orgalorg 1.0.1 binary on OSX Catalina:

cannot be opened because the developer cannot be verified.

There are a couple of points.

Short term, I fixed this by:

• Trying to run orgalorg from the terminal, which popped up the warning referenced above
• Then I went to System Preferences -> Security & Privacy -> Allow Anyway
• When I ran orgalorg again, I got one final info pop-up... afterwards, it runs fine...

Long-term, orgalorg can fix this with an apple developer signing certificate
I'm opening this PR to document the short-term and long-term fixes.

Support Go Modules？

Support password-less SSH via config

Is there a way this tool can support password-less SSH via config setup? We don't have .ssh/id_rsa.

Reference:

https://www.digitalocean.com/community/tutorials/how-to-configure-custom-connection-options-for-your-ssh-client

orgalorg asks for key passphrase

Hi.

Orgalorg asks for key passphrase, even though I have already entered it and it's not needed via simple ssh.

Can orgalorg use same saved passphrase?

This works without passphrase:

$ ssh foobar1.mlan

This does not:

$ orgalorg -o foobar1.mlan -o foobar2.mlan -C "uname -a"
Key passphrase: 

Was release 1.2.0 retagged?

Homebrew has merged orgalorg version 1.2.0 18 days ago, however the current the current 1.2.0 tag is linked with a commit from 14 days ago. Was this retag intentional?

If so, I strongly encourage you to release new versions in the future instead. The git documentation indicates that pushing retag, while possible, is "the insane thing" to do in that situation.

Homebrew suggestion

Hi guys, long time ago I've created homebrew files but only for personal usage and now found releases.
So, I would like to suggest https://github.com/arren-ru/homebrew-orgalorg

Support for ssh-agent and forwarding?

Hi, are there any plans to support this?

Could you add LICENSE file?

First of all, thanks for sharing a nice software!

Could you add LICENSE file?
The following page may help for choosing a license.
Choose an open source license - Choose a License

I hope you choose a permissive license like MIT or Apache v2.
Thanks!

Windows 10 error: undefined: syscall.Stat_t

system: windows 10
golang 1.7.4

github.com/reconquest/orgalorg

D:\Home\Go\src\github.com\reconquest\orgalorg\archive.go:190: undefined: syscall.Stat_t
D:\Home\Go\src\github.com\reconquest\orgalorg\archive.go:197: undefined: syscall.Stat_t
D:\Home\Go\src\github.com\reconquest\orgalorg\archive.go:201: undefined: syscall.Stat_t

MacOS Orgalore doesnt work - Bug

Expected behavior:

orgalorg -o [email protected]:2266  -C id
uid=1000(op) gid=1000(op) groups=1000(op),27(sudo),998(docker)

I cant seem to get orgalorg to work on Mac. Tried by installing via go get and via the binary.
Bug:

./orgalorg -o [email protected]:2266  -C id -k /Users/desktop/.ssh/id_rsa.pub

2021-07-23 23:53:47 [FATAL] connecting to cluster failed
                            └─ connection to 1 of 1 nodes failed
                               └─ can't connect to nodes
                                  └─ can't connect to address: [[email protected]:2266]
                                     └─ ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Running the same command from Ubuntu 20.04 works.
SSH-ing normally from my Mac to 55.55.55.55 works

ssh [email protected] -p2266 < - works.

orgalorg seems to not be able to find the key file? Even specifying the keyfile in the command line manually doesn't work. Not specifying in the command and letting orgalorg picks the default still doesnt work.

./orgalorg -o [email protected]:2266  -C id                                  

2021-07-24 00:00:12 [FATAL] connecting to cluster failed
                            └─ connection to 1 of 1 nodes failed
                               └─ can't connect to nodes
                                  └─ can't connect to address: [[email protected]:2266]
                                     └─ ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Wenk wenk wenk wenk. Wenk wenk WENK!

can't acquire lock, lock already obtained by another process or unavailable

orgalorg -u big -p -o 192.168.6.204 -o 192.168.6.206 -o 192.168.6.208 -i ls.script -C bash > ls.log

ls.script: cd /home/cbss/20161129/ && ls

2016-11-29 09:27:35 [FATAL] connecting to cluster failed
└─ connection to 3 of 3 nodes failed
└─ can't connect to nodes
├─ [[email protected]:22] can't acquire lock, lock already obtained by another process or unavailable
├─ [[email protected]:22] can't acquire lock, lock already obtained by another process or unavailable
└─ [[email protected]:22] can't acquire lock, lock already obtained by another process or unavailable

ssh: must specify HostKeyCallback error

Hi,

I created the latest binary of orgalorg and got the error

ssh: must specify HostKeyCallback when connecting

Is this a bug?

Regards,
Kevin

unexpected reply string encountered: '"locked

Hey guys,

I've been using orgalorg with great success for the past few weeks!
I have now installed it on another machine, but unfortunately doing the exact same thing doesn't work anymore.

The error:

2016-08-30 16:07:38 [FATAL] acquiring global cluster lock failed
└─ connection to 9 of 9 nodes failed
   └─ can't connect to nodes
      ├─ [[email protected]:22] unexpected reply string encountered instead of 'acquired' or 'locked': '"locked
      │  '
      ├─ [[email protected]:22] unexpected reply string encountered instead of 'acquired' or 'locked': '"locked
      │  '
      ├─ [[email protected]:22] unexpected reply string encountered instead of 'acquired' or 'locked': '"locked
      │  '
      ├─ [[email protected]:22] unexpected reply string encountered instead of 'acquired' or 'locked': '"locked
      │  '
      ├─ [[email protected]:22] unexpected reply string encountered instead of 'acquired' or 'locked': '"locked
      │  '
      ├─ [[email protected]:22] unexpected reply string encountered instead of 'acquired' or 'locked': '"locked
      │  '
      ├─ [[email protected]:22] unexpected reply string encountered instead of 'acquired' or 'locked': '"locked
      │  '
      ├─ [[email protected]:22] unexpected reply string encountered instead of 'acquired' or 'locked': '"locked
      │  '
      └─ [[email protected]:22] unexpected reply string encountered instead of 'acquired' or 'locked': '"locked
         '

The version on that node was built from commit 593781a.
The working version on my machine was built from e292c36.

sync tool not included with download?

This project looks very promising, but seems like something is fundamentally missing. When I follow the instructions and do a

go get github.com/reconquest/orgalorg

I get the orgalorg binary built, no problem, but then executing a sync command like

./orgalorg -o 192.168.111.111 -o 192.168.111.101 -x -S /home/mallen/my-big-set-of-files

results in an error:

192.168.111.101 bash: /usr/lib/orgalorg/sync: No such file or directory
192.168.111.111 bash: /usr/lib/orgalorg/sync: No such file or directory
2016-07-25 18:42:57 [FATAL] failed to run sync command
└─ failed to finish sync tool command
└─ commands are exited with non-zero exit code on all 2 nodes
└─ code 127 (2 nodes)

It's not entirely clear to me what I have to do to make this part of it work.

Simplify / document build process

Hi,

I've got the orgalorg-1.0 source on OSX Catalina... I'm trying to build and this seems a little challenging. Is there a way to rip out the reconquest/go-test.bash and reconquest/test-runnner dependencies as a requirement just to install locally? Another option would be more clear documentation about getting dependencies to build orgalorg.

(py38_osx) mpenning@MPENNING-M-PB3W orgalorg-1.0 % make
Makefile:5: vendor/github.com/reconquest/test-runner.bash/Makefile: No such file or directory
Makefile:6: vendor/github.com/reconquest/go-test.bash/Makefile: No such file or directory
bash -c 'source vendor/github.com/reconquest/import.bash/import.bash && \
		import "github.com/reconquest/go-test.bash"'
bash: vendor/github.com/reconquest/import.bash/import.bash: No such file or directory
make: *** [vendor/github.com/reconquest/go-test.bash/Makefile] Error 1
(py38_osx) mpenning@MPENNING-M-PB3W orgalorg-1.0 %

I exported GOROOT="" and GOPATH="~/go"... I extracted the source tarball in ~/ when I tried installing.

unable to connect to nodes using key

Hey guys, thanks for the tool, it's just what I need right now!

Unfortunately I can't seem to connect to the nodes using my key, this is what the log says:

2016-07-12 10:26:41 [FATAL] acquiring global cluster lock failed
└─ connection to 10 of 10 nodes failed
   └─ can't connect to nodes
      ├─ can't connect to address: [[email protected]:22]
      │  └─ can't parse pem data: asn1: structure error: tags don't match (16 vs {class:3 tag:25 length:12 isCompound:false}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2
...

Using the password-based approach works, but of course using a key would be nicer :)

I cannot send you my key for obvious reasons, but this is what it looks like:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,<probably fingerprint?>

<key data>
-----END RSA PRIVATE KEY-----

Any ideas? Is it because the key is encrypted?

tsocks support?

hello,
Would you support "tsocks" in the future ?

Mention -u flag in README.md

There's plenty of usage examples in README.md, but none of it mentions how to use alternate login.

Since there's no support for ssh-config at the moment, user might think that this is entirely not possible until he runs orgalorg --help.

Taking commands from file or STDIN doesnt return.

Hi all, first off thank you very much for this killer tool.

When using --stdin or -i, after the last command has executed the terminal never returns. I have to control+c to close the process. Is this expected behavior? I built from source via go get github.com/reconquest/orgalorg go version go1.16.3 linux/amd64 on Ubuntu 20.04. Thank You in Advance!

root@localhost:~# cat commands.txt 
echo "first command executed"
echo "second command executed"
echo "This is the last command in the file"

root@localhost:~# orgalorg -q -o hosts -i commands.txt -C /bin/bash
op@localhost:~$ first command executed
op@localhost:~$ second command executed
op@localhost:~$ first command executed
op@localhost:~$ second command executed
op@localhost:~$ This is the last command in the file
op@localhost:~$ This is the last command in the file
op@localhost:~$ first command executed
op@localhost:~$ second command executed
op@localhost:~$ This is the last command in the file

^C
root@localhost:~# 

and with the -l or --serial the first node never finishes so the second node and so-on are never invoked

root@localhost:~# orgalorg -l -q -o hosts -i commands.txt -C /bin/bash
op@localhost:~$ first command executed
op@localhost:~$ second command executed
op@localhost:~$ This is the last command in the file

[windows] panic: interface conversion: interface {} is nil, not bool

orgalorg version: 1.2.0
platform: windows x64

orgalorg -v -s -u root -y -e -r / -S etc
2022-04-17 18:00:48 [DEBUG] building files list from 1 sources
2022-04-17 18:00:48 [DEBUG] file list contains 1 files
2022-04-17 18:00:48 [DEBUG] trying windows pageant
2022-04-17 18:00:48 [DEBUG]     pageant is not found
2022-04-17 18:00:48 [DEBUG] trying windows openssh-agent
2022-04-17 18:00:48 [DEBUG] using 16 threads
2022-04-17 18:00:48 [DEBUG] connecting to 3 nodes
2022-04-17 18:00:48 [DEBUG]    1/3 (0 failed) connection established: [root@srv1:22]
2022-04-17 18:00:48 [DEBUG]    2/3 (0 failed) connection established: [root@srv2:22]
2022-04-17 18:00:48 [DEBUG]    3/3 (0 failed) connection established: [root@srv3:22]
2022-04-17 18:00:48 [DEBUG] connection established to 3 nodes
panic: interface conversion: interface {} is nil, not bool

goroutine 1 [running]:
main.upload(0xfaddb7, 0xc0003b4c00, {0xc0003ae120, 0x1, 0x1})
        /home/runner/work/orgalorg/orgalorg/main.go:543 +0x566
main.handleSynchronize(0xf62b00)
        /home/runner/work/orgalorg/orgalorg/main.go:484 +0x6a7
main.main()
        /home/runner/work/orgalorg/orgalorg/main.go:282 +0x6ef

where etc is a local directory containing a file

Expected: synchronize /some/dir/etc/file to remote /etc/file

The name of this tool is too hard to pronounce

Should change to or or or such easy name else just for simple?

not maintained any longer?

no support?