GithubHelp home page GithubHelp logo

windvalley / gossh Goto Github PK

View Code? Open in Web Editor NEW
163.0 14.0 41.0 378 KB

🚀🚀A high-performance and high-concurrency ssh tool written in Go. It is 10 times faster than Ansible. If you need much more performance and better ease of use, you will love it.

License: MIT License

Go 94.39% Makefile 5.46% Awk 0.16%
batchssh gossh ssh ssh-client sre devops ops sa ansible high-concurrency

gossh's Introduction

GoSSH

Language Github Workflow Status Quality Gate Status Version Supports Page Views Release Download Total LICENSE

Gossh is a high-performance and high-concurrency ssh tool written in Go. It can efficiently manage tens of thousands of Linux server clusters. It is 10 times faster than Ansible. If you need much more performance and better ease of use, you will love gossh.

Feel free to open a new issue if you have any issues, questions or suggestions about this project.

💝 Features

  • Four kinds of ssh tasks:
    command: Execute commands on target hosts.
    script: Execute a local shell script on target hosts.
    push: Copy local files and dirs to target hosts.
    fetch: Copy files and dirs from target hosts to local.

  • Auto detect following authentication methods for the login user(default $USER):
    Password: from inventory file, or from flag -k/--auth.ask-pass,-p/--auth.password,-a/--auth.pass-file, or from configuration file.
    Pubkey Authentication: by identity files(default ~/.ssh/id_rsa), also include that with passphrase.
    SSH-Agent Authentication: through the system environment variable $SSH_AUTH_SOCK.
    If the above three authentication methods are valid at the same time, the priority order is: SSH-Agent > Pubkey > Password.

  • Provide the target hosts by:
    Hosts/host-patterns/host-group-names as positional arguments separated by space.
    Inventory file by flag -i/--hosts.inventory or from configuration file.

  • Expand host patterns that from positional arguments or a inventory file to host list. Supported host patterns e.g.:

    10.16.0.[1-10]
    foo[01-03].bar.com
    foo[01-03,06,12-16].idc[1-3].[beijing,wuhan].bar.com
    
  • Allow adding variables to inventory file.
    Available variables: host, port, user, password, keys, passphrase. E.g.:

    alias_name_node1 host=node1.sre.im
    alias_name_node2 host=192.168.33.12 port=8022 user=vagrant password=123456 keys=~/.ssh/id_rsa passphrase=xxx
    node3.sre.im user=vagrant password=GOSSH-AES256:9cfe499133b69a6c7fc62b5b6ba72d3d8dfb4d0e7987170a40c5d50bb5d71e19
    
  • Group hosts in inventory file. E.g.:

    # no group hosts
    node1.sre.im
    
    # group hosts
    [webserver]
    node2.sre.im port=6022
    node3.sre.im
    
    # host variables for group webserver
    [webserver:vars]
    port=8022
    user=zhangsan
    password=plaintextOrCiphertextByVault
    
    [dbserver]
    db[1-3].sre.im
    
    # group project1 has hosts that from both group webserver and dbserver
    [project1:children]
    webserver
    dbserver
    
  • Use sudo to run as other user(default root) to execute the commands/shell-script or fetch files/dirs.

  • Specify i18n environment variable value while executing commands or a shell script to help keep the language of the outputs consistent. For example: zh_CN.UTF-8, en_US.UTF-8.

  • Three kinds of timeout in seconds:
    Connecting to each target host (default 10).
    Subcommand command, script, push, fetch for each target host.
    The entire gossh task.

  • Output to a file or screen or a file and screen at the same time.
    Colorful output, json format output, verbose(debug) output, and silent output.

  • High-performance and high-concurrency. Customize the number of concurrent connections (default 1).

  • SSH Proxy can be specified to connect to the target hosts.

  • Provides subcommand vault to encrypt/decrypt confidential information such as password or passphrase without compromising security.

  • Support the detection of dangerous commands that will be executed on target hosts, and allow users to customize a blacklist of commands.

  • For ease of use, it supports config file. You can write flags that are not frequently changed into the config file, so you don't need to laboriously specify these flags on the command line. If the flag in both command line and config file, flag that from command line takes precedence over the other.
    The default config file is $PWD/.gossh.yaml or $HOME/.gossh.yaml, and $PWD/.gossh.yaml has a higher priority. Note that the config file is optional, that is, there can be no config file.

  • Provides subcommand config to help generate configuration file in easy way.

🛠 Installation

Prebuilt binaries for macOS and Linux can be downloaded from the GitHub releases page.

Also you can install gossh by compiling:

$ git clone --depth 1 https://github.com/windvalley/gossh

$ cd gossh

# Note: need to install Go environment first.
$ make && make install

📜 Usage

$ gossh -h

Gossh is a high-performance and high-concurrency ssh tool.
It can efficiently manage tens of thousands of Linux server clusters.

Find more information at: https://github.com/windvalley/gossh

Usage:
  gossh [command]

Available Commands:
  command     Execute commands on target hosts
  script      Execute a local shell script on target hosts
  push        Copy local files and dirs to target hosts
  fetch       Copy files and dirs from target hosts to local
  vault       Encryption and decryption utility
  config      Generate gossh configuration file
  version     Show gossh version information
  help        Help about any command
  completion  Generate the autocompletion script for the specified shell

Flags:
  -u, --auth.user string                login user (default $USER)
  -p, --auth.password string            password of login user
  -k, --auth.ask-pass                   ask for the password of login user
  -a, --auth.pass-file string           file that holds the password of login user
  -I, --auth.identity-files strings     identity files (default [~/.ssh/id_rsa])
  -K, --auth.passphrase string          passphrase of the identity files
  -V, --auth.vault-pass-file string     text file or executable file that holds the vault password
                                        for encryption and decryption
  -i, --hosts.inventory string          file that holds the target hosts
  -P, --hosts.port int                  port of the target hosts (default 22)
  -l, --hosts.list                      outputs a list of target hosts, and does not do anything else
  -s, --run.sudo                        use sudo to execute commands/script or fetch files/dirs
  -U, --run.as-user string              run via sudo as this user (default "root")
  -L, --run.lang string                 specify i18n while executing command
                                        (e.g. zh_CN.UTF-8|en_US.UTF-8)
  -c, --run.concurrency int             number of concurrent connections (default 1)
  -B, --run.command-blacklist strings   commands that are prohibited from execution on target hosts
                                        (default: [rm,reboot,halt,shutdown,init,mkfs,mkfs.*,umount,dd])
  -o, --output.file string              file to which messages are output
  -j, --output.json                     output messages in json format
  -C, --output.condense                 condense output and disable color
  -q, --output.quiet                    do not output messages to screen
  -v, --output.verbose                  show debug messages
  -X, --proxy.server string             proxy server address
      --proxy.port int                  proxy server port (default 22)
      --proxy.user string               login user for proxy (default same as 'auth.user')
      --proxy.password string           password for proxy (default same as 'auth.password')
      --proxy.identity-files strings    identity files for proxy (default same as 'auth.identity-files')
      --proxy.passphrase string         passphrase of the identity files for proxy
                                        (default same as 'auth.passphrase')
  -t, --timeout.command int             timeout seconds for handling each target host
      --timeout.task int                timeout seconds for the entire gossh task
      --timeout.conn int                timeout seconds for connecting each target host (default 10)
      --config string                   config file (default {$PWD,$HOME}/.gossh.yaml)
  -h, --help                            help for gossh

Use "gossh [command] --help" for more information about a command.

🚀 Performance

Client server: 4vCPUs and 8GiB

Target servers: hosts.list contains 936 servers distributed in 86 different IDCs across the country.

Ansible:

$ time ansible all -i hosts.list -m command -a "uptime" -f 100 -k

Output:

...

real    1m18.858s
user    3m18.566s
sys     1m24.263s

Gossh:

$ time gossh command -i hosts.list -e "uptime" -c 100 -k

Output:

...

[INFO] 2021-12-22 23:06:50.837228 success count: 936, failed count: 0, elapsed: 6.30s

real    0m6.316s
user    0m13.529s
sys     0m0.592s

📄 Documentation

📝 Changelog

CHANGELOG

✨ Contributing

I actively welcome your pull requests, please follow the steps below:

  1. Open a new issue at gossh/issues.

  2. Fork the repo and create your branch from main.

  3. Do the changes in your fork.

  4. Send a pull request.

Note: Any contributions you make will be under the MIT Software License.

⚖️ License

This project is under the MIT License. See the LICENSE file for the full license text.

gossh's People

Contributors

windvalley avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gossh's Issues

The outputs that were originally 'FAILED' was marked as 'SUCCESS'

For example:

$ gossh command "f[01-02].infra.sre.im" -e "ls -l /root" -k

Output:

f01.infra.sre.im | 2022-01-07 15:52:42 | FAILED >>
ls: /root: Permission denied

f02.infra.sre.im | 2022-01-07 15:52:43 | SUCCESS >>
ls: /root: Permission denied

All the records output above should be FAILED.

Add flag '-t/--tmp-dir' for subcommand 'fetch'

One of the cores of the subcommand fetch is to zip(zip command required on target host) the target files and directories into a temporary directory on target host. The default temporary directory is /tmp, you can change it by flag -t/--tmp-dir for some reason, e.g. insufficient space in default /tmp.

Plan to add flag:

-t/--tmp-dir directory of target hosts for storing temporary zip file (default /tmp/)

add feature to provide different username & passwords/keys for servers

@windvalley thanks for creating this awesome tool.

let me how to provide different usernames for servers like providing ansible_user in ansible inventory file.

[targets]

localhost              ansible_connection=local
other1.example.com     ansible_connection=ssh        ansible_user=myuser
other2.example.com     ansible_connection=ssh        ansible_user=myotheruser

Add flag '-L/--hosts.list' for outputing a list of the target hosts for subcommand 'command/script/push'

For example:

$ gossh script foo66.shenzhen.bar.com "foo[01-20].[beijing,wuhan].bar.com" -L

Output:

foo66.shenzhen.bar.com
foo01.beijing.bar.com
foo01.wuhan.bar.com
foo02.beijing.bar.com
foo02.wuhan.bar.com
foo03.beijing.bar.com
foo03.wuhan.bar.com
foo04.beijing.bar.com
foo04.wuhan.bar.com
foo05.beijing.bar.com
foo05.wuhan.bar.com
foo06.beijing.bar.com
foo06.wuhan.bar.com
foo07.beijing.bar.com
foo07.wuhan.bar.com
foo08.beijing.bar.com
foo08.wuhan.bar.com
foo09.beijing.bar.com
foo09.wuhan.bar.com
foo10.beijing.bar.com
foo10.wuhan.bar.com

hosts (21) 
$ gossh script foo66.shenzhen.bar.com "foo[01-20].[beijing,wuhan].bar.com" -L | head 

Output:


hosts (21)
foo66.shenzhen.bar.com
foo01.beijing.bar.com
foo01.wuhan.bar.com
foo02.beijing.bar.com
foo02.wuhan.bar.com
foo03.beijing.bar.com
foo03.wuhan.bar.com
foo04.beijing.bar.com
foo04.wuhan.bar.com
foo05.beijing.bar.com    
$ gossh script foo66.shenzhen.bar.com "foo[01-10].[beijing,wuhan].bar.com"

Output:

Error: need flag '-e/--execute' or '-L/--hosts.list'    

Add dangerous commands detection feature for subcommand `command` and `script`

For security reasons, it is necessary to provide a feature for detecting dangerous commands, and user can customize the command blacklist. If there is no customization, the default blacklist provided by the tool will be used.

The command line:

$ gossh command -h    OR    $ gossh script -h
...
Flags:
...
  -n, --no-safe-check    ignore dangerous commands (from '-B,--run.command-blacklist') check
...
Global Flags:
...
  -B, --run.command-blacklist strings   commands that are prohibited from execution on target hosts
                                        (default: rm,reboot,halt,shutdown,init,mkfs,mkfs.*,umount,dd)
...

Or customize command blacklist in config file:

run:
  # Linux Command Blacklist for gossh subcommands 'command' and 'script'.
  # Commands listed in this blacklist will be prohibited from executing on remote hosts for security reasons.
  # You can add flag '-n, --no-safe-check' to disable this feature.
  # Default: ["rm", "reboot", "halt", "shutdown", "init", "mkfs", "mkfs.*", "umount", "dd"]
  command-blacklist: []

'gossh vault encrypt' get plaintext that to be encrypted from prompt

Hi, thanks for your excellent work. Currently command gossh vault encrypt only obtains the plaintext to be encrypted from the argument. For security reasons, can you consider adding the feature of obtaining plaintext from promt?

For example:

$ gossh vault encrypt -V ./vault-pass.txt
Plaintext:
Confirm plaintext:

Sudo password prompt output not be trimmed as expected

Sudo password prompt like follows should be trimmed from output of the target hosts:

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for userfoo:

Optimize the priority of the ssh authentication methods

Now, there are three authentication methods: password, pubkey, ssh-agent.
If the 3 auth methods are all valid, the current priority order is password > pubkey > ssh-agent.
But I think a more reasonable order should be: ssh-agent > pubkey > password. The next gossh version will adjust the order to this.

Tips:
You can observe the actual authentication method used from the /var/log/secure file of the target host.

tail -f /var/log/secure

Default 'completion' command description is inconsistent with others

Available Commands:
  command     Execute commands on target hosts
  completion  generate the autocompletion script for the specified shell
  config      Generate gossh configuration file
  fetch       Copy files/dirs from target hosts to local
  help        Help about any command
  push        Copy local files/dirs to target hosts
  script      Execute a local shell script on target hosts
  vault       Encryption and decryption utility
  version     Show gossh version information

The default completion command that from cobra is lowercased, but another default help command that from cobra is uppercased.

Recommendation: Make them all uppercase.

AIX make crashed - undefined: fsnotify.Watcher

AIX v.7.2
go env
GO111MODULE=""
GOARCH="ppc64"
GOBIN=""
GOCACHE=.
GOENV=
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="ppc64"
GOHOSTOS="aix"
GOINSECURE=""
GOMODCACHE="/home/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="aix"
GOPATH="/home/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/freeware/lib/golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/freeware/lib/golang/pkg/tool/aix_ppc64"
GOVCS=""
GOVERSION="go1.20.2"
GCCGO="gccgo"
GOPPC64="power8"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/go/src/gossh/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -maix64 -pthread -mcmodel=large -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1254645141=/tmp/go-build -gno-record-gcc-switches"

==========> Building binary 'gossh' v1.15.1 for aix ppc64
github.com/spf13/viper
../../pkg/mod/github.com/spf13/[email protected]/watch.go:8:25: undefined: fsnotify.Watcher
../../pkg/mod/github.com/spf13/[email protected]/watch.go:11:18: undefined: fsnotify.NewWatcher
make[1]: *** [scripts/makefiles/go.makefile:54: go.build.aix_ppc64.gossh] Error 1
make: *** [Makefile:43: build] Error 2

I found a similar case spf13/viper#1416
Wouldn't it be possible to solve/bypass it in a similar way?

Output in json format by flag '-j/--output.json' not correct

$ gossh command -e "uptime" 10.16.130.100 -k -j

Output:

Password:
{"hostname":"10.16.130.100","level":"error","msg":"failed","output":"15:16:07 up 434 days, 11:38, 11 users,  load average: 0.00, 0.00, 0.00","status":"SUCCESS","time":"2022-01-05 15:16:07"}
{"level":"info","msg":"success count: 1, failed count: 0, elapsed: 0.91s","time":"2022-01-05 15:16:07"} 

Should be:

Password:
{"hostname":"10.16.130.100","level":"info","msg":"success","output":"15:16:07 up 434 days, 11:38, 11 users,  load average: 0.00, 0.00, 0.00","status":"SUCCESS","time":"2022-01-05 15:16:07"}
{"level":"info","msg":"success count: 1, failed count: 0, elapsed: 0.91s","time":"2022-01-05 15:16:07"} 

[Plan] Add subcommand 'vault' that helps you encrypt/decrypt confidential information without compromising security

Demo:

$ gossh vault -h

Encrypt sensitive content such as passwords so you can protect it rather than
leaving it visible as plaintext in public place. To use vault you need another
password(vault-pass) to encrypt and decrypt the content.

Usage:
  gossh vault [command]

Available Commands:
  decrypt     Decrypt content encrypted by vault
  encrypt     Encrypt sensitive content

Flags:
  -h, --help                     help for vault

Global Flags:
  -V, --auth.vault-pass-file string    file that holds the vault password for encryption and decryption

Support SSH Proxy

I plan to add a new feature: Allow to connect to the target linux servers by specifying a ssh proxy server.

[Plan] Support group hosts, group vars and group combination in inventory file(hosts file)

Inventory file demo:

# This is a hosts inventory file for gossh

# no group hosts
alias_name_node1 host=node1.sre.im
node100.sre.im

# hosts group
[webserver]
alias_name_node2 host=192.168.33.12 port=8022 user=vagrant password=123456 keys=~/.ssh/id_dsa,~/.ssh/id_rsa passphrase=xxx
node[06-07].sre.im port=9022 user=lisi password=654321
node08.sre.im 

# host vars for group webserver
[webserver:vars]
port=8033
user=wangwu

[dbserver]
192.168.1.10

[dbserver:vars]
user=vagrant2
password=abcdefg

# hosts group project1 has hosts that defined in group dbserver and group webserver
[project1:children]
dbserver
webserver

The default config doesn't use the default identity-files

the default config generated by

gossh config >$HOME/.gossh.yaml

is

  # Identity files of pubkey authentication.
  # Default:
  #   - $HOME/.ssh/id_rsa
  #   - $HOME/.ssh/id_dsa
  identity-files:
    -
gossh exec 192.168.5.31 -e "uptime" -v

level=debug msg=Using config file: /home/mate/.gossh.yaml time=2021-12-31 14:00:04
level=debug msg=Config contents: {"auth":{"user":"mate","password":"","ask-pass":false,"file":"","identity-files":[""],"passphrase":""},"hosts":{"file":"","port":22},"run":{"sudo":false,"as-user":"root","lang":"","concurrency":1},"output":{"file":"","json":false,"quiet":false,"verbose":true},"timeout":{"conn":10,"command":0,"task":0},"proxy":{"server":"","port":22,"user":"","password":"","identity-files":[""],"passphrase":""}} time=2021-12-31 14:00:04
level=debug msg=Auth: password of the login user not provided time=2021-12-31 14:00:04
level=debug msg=Auth: read identity file '' failed: open : no such file or directory time=2021-12-31 14:00:04
msg=Auth: no valid identity files time=2021-12-31 14:00:04 level=debug
level=debug msg=Auth: no valid authentication method detected. Prompt for password of the login user time=2021-12-31 14:00:04
Password:

the default config should be

  # Identity files of pubkey authentication.
  # Default:
  #   - $HOME/.ssh/id_rsa
  #   - $HOME/.ssh/id_dsa
  identity-files: []

gossh version v1.0.1

chdir: error retrieving current directory: getcwd: cannot access parent directories

I have two logins roze and roze2.
On target host (kom) sudoers with such entry.
roze,roze2 ALL=(ALL) NOPASSWD: ALL

I can run right on target host simple:
/tmp: sudo -u roze touch b2

ls -l b2
-rw-r----- 1 roze xxx 0 2 bře 23:23 b # OK

but when I run
gossh command kom -U roze -s -e "cd /tmp; touch /tmp/pokus " -v -p ....

[DEBUG] 2024-03-02 23:39:56.182174 kom: execute command 'sudo -u roze -H bash -c ' cd /tmp; touch /tmp/pokus'' success, output: shell-init: chyba při zjišťování současného adresáře: getcwd: rodičovské adresáře nejsou přístupné: Přístupová práva souboru neumožňují provádět zadanou činnost.
chdir: error retrieving current directory: getcwd: cannot access parent directories: The file access permissions do not allow the specified action.

kom 2024-03-02 23:39:56.182553 | SUCCESS >>
shell-init: chyba při zjišťování současného adresáře: getcwd: rodičovské adresáře nejsou přístupné: Přístupová práva souboru neumožňují provádět zadanou činnost.
chdir: error retrieving current directory: getcwd: cannot access parent directories: The file access permissions do not allow the specified action.

[INFO] 2024-03-02 23:39:56.182597 success count: 1, failed count: 0, elapsed: 0.43s

/tmp/pokus was not created
how is it possible that failed count is 0 ??

================
AIX 7.2
gossh version
gitVersion: 92e3206
gitCommit: 92e3206
gitTreeState: clean
buildDate: 2024-02-25T11:23:47Z
goVersion: go1.20.2
compiler: gc
platform: aix/ppc64

Flag '--timeout.command' does not work in some case

When encountering the following error output of the remote server, it had been stuck for a long time, and flag --timeout.command did not work in this scenario:

 ssh: handshake failed: read tcp 10.161.115.122:47572->10.16.153.130:22: read: connection reset by peer

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.