Use a yaml file to stitch together commands and bash snippits and run them with a bit of style. Why? Because your bash script should be quiet and shy-like (...and not such a loud mouth).
License: MIT License
Would be really cool if there would be a brew package (and maybe RPM and DEB) to easily install and update on development machines. This is a great replacement for Makefiles - which are rather annoying - and now it's a bit more complicated to use it...
Taking into consideration the purpose of max-parallel-commands it should be possible to override the value via a flag. In my case, depending on whether it's my laptop or CI server I might want to limit or raise the number of parallel executions to match my environment better. Having it in the file is not as flexible as I would expect...
One of my use cases it to be able to deploy my docker images to remote servers using docker-machine. For that, there is eval (docker-machine env machine-name) which will set the env variables to a remote host.
It doesn't seem like it works well with bashful.
tasks:
- name: Updating env
cmd: eval (docker-machine env machine-name)
- name: Test
cmd: env | grep DOCKER
This doesn't return anything and it fails instead. Would be possible to add this where env variables can be sourced and they would live for the duration of the script.
Instead of a negative time value an unknown, not-available or infinite symbol might be nicer looking.
tasks:
- name: pull
cmd: docker pull ubuntu:14.10
show-output: true
shows "b0efe5c05b4c: Download complete" as the output, because the last line is not the one that shows the progress:
$ docker pull ubuntu:14.10
14.10: Pulling from library/ubuntu
b0efe5c05b4c: Downloading [==================================> ] 46.83MB/68.32MB
0a1f1b169319: Download complete
1ceb0a3c7c48: Download complete
a3ed95caeb02: Download complete
Can this be fixed?
Should be able to override x-ref-data with another list/hash via cli or a graph file
Allow a 'cwd' option to cd to a given configured path before executing.
When running apt-get upgrade on Ubuntu 18, I see the following warning logged multiple times:
dpkg: warning: parsing file '/var/lib/dpkg/status' near line 42388 package 'bashful':
missing maintainer
Add a maintainer - give yourself some credit!
Trying to run a bunch of small tasks in parallel (trying 12x once)
Error:
Line: 193 ash /hFile: /home/wagoodman/go/src/github.com/wagoodman/bashful/screen.go s/leftcenter/the-forward/the-forward-all.html
fork/exec /bin/stty: too many open files Runtime[00:01:22] ETA[00:-1:-22]
&{0xc4200a0060} Unable to determine screen width.
yml file is built from a list:
config:
max-parallel-commands: 12
tasks:
- name: Download 1 of 227
parallel-tasks:
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- name: Download 2 of 227
parallel-tasks:
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]
- cmd: bash inner-find-domain.sh [a local path]Add 'bundle' feature that zips the yaml and referenced scripts (with listed dependencies) into a single binary that can be directly ran.
Currently this cannot be compiled for windows since it references specific syscalls. Platform specific code should be broken into its own file or module first, then windows support can be added.
Here is my script:
tasks:
- name: phpqa
parallel-tasks:
- name: PHPStan
cmd: docker run -t -w /project jakzal/phpqa:alpine phpstan analyse --no-progress EXAMPLEHere is the output:
Running /home/alexis/scripts/phpqa.yaml
• phpqa
╰─ PHPStan Exited with error (1)
100.00% Complete Tasks[1/1] See log for details (/dev/null) Runtime[00:00:00]
...Some tasks failed, see below for details.
⏺ Failed task: PHPStan
├─ command: docker run -t -w /project jakzal/phpqa:alpine phpstan analyse --no-progress EXAMPLE
├─ return code: 1
╰─ stderr: There's no output.
I add 1>&2 to the end of the command.
Here is the output:
$ bashful run ~/scripts/phpqa.yaml
Running /home/alexis/scripts/phpqa.yaml
• phpqa
╰─ PHPStan Exited with error (1)
100.00% Complete Tasks[1/1] See log for details (/dev/null) Runtime[00:00:01]
...Some tasks failed, see below for details.
⏺ Failed task: PHPStan
├─ command: docker run -t -w /project jakzal/phpqa:alpine phpstan analyse --no-progress EXAMPLE 1>&2
├─ return code: 1
╰─ stderr: ------ --------------------------------------
Line EXAMPLE
------ --------------------------------------
Path /project/EXAMPLE does not exist
------ --------------------------------------
[ERROR] Found 1 error I have to use 1>&2 because the command outputs result on stdout instead of stderr.
Is it possible to display stdout instead of strerr id a command fails? Could it be configurable for each command?
Most config issues result in odd output or panics:
We should stop execution gracefully before starting tasks.
example:
2017/12/05 15:00:12 Started Task: Cleaning up workspace
2017/12/05 15:00:12 Task full output: Packaging RPM
worker (1/2): 7Xn5Stth
worker (2/2): TjTo4e2JUoo2szGDVbaVdRpiydCg3EEn5wrTgGFe5Km7d0eWXeLKl5jpOSeXk8W4Q
2017/12/05 15:00:14 Completed Task: Cleaning up workspace (rc: 0)
2017/12/05 15:00:14 Finished example/06-vintage.yml <--- THIS: should show up after the process output block that follows
2017/12/05 15:00:12 Task full output: Cleaning up workspace
worker (1/10): qfjTvAflXNmOrfS6XaKvUVa
worker (2/10): pXTIpb0VcGn08484z4yL3HCdmZgdWY6GabuDxZ8npynmiZIhqImeLSebK9Mq4GJmnScAbWraumL
worker (3/10): eU13MfKCRE7wgB6iL3kfFLa3QeBVx6jMmfM9DZNnZBI52yMmm4rrPUA
worker (4/10): bDlqtFNc6kgaz4NeeGiOnUdBkEwofqeUbA92o
worker (5/10): ipsgxddZPQJbFo1ceowptRR56RAXIYF3spcWeO79FFKqQGxrrtmrfcw04DLUP7m3b6SjC1eY3o
worker (6/10): 6QNHemYxwl9djed8mRtW79J2ICFpLnvmyBTf3U
worker (7/10): ogSHZFbKPBggdfvIWT
worker (8/10): Mw8inKDwk9Mwfx41GEZDKYZotNY
worker (9/10): TMeQmA
worker (10/10): 1zhodaty2VrbUBOwkqVdLx9UVKeFUomGGy936WrS3dB2RUxwd6TBplqOnKCFdPKXAV3i
2017/12/05 15:00:14 Exiting
Hi, thanks for sharing a great tool!
While trying to author a relatively large tool based on bashful, I came up with this, so that I could write a lengthy bash script on our favorite editor with nice syntax highlighting, lint, etc.
Instead of:
tasks:
- name: Building app
cmd: go build -ldflags '-linkmode external -extldflags -static -s'
tags: build
I'd like to organize it by extracting the cmd to a dedicated bash script:
tasks:
- name: Building app
script: scripts/build
tags: build
Whereas scripts/build is:
#!/usr/bin/env bash
set -e
go build -ldflags '-linkmode external -extldflags -static -s'
Note that we can't just write it cmd: ./scripts/build as it doesn't get bundled in bashful bundle.
If a task fails in a parallel group, the errors counter does not update until the group has completed.
- name: Install NPM
url: http://npmjs.org/install.shshould be able to take an optional checksum:
- name: Install NPM
url: http://npmjs.org/install.sh
md5: 9c5d46a77f54bbc3ea1ace3a21b26fa7...and runtime options (? is replaced with the downloaded script):
- name: Install NPM
url: http://npmjs.org/install.sh
runtime: sudo ? --some options -h here I have not managed to cross compile the bundle to a linux native from a macOS. Am I missing something? Is there a way to do it currently?
When the log is specified in the config, indicate that there is logged output when something fails.
(update exitwitherrormessage())
It would be nice if you could include a bashful yaml in another one.
Question marks are left instead
Allow a task to have a list of commands that are always run back-to-back in series.
Could be achieved by nested parallel blocks?
Or should there be only one background block allowed?
Add a flag to a task block that allows a for-each sweep through parameters to be parallelized across parameter values #
This takes several seconds to run:
config:
# log-path: build.log
max-parallel-commands: 12
event-driven: false
tasks:
- name: Processing
parallel-tasks:
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
- cmd: echo test
causes panic
Add an 'on-exit', 'on-error', 'on-success' exit yaml sections (probably residing outside the 'tasks' section). These would be invoked as callback hooks when particular events occur.
https://github.com/golang/go/wiki/MacOS12BSDThreadRegisterIssue
fatal error: runtime: bsdthread_register error
runtime stack:
runtime.throw(0x142ad72, 0x21)
/usr/lib/golang/src/runtime/panic.go:605 +0x95 fp=0x7ff7bfeff680 sp=0x7ff7bfeff660 pc=0x1029e95
runtime.goenvs()
/usr/lib/golang/src/runtime/os_darwin.go:108 +0x83 fp=0x7ff7bfeff6b0 sp=0x7ff7bfeff680 pc=0x1027733
runtime.schedinit()
/usr/lib/golang/src/runtime/proc.go:492 +0xa4 fp=0x7ff7bfeff6f0 sp=0x7ff7bfeff6b0 pc=0x102c874
runtime.rt0_go(0x7ff7bfeff728, 0x3, 0x7ff7bfeff728, 0x0, 0x1000000, 0x3, 0x7ff7bfeff910, 0x7ff7bfeff918, 0x7ff7bfeff91c, 0x0, ...)
/usr/lib/golang/src/runtime/asm_amd64.s:175 +0x1eb fp=0x7ff7bfeff6f8 sp=0x7ff7bfeff6f0 pc=0x105488b
It would be ideal if there was a simple feature that tracked the time taken to complete a run. This could be at the end of output after the run, or persistently in the summary footer.
Bonus points if you can cache previous run times and predict an ETA in the summary footer.
Allow for typical bash operations (pipes and redirects).
Also allow for yaml blocks of embedded bash.
Maybe too many screen updates? Maybe the system it running out of resources?
I can not run the simplest example.
YML file content:
tasks:
- cmd: echo "Hello, World!"
Result of bashful run hello.yml:
Running hello.yml
• echo "Hello, World!" Exited with error (-1)
100.00% Complete Tasks[1/1] Runtime[00:00:00]
...Some tasks failed, see below for details.
⏺ Failed task: echo "Hello, World!"
├─ command: echo "Hello, World!"
├─ return code: -1
╰─ stderr: Failed to run: exec: not started
go version is go version go1.8 linux/amd64
go env is
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/username/.go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build246215807=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
Looking forward for your support
I think. Happens with this config:
config:
log-path: build.log
collapse-on-completion: true
show-summary-footer: false
show-task-output: false
# show-task-times: false
I expected the tasks to still be shown.
read from the SHELL env var to support zsh and other shells
I ran a bashful script in a project from another directory and I was surprised to see that git detected changes: a .bashful directory has been created in my project.
I think that bashful should not create a directory automatically, but there could be an option so that users who want that could enable this option.
Context: I'm running code analysis tools so I don't need to store results or any other data, I only want to see if commands return 0 or fail, and the output if the command failed.
Hi, your tool looks useful, so I wanted to give it a try. I did:
go get github.com/wagoodman/bashful
And then I made a simple hello.yaml:
tasks:
- cmd: echo "Hello, World!"When i try to run it, I get a panic:
bashful run hello.yaml
Running
• echo "Hello, World!" panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1092945]
goroutine 1 [running]:
os.(*File).Name(0x0, 0x22, 0x1446f0d)
/usr/local/Cellar/go/1.10.1/libexec/src/os/file.go:49 +0x5
github.com/wagoodman/bashful/pkg/runtime/handler.(*TaskLogger).doRegister(0xc420115400, 0xc420099860)and so on.
Am I doing something wrong?
Currently you can do singular groups of parallel/serial tasks. What would be nice is to somehow have a secondary parallel track that runs in parallel to another track. It is not clear how this would be displayed if screen realestate runs out.
Allow a task to have a list of commands that are always run back-to-back in series.
Could be achieved by nested parallel blocks?
Or should there be only one background block allowed?
From @amir20 "You can make it more explicit where you name the env variable context and then say something like env_context: foo_bar. This would allow the variable context to be controlled." on #32
The default behavior now is to add environment changes (from serial tasks) to a single environment context. However, it would be powerful to allow multiple (optional) contexts.
It would be nice to have something like
x-reference-data:
apps: &apps
- app1
- app2
- app3
tasks:
- name: Clone repos
cmd: git clone [email protected]/somewhere/$1.git
for-each: *apps
- name: Build apps
cmd: cd $1 && make
for-each: *appsDocker image: jboss/base-jdk:8
OS: RHEL
Error:
Line: 193 File: /home/wagoodman/go/src/github.com/wagoodman/bashful/screen.go
exit status 1
&{0xc42000c240} Unable to determine screen width.
Firstly, this project seems amazing. I'd love to see where it goes.
I just tried running the bundle command using the sample provide on the README:
tasks:
- cmd: echo "Hello, World!"When I just run bashful run ./sample.yaml, this runs fine
I get the following output:
Bundling ./sample.yaml to ./sample.yaml.bundle
However when I run ./runner, I'm getting:
./runner: line 9: ./bashful: No such file or directory
I tried this with both install variants, go get and trying to pull it through homebrew with the same results.
OS
Mac Sierra (10.12.6)
Go Version
$ go version
go version go1.9.2 darwin/amd64
Let me know if you need any other info
It could be interesting if other applications can import bashful as a library and use it instead of calling out to a binary in another shell.
Currently they continue to run in the background
As seen in #62: ...See log for details (/dev/null) ...
Hello,
I have been trying to automate a linux install script. But I can't manage to enter my password when using 'sudo' in my command. IN this case the script hangs and never finished.
See the code below for a yaml example :
tasks:
- name: 'Basic Tools install'
cmd: sudo apt install -y <replace>
for-each: *basic-tools-names
Would it be possible to add the possibility of entering the input asked for any command ?
Still, this is an awesome project. Thank you for your hard work.
Thanks in advance.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.