kvz / bash3boilerplate Goto Github PK

We received feedback by galaktos let's put it to good use!

if [ "${BASH_SOURCE[0]}" != ${0} ]; then

• Unquoted variable access within [ tests… bad idea.

Use a single equal sign when checking if [ "${NAME}" = "Kevin" ], double or triple signs are not needed in Bash

• Except that what you’ve just written has nothing to do with Bash – [ is an external program (the builtin is [[). And the [ a = b ] syntax is specified in POSIX, so I’d be surprised if any other shell “needed” double or triple equals signs there – so why even mention this?

Use {} to enclose your variables in. Otherwise Bash will try to access the $ENVIRONMENT_app variable in /srv/$ENVIRONMENT_app, whereas you probably intended /srv/${ENVIRONMENT}_app.

• Sure, in this case you have to use {}. But why use it when it’s not necessary? That’s just cluttering the codebase with needless extra symbols.

Use :- if you want to test variables that could be undeclared. For instance: if [ "${NAME:-}" = "Kevin" ] will set $NAME to be empty if it’s not declared instead of throwing an error. You can also set it to noname like so if [ "${NAME:-noname}" = "Kevin" ]

• No, this will not set the variable. It will evaluate to that value if the variable is empty, but the variable itself will remain unchanged. The syntax to assign a default value is ${parameter:=word}.

https://github.com/mvdan/sh

Possible implementation here:
https://coderwall.com/p/wil8da/debug-bash-scripts-with-line-numbers-more-info

I'd like to provide some info for the options after "--" in my script.

Ideally, I would do this by writing it in the usage doc, but it gets parsed.

I'd like a way for lines to be ignored by the parser.

I see examples for setting defaults for arguments with options, what about arguments without options?

This is a feature request.

Having the script automatically parse some sort of configuration file (json,xml,yaml,bash) that dictates default values for its arguements would nice.

Why not just use default parameters?

Assume a case where the scripts are written for a multitude of runtimes (say for e.g; servers) and the default parameters are meant to be different for each runtime.

maybe we should add PR templates to remind people...

Example:

read -r -d '' __usage <<-'EOF' || true # exits non-zero when EOF encountered
  -s --subject     [arg] Specific subject files to process.
  -v --verbose           Enable verbose mode for all scripts.
  -d --debug             Enables debug mode.
  -h --help              This help page.
  -n --dry-run           Don't submit any jobs.
  -r --reg-command [arg] Provide an alternative registration command. Default="mb_register.sh"
  -f --factor      [arg] Scaling factor for time and memory estimates. Default="1.15"
EOF
read -r -d '' __helptext <<-'EOF' || true # exits non-zero when EOF encountered
  MAGeTBrain implementation using ANTs
  Supports MINC and NIFTI input files (ANTs must be built with MINC support)
  Invocation: mb.sh [options] -- [stage 1] [stage 2] ... [stage N]
  Standard stages: template, subject, resample, vote, run (template, subject, resample, vote)
  Multiatlas stages: multiatlas-resample, multiatlas-vote, multiatlas (template, multiatlas-resample, multiatlas-vote)
  Other stages: init, status, cleanup
  Multiple commands will run multiple stages. Order is not checked.
EOF

User tried to do:

$ mb.sh --template
#Instead of
$ mb.sh -- template

There was no error thrown for the invalid command-line option "--template"

No sure if this behaviour is intended. I'm using bash4. This is what I had to do to work around it: QwertyZW@0cdc704

qwertyzw% ./example.sh       

(18-06-25 6:20:41) <1> [~/bash3boilerplate]  
qwertyzw% PS4='Line ${LINENO}: ' bash -x ./example.sh
Line 31: read -r -d '' __usage
Line 39: true
Line 42: read -r -d '' __helptext
Line 46: true
LLLine 49: dirname ./example.sh
LLine 49: cd .
LLine 49: pwd
Line 49: source /home/alkhishm/bash3boilerplate/main.sh
LLine 19: set -o errexit
LLine 21: set -o errtrace
LLine 23: set -o nounset
LLine 25: set -o pipefail
LLine 29: [[ /home/alkhishm/bash3boilerplate/main.sh != \.\/\e\x\a\m\p\l\e\.\s\h ]]
LLine 30: __i_am_main_script=0
LLine 32: [[ -n x ]]
LLine 33: [[ ./example.sh = \.\/\e\x\a\m\p\l\e\.\s\h ]]
LLine 34: __i_am_main_script=1
LLine 37: __b3bp_external_usage=true
LLine 38: __b3bp_tmp_source_idx=1
LLLLine 47: dirname ./example.sh
LLLine 47: cd .
LLLine 47: pwd
LLine 47: __dir=/home/alkhishm/bash3boilerplate
LLLine 48: basename ./example.sh
LLine 48: __file=/home/alkhishm/bash3boilerplate/example.sh
LLLine 49: basename /home/alkhishm/bash3boilerplate/example.sh .sh
LLine 49: __base=example
LLLine 50: printf %q /home/alkhishm/bash3boilerplate/example.sh
LLLine 50: (( 0 ))
LLine 50: __invocation=/home/alkhishm/bash3boilerplate/example.sh

(18-06-25 6:20:44) <1> [~/bash3boilerplate]  
qwertyzw% git checkout master
Previous HEAD position was 327cba7... Add magic variable that stores the full command line invocation (#99)
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.

(18-06-25 6:20:48) <0> [~/bash3boilerplate]  
qwertyzw% ./example.sh 

 Option -f (--file) requires an argument

  -f --file  [arg] Filename to process. Required.
  -t --temp  [arg] Location of tempfile. Default="/tmp/bar"
  -v               Enable verbose mode, print script as it is executed
  -d --debug       Enables debug mode
  -h --help        This page
  -n --no-color    Disable color output
  -1 --one         Do just one thing

 This is Bash3 Boilerplate's help text. Feel free to add any description of your
 program or elaborate more on command-line arguments. This section is not
 parsed and will be added as-is to the help.

This may be the same bug as #5 and I think I have found a fix.... but it's possible they are distinct.

./main.sh -f foo -t should populate arg_t with the value parsed from the usage string default. However, opt is set to ?, the help function is called and getopts aborts.

As detailed on this SO post you can actually put getopts into silent mode an explicitly test for missing option arguments. Here is some more info from bash hackers wiki: http://wiki.bash-hackers.org/howto/getopts_tutorial#error_reporting

This technique will allow b3bp to not barf when optional option arguments are missing.

This is a follow-up to Issue #35

easy fix, just need an ||true

We use lots of bashisms, for good reason.

Perhaps we should check and bail out if a non-bash shell is running the script?

One way to check: https://askubuntu.com/questions/97522/how-can-my-script-determine-whether-its-being-run-by-bash-or-dash

I noticed that the logging colors are defined in the __b3bp_log function and that they are set every time the function is called. Highly unlikely it have a noticeable impact on performance, but debugging stuff is terrible with the overhead. Is it necessary to redeclare them on each call? Of course I can move them outside myself, but if there is a specific reasoning for this structure I would like to hear it.

As Browsing_From_Work mentioned on https://www.reddit.com/r/programming/comments/4parog/bash3_boilerplate_template_for_writing_better/d4ji2yh there are some good arguments for it. For convenience, the comment also points to this page.

I see a lot of pros, and only a possible con that we might have more explaining to do. But I think we can figure out that part fine. It might also save us some explaining of pitfalls with [

We are already using it in order to do pattern matching to figure out the OS family

Thoughts?

e.g.

IFS=$'\n'
for line in $(cat many-lines.txt); do
  echo "${line}"
done
unset IFS

We can trap CTRL-C and call a function in bash:
https://rimuhosting.com/knowledgebase/linux/misc/trapping-ctrl-c-in-bash

# trap ctrl-c and call ctrl_c()
trap ctrl_c INT

function ctrl_c() {
        echo "** Trapped CTRL-C"
}

I don't have a use case right now, but it's definitely helpful to allow graceful cleanup of an interrupted script.

Hi, and thanks for your work, this is really helpful :)

I have a little question, how would you do for short and long argument name.
Let's say I have an argument filename, and you can either use it with -f or --filename. It is also required.
Would you have a quick way to adapt you boiler plate to fit that?

Thanks a lot again :)

Pierre

It's there in the examples but the existing color code doesn't check/use it.

I'd like to add arguments for some features that have the same starting letter:
--walltime-template
--walltime-subject

Right now, short options are limited to a single letter, so it's not clear how I can expose these.

It would be nice to be able to have better control over the usage statement. In particular:

• Option flag descriptions spanning more than one line
• More control over general usage statement formatting and content
• Additional, non option text, such as usage examples etc.

It's possible that the last point is a non-issue, I need to examine the usage text parsing more carefully.

The first point will need some different usage statement parsing logic. It's possible that some sort of special line break character or string could be used, and then when the usage statement is parsed for options, consecutive lines containing the special character/string would be concatenated, but when the usage statement is printed a line break is inserted at that character. An easy candidate could be \n which when echoed with echo -e would insert a line break. Thoughts?

Not sure what to do about the second point yet...

templater.sh does its magic trying to use sed.
this works out fine as long as there is no surprise in any env var.
this cannot even really be fixed as such because really sed is the wrong tool for this job.
possibly use perl where quoting (\Q and \E) can be used or do as most others suggest and use eval+cat.
or at least try to replace only vars which actually get referenced in the template and live with the fact that if this variable has unexpected content stuff breaks - right now a sed command is issued for each existing env var. this is bound to break no matter what.

opinions? remarks? ideas?

The second URL listed under "More info" in main.sh appears to be broken.

It would be cool to have basic tests ran on Travis CI.
Bonus points for running against multiple bash versions and GNU/BSD tools. Unsure yet how to do that tho.

I did not realize until after some trial and error that the best place to insert the code within the script is within the function that you want to export. I was having issues with the exporting the function. I think a recommendation within the read me on where to insert the boiler plate code would help others would reduce quick start friction:

#!/usr/bin/env bash

function foo(){
// Boilerplate code
}

#Recommended export code already outlined within the Function Packaging section of the  read me

if [ "${BASH_SOURCE[0]}" != "${0}" ]; then
  export -f foo
else
  foo "${@}"
  exit $?
fi

As Vaphell suggests:

on top of what parent said the [[ ]] / (( )) dichotomy is really useful because it allows for 2 visually distinct modes of work which reduces the cognitive cost:
text [[ ]] with patterns regexen and shit
ingeger math (( )) using C syntax lookalike for int math and comparisons.
(( c++ ))
(( x = y + 3 )) # equivalent to x=$(( y + 3 ))
if (( RANDOM%17 == 4 ))
text is assumed to be a variable, so $ can be dropped for additional readability ($ is required only if the > expression is supposed to make use of positional params to disambiguate 1 and $1
[ ] is everything crammed together, using less than stellar switch based syntax.

Perhaps we have use cases already where we want to apply this, if not a deciding on one form and adding that to the README/website would be nice

My sense is that bash3boilerplate has at least two distinct but related goals: (1) provide some useful code that others might copy, paste, and modify/include and (2) demonstrate best practices. Toward the latter end, it would be great to have a style guide that explains come of the choices (and any related tradeoffs) employed in bash3boilerplate codes. Alternatively, style questions could be added to the FAQ, but I think of the FAQ as being more about how to use bash3boilerplate and a Style Guide being about how a user can write their own better bash 3 code. Either way, here are a couple of topics:

1. Single- vs. double-bracket tests:
   main.sh uses double-bracket tests in some places such and single-bracket tests in other places. For example, line 91 uses both. It would be great to have a style guide that explains such choices.
2. Curly brace variable evaluation:
   A stack overflow page provides a good explanation of why it's a good practice to always use curly braces when evaluating a variable (e.g., ${foo})

P.S. With a Style Guide, you guys probably would have the beginnings of a great book. :) It probably wouldn't have to be very long (e.g., 25-50 pages) to be useful. In fact, it's probably more useful if it's short and can be devoured quickly. The content could be driven completely by going through bash3boilerplate and detailing the decisions you made at each point in its development.

Folks,

It looks like the latest version of shellcheck complains about a few things in main.sh. Maybe SC#### numbers have changed for some of these things that might need to be skipped?

`
$ shellcheck main.sh

In main.sh line 65:
local color_debug="\x1b[35m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".

In main.sh line 67:
local color_info="\x1b[32m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".

In main.sh line 69:
local color_notice="\x1b[34m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".

In main.sh line 71:
local color_warning="\x1b[33m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".

In main.sh line 73:
local color_error="\x1b[31m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".

In main.sh line 75:
local color_critical="\x1b[1;31m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".

In main.sh line 77:
local color_alert="\x1b[1;33;41m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".

In main.sh line 79:
local color_emergency="\x1b[1;4;5;33;41m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".

In main.sh line 84:
local color_reset="\x1b[0m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".

In main.sh line 86:
if [[ "${NO_COLOR:-}" = "true" ]] || ( [[ "${TERM:-}" != "xterm"* ]] && [[ "${TERM:-}" != "screen"* ]] ) || [[ ! -t 2 ]]; then
^-- SC2235: Use { ..; } instead of (..) to avoid subshell overhead.

In main.sh line 252:
((OPTIND+=_b3bp_tmp_has_arg${__b3bp_tmp_opt}))
^-- SC1105: Shells disambiguate (( differently or not at all. For subshell, add spaces around ( . For ((, fix parsing errors.
^-- SC2030: Modification of OPTIND is local (to subshell caused by (..) group).

In main.sh line 269:
shift $((OPTIND-1))
^-- SC2031: OPTIND was modified in a subshell. That change might be lost.

In main.sh line 383:
info "$(echo -e "multiple lines example - line #1\nmultiple lines example - line #2\nimagine logging the output of 'ls -al /path/'")"
^-- SC1117: Backslash is literal in "\n". Prefer explicit escaping: "\n".
^-- SC1117: Backslash is literal in "\n". Prefer explicit escaping: "\n".
`

Is it intended that only an "emergency" causes an exit?
I would expect anything above critical should immediately exit.
From the docs:
"Safe by default (break on error, pipefail, etc.)"

See, for example, https://travis-ci.org/kvz/bash3boilerplate/jobs/110544645

I propose the text below as a new FAQ question. If you agree, please feel free to copy, paste, and edit it or let me know if you want me to branch, commit, and submit a pull request.

How do I access a potentially unset environment variable?
The set -o nounset line in main.sh causes error termination when an unset environment variables is detected as unbound. To avoid this, declare all environment variables before use, e.g.,

#!/usr/bin/env bash
# Include main.sh (NOTE: comment final line before executing this script):
. main.sh
declare FOO
# Testing FOO below would cause termination with a non-zero exit code without the above declaration
if [[ -z ${FOO} ]]; then
  info "FOO is empty."
else
  info "FOO has a value"
fi

The line

if [ "${NO_COLOR}" = "true" ] || [[ "${TERM:-}" != "xterm"* ]] || [ -t 1 ]; then

seems to be wrong really in at least 2 different ways.

1. It checks STDOUT to be connected to a terminal while logging happens on STDERR only.
2. It turns OFF colors if STDOUT is actually connected to a terminal.

I'm guessing here but a quick check seems to suggest that nobody noticed until now because __b3bp_log always got called in a subshell which actually was not connected to a terminal. If the function would be just called - as suggested by shellcheck - the code would not work as expected.

Now I can fix that, but before that I wanted to ask if there is something about this code, or the decisions leading to this code, which I overlooked or I don't understand.

Allow debug info with line number and function name when ERR signal is raised:

1. http://unix.stackexchange.com/a/39660/84740
2. http://stackoverflow.com/a/25380229/403516

You can't just assume __os="Linux" (

Suggestion: use uname(1) instead.

https://github.com/livibetter-backup/template.sh/blob/master/template.sh

I'm testing your script and it is really useful. However, I found one problem, not sure if I am using it wrong or it's actually a bug.

When you use a tilde (~) in paths, it seems that is converted to something else (although when you print it, it looks OK).

For example, take your main and place a file in home (~) and try to access it and all the commands will fail.

[ -f ${arg_t} ] && echo "temp ${arg_t} exist" || echo "temp ${arg_t} does not exist"
stat ${arg_t}

And I set the read block as

-t --temp  [arg] Location of tempfile. Default="~/foo"

But if you set it to something else, without the tilde it works.

The __os variable is set incorrectly on Windows (using git bash). Recommend adding this to the boilerplate.

if [[ "${OSTYPE:-}" == "msys"* ]]; then
__os="Windows"
fi

It would be great to have a FAQ or User Guide describing some common use cases. I'd like to know how to add a new command-line flag and have spent at least of couple of hours trying so far. Is there an example somewhere?

I had the idea to create a simple website for this under e.g. the bash3boiletplate.sh domain name using GitHub pages. As a first iteration it would render the README.md to HTML so it could serve as a simple single front page. But I feel it would give the project some more substance, what do you think?

Right now, if I want to provide multiple inputs to an argument I have to do this:

-s "arg1 arg2"

It would be nice if it instead could be:

-s arg1 -s arg2

Or possibly,

-s arg1 arg1

See 9350e05#commitcomment-30019068

This change breaks the following call:
script.sh --long value --switch
The value of arg_l will be "value--switch" instead of "value", and arg_s will not be defined.

Cause is in line 242ff, where OPTIND is advanced by the number in has_arg, which before was either 0 or 1, but now 2 is possible and has a special meaning but also affects this operation: ${@:OPTIND:${!__b3bp_tmp_varname}}

Following calls are unaffected, which is why the tests don't break (insufficient test cases ;)) and may explain why this hasn't been reported yet:
script.sh --switch --long value (only last option has value)
script.sh --long=value --switch (using "long-option=value" syntax)
script.sh -l value --switch (single-character options are unaffected for some reason, though I'm not sure why...)

My personal solution is to use a separate (bool) variable instead of extending the has_arg by another value: _b3bp_tmp_requires_arg${__b3bp_tmp_opt:0:1}

Not sure how hard this would be to implement, but parsing the usage block to find the "Required" arguments would eliminate the extra code at line:
https://github.com/kvz/bash3boilerplate/blob/master/main.sh#L234

Hi @kvz,
First of all I want to say, your project is awesome, and you deserve credit, recognition and attribution for it! 💯 However, I am wondering if adopting a CC0 or, IMO less desirable, the unlicense might help promote adoption. Since this is intended to be a boiler plate, it is not clear to me, exactly how and under what conditions the MIT license needs to be preserved and distributed with projects that bootstrap shell scripts with bash3boilerplate. Part of this is my admitted inexperience with software licenses, but I also feel like this is a "ship of Theseus" problem at what point does borrowing components from the script require that the MIT license be included, referenced etc.

I certainly defer to your judgement here, this issue is only intended to prompt discussion and reconsideration of licensing concerns.

As mhurron mentions, it might be a good idea to adopt / advise on linting

A nice optional feature would be the ability to log debug/info/notice/warn/error/critical/alert to file.

https://github.com/sstephenson/bats

We should consider recommending this in best practices, as well as using it for our own tests

See discussion in PR #81