Allow custom env variables to be set about bashful

[wagoodman]

I've been wanting to incorporate this exact functionality, but am not certain yet how to do that easily. Maybe adding a source keyword that would run a command and incorporate all new environment variables:

tasks:
  - name: Updating env
    source: eval (docker-machine env machine-name)
    
  - name: Test
    cmd: env | grep DOCKER

Or simply export all environment variables in each cmd statement and somehow inject it into the next bash environment.

The problem now is that each task is executed in a subshell of it's own.

[amir20]

The problem now is that each task is executed in a subshell of it's own.

That probably needs to be fixed first :) I myself have had to do this by reading the env variable first and then providing it to the other commands.

I like the idea of source: KEY=VALUE but I am not sure how it would work with docker-machine env machine-name

[wagoodman]

Ok, my current thought is to keep the subshells --attempting to share a single shell seemed like the way to go initially, but you get stuck essentially writing a bash parser and evaluator in the end. However, I can get subshells to show the environment they are in after execution, and keep it separate from stdout/stderr by passing an os.pipe write file via exec.Command.ExtraFiles and read it after execution is done. The subshell could exfiltrate the env vars by appending a ; env >&3 onto the end of the given user cmd, writing all env vars to the given pipe fd (I'm not worried about the user injecting a malicious cmd since the user already has total control over commands given).

So as commands are run serially, the environment is passed from one task to the next. This implies that there isn't a way to achieve this with parallel tasks, it would be race condition galore. You would use it just as you'd expect... the same way you'd inject env vars into your current environment:

tasks:
  - name: Updating env
    cmd: eval (docker-machine env machine-name)
  
  - name: Updating SOMETHING env var
    cmd: export SOMETHING=sweet

  - name: Updating OTHER env var
    cmd: source ascript.sh    # <--- this exports env the $OTHER var

  - name: Test
    cmd: echo $DOCKER   $SOMETHING   $OTHER

It also doesn't introduce another keyword into the yaml syntax and uses current bash primitives the way you'd expect.

@amir20 I think I'll get started on this tomorrow unless you had any other thoughts or suggestions to add for this feature.

[wagoodman]

Actually, it was an easier lift than I originally thought. @amir20 , you wanna try out the POC and let me know if this fits your needs? Otherwise I'll probably merge it this weekend if I don't see other problems.

https://github.com/wagoodman/bashful/compare/keep-env-vars-across-tasks

[amir20]

Nice. So I tried it with the following file:

tasks:
  - name: Building images
    cmd: docker-compose build
    
  - name: Pushing images to docker hub
    cmd: docker-compose push
          
  - name: Updating env variables to clashstats
    cmd: eval (docker-machine env clashstats)
  
  - name: Pulling to remote box
    cmd: docker-compose pull

And for some reason I am getting index out of bound exceptin:

⨉⪧ ~/go/src/github.com/amir20/clashstats on master * ⟩ bashful run test.yml                                                                                                   
Running test.yml
    • Building images                                                                                                                                                               
    • Pushing images to docker hub                                                                                                                                                  
    • Updating env variables to clashstats                                                                                                                                          
    ⠼ Pulling to remote box     Status: Image is up to date for amir20/clashleaders:latest                                                                                          
      75.00% Complete  Tasks[3/4]                                                                                                                   Runtime[00:00:45] ETA[00:00:-45]
panic: runtime error: index out of range

goroutine 2638 [running]:
main.(*Task).runSingleCmd(0xc4201a0800, 0xc420172360, 0xc4201a09d0, 0xc42015f140)
	/Users/amirraminfar/Desktop/bashful/task.go:570 +0x1473
created by main.(*Task).StartAvailableTasks
	/Users/amirraminfar/Desktop/bashful/task.go:618 +0x16d

[amir20]

Also during eval step, I see an error. Here is a screenshot.

[wagoodman]

Looks like I was wrapping env vars with quotes, which made it fail in most cases, and I wasn't considering blank env vars. I've updated the branch with the fixes. (not certain about the error while reading file '-' yet...)

[wagoodman]

Another thing, it would be great if you could source a script with common bash functions and have them persist to different tasks. Extracting functions via declare -pf and injecting them before script execution could be the way to go here.

[amir20]

I think that worked! I am at work so I can't test it with a real server. But the error is gone.

I was thinking about it this morning and I think you would need to think about the following when reading the vars:

1. Read vars from a file (sourcing)
2. Read vars from an output of a command (like above)
3. Read vars from inline command like VAR=VALUE

When executing or needing the variable, I was thinking of two options.

1. After sourcing the env variables, all subsequent commands will have those env vars set. The only issue here is that you may need to add support for deleting variables or resetting.
2. You can make it more explicit where you name the env variable context and then say something like env_context: foo_bar. This would allow the variable context to be controlled.

Either way, great job. This at least let's me throw away my crappy deploy script and use bashful for now. 👍

Let me know if you need anything else.

[wagoodman]

Since the source of truth here is the environment itself (according to the env cmd) then I feel confident that the cases you mentioned are covered (except for inline variable expressions I don't believe they persist in the environment after the given command exits, so I think it should be left off).

I've merged whats there so far (1012bef)

I like the idea optional env contexts, that could be quite powerful for folks that like keeping particular environments clean... I'll open up another feature for this