PaSh incompatibility regarding exported environment variables about pash

[mgree]

Interesting!

On-the-fly rewriting indeed solves this problem, though I suspect that the hacky approaches we've talked about will do fine. E.g., early expansion will turn $EXP into export and the analysis will see that MYVAR is updated...

[angelhof]

I think that the problem is that the preprocessing step (that replaces parts of the script with calls to PaSh's runtime) does not expand. Since we are not tighly integrated, PaSh now "guesses" which areas could be potentially optimization worthy, and then at runtime, PaSh can expand and decide whether to optimize (but even if it doesn't optimize, we are now running in a different process/shell). Therefore $EXP $MYVAR would be still executed from the PaSh runtime, making the export not propagate to the top process.

Some more "hacky" alternative solutions to address this could be:

• When compiling the DFG back to a script, compile export to a custom export that both exports (so that the script works) but also keeps the exported variables in a log, which can then be used to replay all exported variables at the top script.
• Ending the inside script with bash link
• Is there a way to execute pash_runtime in the same shell (something like source or exec), essentially seeing the changes in the environment that happen in pash_runtime?

[mgree]

Ah. One option would be to have pash_runtime try to detect the issue, and then somehow rollback. Tricky in fully AOT setting.

If pash_runtime were itself a shells script, running source or eval would let us see all of the internal shell state. But the moment you exec (or run a program with execve), only the exported state shows up. Which might be fine in this case?

The replay approach seems like a straightforward enough hack. But a practical question: does this come up? I imagine most of what pash_runtime actually runs ends up in subshells/pipes, which would never affect the outer shell anyway.

[angelhof]

Ah. One option would be to have pash_runtime try to detect the issue, and then somehow rollback. Tricky in fully AOT setting.

Rollback indeed sounds tricky :(

If pash_runtime were itself a shells script, running source or eval would let us see all of the internal shell state. But the moment you exec (or run a program with execve), only the exported state shows up. Which might be fine in this case?

It might be possible (and easy) to make pash_runtime a shell script that we source that contains:

1. A python call to the pash_runtime that outputs the compiled, sequential (unmodified) or parallel (optimized) to a file
2. source this file, executing it in the parent shell.

The replay approach seems like a straightforward enough hack. But a practical question: does this come up? I imagine most of what pash_runtime actually runs ends up in subshells/pipes, which would never affect the outer shell anyway.

In practice we could avoid the most common scenario of export X by not replacing it with a call to the PaSh runtime. But this would leave the corner case I mentioned above ($EXP ...) uncovered. We could certainly not cover this for now though.

Note that in all the above cases, the issue #57 remains and needs to be handled independently.

We could also discuss this in detail tomorrow :)

[mgree]

Let's chat a bit about it tomorrow. I think there are hacks that could resolve #57 (e.g., just exporting all of the local variables to pash_runtime, or doing an analysis to choose the live ones to export, or whatever), but it seems clear to me that the best answer is tighter integration, since it would resolve both of these issues.

[angelhof]

I agree! Let's discuss in detail tomorrow.

[angelhof]

I solved this issue for now with the source method. The PaSh runtime is now a bash script that calls the python runtime. Issue #57 remains.