codekitchen / pipeline

the `pipeline` shell command

The pipeline shell command

A utility to make building up a pipeline of shell commands easier, especially when doing data exploration.

If you've ever found yourself writing shell code, in an endless loop of piping output to less, scanning it over and making changes, then pipeline can make your life just a little bit more beautiful.

This is just a thin wrapper around your shell, not some totally new data mining tool. Launch pipeline, and start typing shell commands as usual. Every time you hit enter you'll see a one-screen preview of your output, similar to piping output to less, but your cursor will stay right where it was for further editing.

Ctrl-C when you're done.

Installation

Pipeline depends only on ncurses and readline (or libedit), both of which ship with MacOS and most common Linux distros.

MacOS

Install with Homebrew.

brew tap codekitchen/pipeline
brew install pipeline

From Source

You can download the latest release tarball from the releases page, or git clone the repo to build the master branch.

autoreconf -fi    # only if building from git, skip this for release tarballs
./configure
make

After make finishes, you'll be able to use ./pipeline. You can also install it using:

sudo make install

Windows

It might work under MinGW/GitBash? Please let me know.

pipeline's Issues

Allow passing the command line as a command line argument to pipeline

Since pipeline has no history (and that's, I think, intended/good/desired/good/verygood), sometimes it's a pain I have to reselect some text from the terminal just to put it in another session of pipeline.

Passing the command line to pipeline would be handy. Running pipeline on a command which is not doing what we want would be almost as easy as enclosing it in single quotes and prepending pipeline.

And it would have the advantage that that command is saved in the shell history (sure, if I modify the command once in pipeline, then there's no history for that).

Well, maybe another "sibling" improvement would be a key binding to exit pipeline and populate the command line with the last edited line.

Just some ideas. :)

INSTALL LINUX

I know this is a dummy question, but is there a way to install it like with apt on a wget bash script other than from the source?

Regards,

gr00t

pipeline ignores shell initialization routines

Things like custom aliases and functions are not available in the pipeline.

This is what is problematic with current approach of hardcoding /bin/sh instead of using current shell. Not sure if the real shell can be used instead?

Hitting enter when the cursor is on the first <length-of-the-editing-mode-in-the-prompt> character on the second visual line when line wrapping occurs

Below's the demo. To experiment, it is enough to have set show-mode-in-prompt on in the ~/.inputrc file. (I've tested both with editing-mode set to vi and emacs.)

Suggestion: showing in the recap bar (?) whether the file ends with a newline

I was stung by not knowing this here.

Jokes apart, I'd be very useful if that info bar was provided. All lines end with \n?

 4 lines, showing 4 (POSIX-compliant file)

As before, but the last line has no line terminator?

 4 lines, showing 4 (non-POSIX-compliant file)

All lines end with \r\n except the last?

 4 lines, showing 4 (typical Windows file)

Some lines end with \n other with \r\n?

 4 lines, showing 4 (file with Windows and Unix mixed line endings)

All other cases?

 4 lines, showing 4

Something like this I guess can be easily handled by having some function return an enum telling which case it is.

Copying the current line to clipboard or primary selection without using the mouse

Premise: Mice bring deseases.

Your program uses the readline library, so it allows me to take full advantage of the vi editing mode that I use, except that pressing v does not open the editor to edit the command. This, by itself, is not a problem, since your program is just for that, editing the command easily.

However, when I want to copy the command from the shell to somewhere else (e.g. here), I have to approaches:

• selecting the command with the mouse (to have it stored in the primary selection), which is against the premise;
• using this approach (to have it stored in the primary selection and/or clipboard, as I like), which assumes that I can edit the command in vi when pressing v.

I don't know if it's possible for you to allow in-editor editing of the command. Maybe you have another way to copy the command to the clipboard or primary selection?

Corrupted terminal when running a command which goes in loop

Are you able to reproduce the following? I forgot to write in the screencast the following:

• the bash: i: command not found error is because I just pressed iEnter;
• when you see the prompt copied horizontally, that happens when I hit Enter.

issues with long lines

There's a cheap effort to truncate long output lines so they don't wrap, but it counts bytes not characters (ya ya I know), so I'm sure it's easy to break. And there's the larger question of whether truncation is better, or should we just wrap and figure out how to still count lines properly?

There's also definitely issues with the command line being longer than one screen width, too.

Consider side-effect protection

If anything in the pipeline touches e.g. filesystem, destructive changes can be inadvertently executed. I think it would be a nice feature to execute in a "protected" environment by default, although this won't be easy to achieve of course.

One can attempt to intercept syscalls using ptrace or other debugging interfaces in the subprocess, but this is incredibly messy. A cleaner approach might be to execute in a container with root fs mounted from outside and an overlayfs on top preventing any write access.

This is tricky and is guaranteed to have potential holes; we can never support 100% destruction-less environment while simultaneously exposing real OS. Maybe we can protect the FS but that's not subset of syscalls that would allow for screw-ups: signals, shared memory, network and many more would need to be addressed and i'm sure there'd be more things that could be missed.

Is this even worth considering given these complications? I immediately wanted to implement this since I often iterate on pipelines that manipulate file collections, but I realize now FS is just a part of it, and providing some protection may be worse than just being honest with user and letting them shoot themselves in the foot.

output incorrect when the command output is just a single newline

Reported in #7, this is easy to reproduce by typing just echo or echo '' into pipeline. The prompt gets shifted to the next line, leaving an extra copy of the prompt on the screen.

Problems with vi editing mode

First of all, thank you for this "small" useful tool, which I read from here a few minutes ago for the first time.

Thanks to my habit of using vi editing mode in bash, I'm quick enough in command line editing and history navigation, however your tool can make a difference in doing it, especially when it comes to answering some sed/awk/grep/... related question of StackOverflow, for instance.

Unfortunately there are some bit which prevent a productive use with the set editing-mode vi setting in .inputrc.

When that option is set, hitting enter when in command mode (this happens frequently, if you finished the edit by pressing Escape, D, x, or any d-command) exits pipeline by leaving the terminal cluttered.

Terminal emulator: urxvt

$ bash --version | head -2
GNU bash, version 5.0.16(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2019 Free Software Foundation, Inc.

Please, find attached at the bottom a terminal screencast demonstrating the issue.

I've noticed also other issues, which are possibly (and hopefully) all related to a common underline flaw (or, more hopefully, to some configuration of mine which conflicts with the program?):

• If I type just echo, but also if I type echo '', and hit Enter the line gets printed; I suspected this might be related to the zero-lines-of-output, but this is unlikely, as echo -e 'a\nb\nc' | sed -n '/d/p' works fine.
• This sounds a bit vudù to me, but if I remove the option set editing-mode vi line from my .inputrc, open a new terminal, and I hit Escape then Enter... the editing mode changes to vi, for that session of pipeline!

For all the above, if it helps, I can make screencasts demonstrating the issue.

show the full output on exit

The original prototype of pipeline captured CTRL-C and showed the full command output on exiting. I think this is a nice touch, it makes it feel a bit more integrated with the parent shell.

I've punted for now because I'm not sure I like any of the options for making this happen.

error in the line makes pipeline eat lines upward

I think this is closely related the first bullet point in my #7, and therefore to your #8, but it happned in d7181e2 already.

Try just entering cat non_existent_file. It should eat past terminal ouptut.

The following screencast shows the repro steps, but not the effect. (It must be a bug in asciinema!)
In my case the effect is that line goes up, "eating" the previous line of the terminal

As a side note, wouldn't be reasonable to handle Ctrl-D the same way you handle Ctrl-C? I mean, there's not much we want to do, right?

Command line longer than the terminal (wrapping on two lines) makes <Enter> copy the line, thus filling up the terminal

(I'm so bad at choosing issue title; feel free to edit it.)

Using urxvt:

Originally posted by @Aster89 in #2 (comment)

support terminal window resizes

We can catch SIGWINCH to know about the resize, and there are system calls to get the new sizes (I think ncurses might provide wrappers for these?)

I wouldn't re-run the current pipeline on resize, just display some blank lines if necessary until the next run.

I'm not sure yet if we'll need to do something more involved in order to process the resize while playing nicely with readline/libedit.

future plans

Not a lot of future plans, it's a simple idea and it does what I want it to do. Definitely some bugs to fix.

I've considered implementing caching of early steps in the pipeline, so that long steps or side-effecty steps can be cached and not re-run each time. But I'm not convinced it's worth it, and there'd be some UI issues to solve.

If this ends up being a widely useful tool, I do think there's advantages to making it a feature of the shell itself rather than a separate program. But I'm not sure how well that idea would be accepted, for any given shell.