Git add --patch does not use diff-so-fancy about diff-so-fancy

[elboletaire]

I got this properly working, until I've updated ubuntu to 18.04 (which comes with git 2.17.1). Since the upgrade, using git add -p fails with the following message:

fatal: mismatched output from interactive.diffFilter
hint: Your filter must maintain a one-to-one correspondence
hint: between its input and output lines.

All the other usages of diff-so-fancy work as expected, and I wasn't able to find more information on this specific error. Anyone else have experienced this?

Should I open a new issue for this?

[tomviner]

Git 2.9 is now released, and interactive.diffFilter is mentioned in the release notes.

Add like this:

git config interactive.diffFilter "diff-so-fancy | less --tabs=4 -RFX"

Looks like this:

Stage this hunk [y,n,q,a,d,/,e,?]? y

i.e. #167

[wren]

I've been using the below config for months now on a daily basis, and haven't had any big problems using git add --patch. The only caveat I've run into is that when I delete an entire file, the header looks a little off (the filename is above the horizontal rulers). For me, that's very minor and definitely worth having d-s-f for all my other git add --patch needs.

I realize there are some cases where this might not work (as mentioned in above comments), but I apparently don't run into them in my daily workflow. YMMV 🤷‍♀

[pager]
  diff = diff-so-fancy | less --tabs=4 -RFX --pattern '^(Date|added|deleted|modified): '
[interactive]
  diffFilter = (echo && diff-so-fancy)

The key is the echo before invoking d-s-f, to appease Git with the correct number of lines.

[wren]

Took a few days off, but got back to this today.

First a correction; my example above has a mistake in it. The example still works, but I have it labeled as "no newline" when there is, in fact, one newline required to repro the problem (fun fact: the entire vi family of editors including neovim silently insert a newline for you, and it's actually quite difficult to tell it to stop being overly helpful).

Anyway, here's the new example:

diff --git a/dsf-repro b/dsf-repro
index 53f1df8..883a45c 100644
--- a/dsf-repro
+++ b/dsf-repro
@@ -1,4 +1,4 @@
 line one
 line two
 line three
-line four
+line four is edited

And here's a command to make the diff completely unambiguous (lines are numbered,$ is a newline, and ␠ is a space):

$ cat -A dsf-repro | sed 's/ /␠/g' | cat -n
     1  diff␠--git␠a/dsf-repro␠b/dsf-repro$
     2  index␠53f1df8..883a45c␠100644$
     3  ---␠a/dsf-repro$
     4  +++␠b/dsf-repro$
     5  @@␠-1,4␠+1,4␠@@$
     6  ␠line␠one$
     7  ␠line␠two$
     8  ␠line␠three$
     9  -line␠four$
    10  +line␠four␠is␠edited$

So, I put the cat and sed commands above into my git config for interactive.diffFilter to see what was going on, and I think I found the problem.

$ cat dsf-repro | diff-so-fancy | sed -n 'l 999' | sed 's/ /␠/g' | cat -n
     1  \033[0m\033[38;5;227m----------\033[0m$
     2  \033[38;5;227mmodified:␠dsf-repro$
     3  \033[38;5;227m----------\033[0m$
     4  \033[1;35m@␠dsf-repro:1␠@\033[1m\033[0m$
     5  line␠one$
     6  line␠two$
     7  line␠three$
     8  \033[1;31m\033[1;31mline␠four\033[m\033[1;31;48;5;52m\033[m\033[1;31m\033[m$
     9  \033[0m\033[1;32m\033[1;32mline␠four\033[m\033[1;32;48;5;22m␠is␠edited\033[m\033[1;32m\033[m$
    10  \033[0m$

The extra line at the end just has the one character, and it's a color reset character. This explains why no newline at all, and two newlines at the end of the file both work: both of these scenarios mask the extra reset character so that it doesn't end up on its own line by itself.

So, after digging into the dsf code, I found where the character is being inserted, and moved it before the newline:

diff --git a/diff-so-fancy b/diff-so-fancy
index dc04ac9..0945ec8 100755
--- a/diff-so-fancy
+++ b/diff-so-fancy
@@ -544,11 +544,12 @@ sub strip_leading_indicators {
 	if ($manually_color_lines) {
 		if (defined($5) && $5 eq "+") {
 			my $add_line_color = get_config_color("add_line");
-			$line              = $add_line_color . $line . $reset_color;
+			$line              = $add_line_color . $line;
 		} elsif (defined($5) && $5 eq "-") {
 			my $remove_line_color = get_config_color("remove_line");
-			$line                 = $remove_line_color . $line . $reset_color;
+			$line                 = $remove_line_color . $line;
 		}
+		$line =~ s/^(.*)([\n\r]+)$/${1}${reset_color}${2}/;
 	}
 
 	return $line;

This fixes the problem as far as I can tell. The full test suite still passes, and I haven't noticed any negative side effects (it actually still passes even if we just remove $reset_color without moving it, so 🤷).

With this fix in place, I think it would be easy to add a patch mode that pads the header with blank lines depending on the type of diff. I'm happy to make a PR with this fix, a patch mode, and proper tests and whatnot, but I wanted to check if this is an acceptable approach first. Is there any reason this wouldn't get accepted?

[reegnz]

The solution provided by @wren works for me (with the echo).

[core]
	pager = diff-so-fancy | less --tabs=4 -RFX
[interactive]
	diffFilter = (echo && diff-so-fancy)

The versions I am using:

$ git version
git version 2.25.0
$ diff-so-fancy --version
Diff-so-fancy: https://github.com/so-fancy/diff-so-fancy
Version      : 1.2.6

But I do always make sure to have a newline at the end of my files.

EDIT: verified, when deleting the newline at the end of a file, it breaks.

But anyway you should be using newlines at end of files: https://stackoverflow.com/a/729795/205318
A bunch of unix tools don't behave reliably if you don't. Not sure if this is the fault of d-s-f.

[paulirish]

It works!

---

... however it looks like we're getting an error..

Use of uninitialized value $_ in print at /Users/paulirish/code/temp/git/git-add--interactive line 1364, line 1.

it could very likely be however we're handling the output. cc @scottchiefbaker .. you can see the hunk headers are a bit confused:

(on the left i'm using cat as my diffFilter)

[samcv]

I have git version 2.14.1 and I can get git add -p working if I set:

[interactive]
diffFilter = "diff-so-fancy"

Can this be added to the main README? Thanks! Then this issue can probably be closed.

[scottchiefbaker]

After I figured out how to pass ARGV from Bash in to Perl it was pretty trivial. You'll need to set a config option as @OJFord suggested:

git config interactive.diffFilter "diff-so-fancy --patch-mode | less --tabs=4 -RFX"

After you have that in place, please test the branch I created here:

https://github.com/scottchiefbaker/diff-so-fancy/tree/patch-mode

It "works for me" when I tried git add --patch with git 2.9.0, but I only tested a small patch.

[wren]

After seeing the comments on here, and doing some local testing, I suspect the problem is that the problem diffs don't contain a newline at the end of the diff. It seems that there's a difference from git in how d-s-f handles no newline at the end of file (it adds a newline for you). I have my editor set to warn me when I don't have a newline at the end of a file, which is why I don't run into this.

This seems like it would be fixable in d-s-f, though, without having to wait for an upstream fix in git (like some of the other problems with add --patch).

[HaleTom]

Until d-s-f fixes its header line equivalency, this works wonderfully:

[interactive]
  diffFilter = "delta --diff-so-fancy"

https://github.com/dandavison/delta

[OJFord]

@scottchiefbaker It's less than ideal, but since it's a different configuration option we could have a --patch-mode flag. It's more settings and documentation, but at least it would only revert to this level of usability if someone missed it.

[core]
    pager = diff-so-fancy | less --stuff
[interactive]
    diffFilter = diff-so-fancy --patch-mode | less --stuff

[cosurgi]

Hi, I upgraded to git version 2.20.1 and now git add --all --patch produces:

fatal: mismatched output from interactive.diffFilter
hint: Your filter must maintain a one-to-one correspondence
hint: between its input and output lines.

but I still have my strange version of diff-so-fancy.

[HaleTom]

I think I worked it out:

@reegnz @wren adding echo works if only showing the diff for a single file.

If using git add --patch and multiple files are listed, then after displaying the first, I see:

fatal: mismatched output from interactive.diffFilter
hint: Your filter must maintain a one-to-one correspondence
hint: between its input and output lines.

By default, there are 5 lines before the text of a patch:

diff --git a/alia b/alia
index 5214cfe..c6d36bd 100755
--- a/alia
+++ b/alia
@@ -2,14 +2,28 @@

Adding echo before calling dsf will also produce a 5 line header:

────────────────────────────────────────────────────────────────────────────────
modified: alia
────────────────────────────────────────────────────────────────────────────────
@ alia:5 @

However, any subsequent files listed in the diff output will not have the echo, meaning that they will appear to only have a 4 line header.

Can't dsf have a -p/--patch option that ensures a 5 line header without a hack?

[paulirish]

An update here:

@peff put up a patch on the git mailing list (thank you, btw! so perfect). and it's now been merged into git's next branch.

commit: git/git@0114384

When it's available in git's master, folks can try things with brew install --HEAD git and configure the interactive.diffFilter.

If anyone wants to try it now, they can build git from the next branch.

[peff]

No, I meant the opposite. d-s-f should have a special mode that is perhaps less fancy, but retains the 1:1 correspondence in line meanings.

[SimenB]

It works with the caveats of off by one (which makes parts of diff not in the current chunk appears) and a Use of uninitialized value... error which I think is why it's not added yet.

That said, I use diffFilter myself, so maybe add it to the readme with those caveats included?

[olivierlefloch]

Would it be possible to pass options to diff-so-fancy to prevent breaking the 1-1 correspondence between input and output? @HaleTom 's solution to this (which uses diff-highlight instead of diff-so-fancy) seems to go in that direction, and works properly, as far as I can tell?

[hultberg]

We could open a feature request to git itself and see if we could get a config for this?

[scottchiefbaker]

@OJFord this is looking pretty good to me now actually. I'm going to make a formal pull request with what we have so far, and people can test it.

[HaleTom]

It's been 9 months since the last comment.

The best work-around I have is:

[interactive]
  diffFilter = "diff-highlight | less -FRX --tabs=4"

Can someone please explain what is blocking this getting resolved?

[OJFord]

It would be easy to change the diff-so-fancy header to add an empty line, so that the number of line is kept at four. Nobody seems to bother to work on this change.

This is a very long thread, I don't blame you for not bothering to read through it all. 😉

We tried exactly this in summer 2016; it's not so straightforward as 'add an empty line'. Feel free to replace diff-so-fancy in your git config with cat <(echo) <(diff-so-fancy) if you think that will be sufficient for your purposes; it probably won't be!

See from around here if you want to try the attempt(s) - but note that obviously nothing worked in all cases, I don't recall precisely the issues, but it would be merged and this issue closed if it had!

[wren]

I finally got a chance to dig into this, and here's what I turned up.

I think the difference in header size is actually a red herring. dsf consistently outputs a header that is n lines shorter than git (n=1 for line changes, and n=2 for mode changes and file deletions). Since this is consistent, padding the top of the diff with n blank lines should fix the problem. But it doesn't. Why?

So, I compared a bunch of diffs from git and dsf and found some inconsistencies with the way dsf treats lines at the end of the diff.

Here's the repro steps, if you want to follow along at home:

1. Save the following two diffs into their respective files yes-newline and no-newline:

diff --git a/yes-newline b/yes-newline
index 7528f0e..6be9db0 100644
--- a/yes-newline
+++ b/yes-newline
@@ -1,5 +1,5 @@
 line one
 line two
 line three
-line four
+line four with a newline
 

and

diff --git a/no-newline b/no-newline
index 7528f0e..5872fea 100644
--- a/no-newline
+++ b/no-newline
@@ -1,5 +1,4 @@
 line one
 line two
 line three
-line four
-
+line four without a newline

2. Compare yes-newline
   Note: We use grep -c ^ and not wc -l because not all lines have a newline at the end.

$ cat yes-newline | grep -c ^
11

$ cat yes-newline | diff-so-fancy | grep -c ^
10

As expected, the dsf output one line shorter.

3. Compare no-newline

$ cat no-newline | grep -c ^
11

$ cat no-newline | diff-so-fancy | grep -c ^
11

Surprise! The dsf output is the same length as the git output (even though the header is still one line shorter).

I believe this is why padding the header with blank lines sometimes gives us the wrong number of lines as output. Also, not padding the header makes these files work, but gives us off-by-one problems with the lines.

My hypothesis is that if the trailing whitespace is cleaned up, then padding the header with blank lines should consistently make dsf work in patch mode. So, I made a quick script to test this out (it just normalizes trailing newlines, then pads the header). I've been using it for a few days, and throwing every kind of diff I can think of at it, and it's been working pretty well so far.

Obviously, that script is a hack (it may or may not be dependent on my specific git settings, for all I know), but I think it's a proof of concept that this issue might be fixable on dsf's side. I haven't gotten a chance to dig into the dsf source, yet, but does this sound reasonable to y'all? Or am I way off-base?

[scottchiefbaker]

Yes this all seems feasible to me. I don't think there is a way to "detect" --patch mode so we'll have to add a flag to d-s-f that's invoked when required.

What you have laid out above seems both logical and feasible. I appreciate you taking point on this. I've available to collaborate on d-s-f internals if you have questions.

#goteam

[peff]

@paulirish You're welcome. I'll be interested to hear whether diff-so-fancy users run into problems with things like hunk-splitting. The colorization code in add--interactive relies on a one-to-one correspondence between the stock and fancy diff lines. diff-so-fancy doesn't quite do that, but I think the line counts add up, so it should work OK in practice. Caveat emptor. :)

[peff]

The problem is that git doesn't actually re-parse the filtered output (nor would you want it to, since it doesn't understand your fancy output). It just assumes that line 1 of the diff corresponds to line 1 of the filtered output, and so on.

The diff header has 4 lines (diff --git..., the index line, and the ---/+++ lines), but your fancy output only has 3 (the two border lines, and the filename). So everything it shows is off-by-one, and at the end perl complains that we look past the final element of the filtered array, as it's one line shorter than the diff.

I think you'll need to output an extra line somewhere in the fancy header to match git. Note that git's diffs can actually vary a little, too, depending on whether there is other data like renames, and possibly with mode changes, too. So you may want to actually count the number of lines to the first hunk, output your header, and then insert the appropriate amount of filler.

[scottchiefbaker]

@peff is there any reason in the --patch mode output would flip lines two and five of the git header for file delete? What I'm seeing after an hour and a half of debugging is that d-s-f is doing the right thing with the input, but --patch is flipping those two lines BEFORE it displays the output on the screen. This only appears to be happening for deleted file mode, not normal modifications.

I'm able to recreate this if you check out this branch: https://github.com/scottchiefbaker/diff-so-fancy/tree/debug_mode

1. Configure git to use this new d-s-f and patch mode thus:
   1. git config --global interactive.diffFilter 'diff-so-fancy --patch-mode | less --tabs=4 -RFX'
2. rm appveyor.yml
3. git add --patch

Note the output on the screen has the deleted file mode 100644 on the fifth line. Then check /tmp/diff_so_fancy_debug.txt which should be the raw output and note that deleted file mode 100644 is on the second line.

Somewhere between d-s-f printing the output and --patch displaying it those line are getting swapped.

@OJFord noted that adding an extra \n in the output "sort of" fixes this issue. This is because the \n moves the deleted file mode line down a line and --patch swaps two other lines instead.

In short I think git's implementation of --patch is doing screwy things with the output.

[phil-blain]

No, it does not work because the diff-so-fancy header is 3 lines (the error message is pretty clear):

$ git -c interactive.diffFilter=diff-so-fancy add -p
────────────────────────────────────────────────────────────────────────────────────────────────────
modified: .gdbinit
────────────────────────────────────────────────────────────────────────────────────────────────────
@ .gdbinit:9 @ set auto-load safe-path .
set history save on
set history remove-duplicates unlimited
# Save history in $HOME instead of $PWD
# set history filename ~/.gdb_history
set history filename ~/.gdb_history
(1/1) Stage this hunk [y,n,q,a,d,e,?]? n

fatal: mismatched output from interactive.diffFilter
hint: Your filter must maintain a one-to-one correspondence
hint: between its input and output lines.
$ 

whereas the interactive code (git [add | checkout | reset | restore | commit | stash] -p) expects the regular 4 lines (or more, in case of mode change/rename/copy) header :

$ git add -p
diff --git a/.gdbinit b/.gdbinit
index 66551cb..0112325 100644
--- a/.gdbinit
+++ b/.gdbinit
@@ -6,4 +6,4 @@ set auto-load safe-path .
 set history save on
 set history remove-duplicates unlimited
 # Save history in $HOME instead of $PWD
-# set history filename ~/.gdb_history
+set history filename ~/.gdb_history
(1/1) Stage this hunk [y,n,q,a,d,e,?]? n

diff --git a/.gitconfig b/.gitconfig
index 02d6f93..ce0e9af 100644
--- a/.gitconfig
+++ b/.gitconfig
@@ -1,6 +1,6 @@
# ...

It would be easy to change the diff-so-fancy header to add an empty line, so that the number of line is kept at four. Nobody seems to bother to work on this change.

EDIT at least that is my understanding.

[ItsNickBarry]

I don't know what it does, but this seems to work:

[core]
	pager = diff-so-fancy | less --tabs=4 -RFX
[interactive]
	diffFilter = (cat && diff-so-fancy)

Using echo breaks when a diff involves the end of a file, despite the fact that my editor adds a newline automatically.

[paulirish]

Someone else pinged me about this a month ago. I'm not sure.

Based on https://github.com/git/git/blob/master/git-add--interactive.perl#L747-L768 it doesn't look good.

But i'd love to be proven wrong. There's a lot of mention of color in here, so there may be SOME configurability, but not sure if we can intercept with a pager or something.

[paulirish]

There's some other people who were poking around this:

http://www.nickcoding.com/2014/10/30/making-git-prose-friendly/
http://git.661346.n2.nabble.com/Different-diff-strategies-in-add-interactive-td7588815.html

.. which my unveil some solutions.

[mrageh]

Thanks for the swift response @paulirish, would it be worth leaving this issue open or do you think we should close it?

[paulirish]

Leave it open. It still requires some research to see if it's possible or not.

[tonekk]

👍 was just going to add the same issue

[danielcompton]

git commit --verbose doesn't use it either.

[hultberg]

As @danielcompton pointed out, git commit -v does not use diff-so-fancy.

[paulirish]

The solution here appears to be to fork $brewdir/Cellar/git/2.7.0/libexec/git-core/git-add--interactive and make some edits.

I don't see a cleaner way to configure this.

[paulirish]

We could open a feature request to git itself and see if we could get a config for this?

Yup, we'll need to get this supported upstream in git core first.

---

@peff I think you're the right person to help us out. First of all, thank you. This project has been a big success mostly due to your work on diff-highlight.

You can read the above issue for context, but it appears we are lacking support in git-core for a key area. We'd like to propose, I think, that add--interactive can support a configurable pager. Mostly, we want a hook to run diff-highlight and our tool while add--interactive is printing hunks. Does this make sense to you, and does it sound possible?

[OJFord]

Please forgive me for asking the obvious - I'm sure others have a firmer grasp of it than I do - but perhaps it's best to get a note of it anyway.

Does a configurable pager make sense for add--interactive in any context other than diff-so-fancy or similar tool?

For example, most have pager set to less (plus some options) or maybe vim. How does this make sense for multiple fragments? I'm not sure it would be desirable to jump in and out. I think what's needed (that is, desired) is an option separate to PAGER, which does some formatting prior to the pager or interactive prompt and pipes on through.

Ultimately this would mean that whether interactive or not, diff-so-fancy could do it's thing before the pager or the interactive prompt, and would not require modifying the PAGER from default setup at all; perhaps this would be something like FORMATTER=diff-so-fancy. Or have I misunderstood?

[paulirish]

@OJFord Yeah you're right. It's not a pager, but just a hook to pipe the diff output somewhere. It does seem like the value is mostly for diff formatting tools. yup.

[peff]

@paulirish Yes, I think what you're asking for makes sense. I don't use diff-so-fancy, but I have been annoyed at not getting diff-highlight support via git add -p before. :)

The infrastructure is already there to have "diffs to apply" versus "diffs to show the user" (that's how we do the color stuff). It assumes there's a 1:1 correlation in lines between the two versions of the diff (which is certainly true for diff-highlight; I don't know enough about diff-so-fancy to say more there). Without that correlation, all bets are off (because git can do stuff like hunk-splitting, and needs to split the user-visible one, too).

I think you'd basically just need to pipe the colorized git-diff invocation through the custom program. It shouldn't be more than a few-line change (...he says without having looked closely yet).

[peff]

I just sent http://article.gmane.org/gmane.comp.version-control.git/287659 to the list. Not well tested, but it works for me. You can also fetch it from the jk/interactive-diff-filter branch of git://github.com/peff/git.git.

[megalithic]

@peff thanks for implementing this for interactive adds; from what you've seen in the code base, does it seem possible to relatively easily implement for interactive commits, e.g. --verbose?

thanks again!

[peff]

@megalithic I'm not quite sure what you mean. If you mean commit --interactive, that should work already with my patch (along with checkout -p, reset -p, etc, as they are all fed by the add--interactive script).

The commit --verbose code path is quite different, and is all done internally, with no separate processes. I suspect it would not be too hard to add an option to tell git to run some arbitrary process rather than its internal diff, when given --verbose. But then, I think you can pretty much already do that with a prepare-commit-msg hook (you can either add commented-out lines, or you can have a matching commit-msg hook to strip them out after the user has edited).

[paulirish]

@peff i built git from the next branch but so far I'm unable to get the patch working..

I'm doing ~/code/temp/git/git add -p and I have this in my ~/.gitconfig:

[interactive]
    diffFilter = "tail -n 3"

But I'm seeing full headers, lines of context for each prompt. Any suggestions?

[peff]

@paulirish I assume ~/code/temp/git/git is your build directory. Running it directly from there means that git will still find the older version of any external programs (like the git-add--interactive helper) in your $PATH.

Either install it (with make install PREFIX=/wherever...) and run it from the installed directory, or run it at ~/code/temp/git/bin-wrappers/git add -p, which will add the build directory to the front of your $PATH while git is running.

[paulirish]

run it at ~/code/temp/git/bin-wrappers/git add -p, which will add the build directory to the front of your $PATH while git is running.

ah thanks. That sounds like the ticket. will try.

[stevemao]

Is there a way to make an offset somehow? Or even fake it?
If not, we have to add empty lines to make it work (Paul might be sad about it).
Thanks @peff for the help!

[paulirish]

sounds like we can add empty lines at the end to make the line count match. seems fine!​

[stevemao]

at the end

You must mean after ------------------------------------- and before @....@ 😄

[megalithic]

@tomviner: is there a way to take this awesomeness and apply it to verbose commits (though, I imagine at that point, it'd need to be something your $EDITOR would handle (e.g., (neo)vim, atom, emacs, etc).

[toejough]

I found this tool because of those release notes, and hit this error the very first time I did git add -p after installing diff-so-fancy and updating my .gitconfig.

[scottchiefbaker]

To summarize what I see as the issue...

The original git header is four lines (the yellow lines) which we humanize and convert to three lines. Git patch isn't aware it's running inside of d-s-f and uses the original output for line numbers, which is off by one because there are now less lines.

A simple work around would be to add a blank line after the yellow header, and before the purple hunk line. This will get the line count the same before and after d-s-f parses it.

This would add whitespace to viewing of regular diffs, which is not my preference. The best case scenario would be for d-s-f to somehow know that we're in a --patch and add the extra header line only for those types of diffs. I'm not sure if this is possible or not though.

[scottchiefbaker]

@OJFord I like that solution actually... it keeps the two contexts very separate. I can code that up pretty easily, and then someone can test with --patch?

[tomviner]

@scottchiefbaker @OJFord +1 I was going to suggest a parameter too.

[OJFord]

Looks good! Happy to have a crack at some tests tomorrow.

[tomviner]

@scottchiefbaker this continues to fail for deleted files:

tom:~/dev/diff-so-fancy-mess-2$ git config interactive.diffFilter 
diff-so-fancy --patch-mode | less --tabs=4 -RFX
tom:~/dev/diff-so-fancy-mess-2$ rm appveyor.yml 
tom:~/dev/diff-so-fancy-mess-2$ git add --patch 
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

@ appveyor.yml:3 @
deleted: appveyor.yml
install:
  - git clone --depth 1 https://github.com/sstephenson/bats.git

build: false

before_test:
  - git submodule update --init

test_script:
  - C:\cygwin\bin\bash -lc "cd $APPVEYOR_BUILD_FOLDER; bats/libexec/bats test/*.bats"
Use of uninitialized value $_ in print at /usr/lib/git-core/git-add--interactive line 1364.
Stage deletion [y,n,q,a,d,/,?]? n

tom:~/dev/diff-so-fancy-mess-2$

Although to be fair, that error line (Use of uninitialized value...) appears twice without your patch.

People should also test git stash --patch, git reset --patch and git checkout --patch, they use interactive.diffFilter too.

[scottchiefbaker]

@tomviner are we not adding \n on file deletes? I thought we were, so I'm confused why this is causing an error. Can you post the full output using --patch-mode:

git diff --color | diff-so-fancy --patch-mode

I'm curious if we're missing a \n somewhere.

[tomviner]

The output does have the extra line:

But the git add -p output is messed up:

You can try to reproduce by deleting the appveyor.yml file in this repo, doing an interactive patch like git add -p.

And to confirm my settings:

$ git config interactive.diffFilter 
diff-so-fancy --patch-mode | less --tabs=4 -RFX

[OJFord]

It needs another newline at the end of the STDIN loop.

[peff]

@scottchiefbaker Right, this is the rearranging I mentioned in #172 (comment). For deletions (and mode changes, too), git will shuffle the line down to be its own "hunk" that you can say yes or no to. So effectively some parts of the header (before the first hunk), get shifted down.

And git rearranges the filtered output line-by-line as it does the original, without any regard to the content. So you need to have a one-to-one correspondence between the fancy line at a given position and the original (or at least for the ones that get rearranged; the other bits just have to take up the same number of lines).

[scottchiefbaker]

@peff Aha... some how I missed that before.

I don't think the rearrange functionality will be compatible with d-s-f. Specially we throw away the file deleted mode lines all together. Since git add --patch is using d-s-f and expecting it to mangle/rearrange the lines (that's the whole goal of d-s-f) would it be possible to modify git add --patch to NOT move the lines around on interactive adds that use a diff-filter?

We do our best to make sure that we match the same header line count to preserve --patch line mappings, but I don't see a way to work around line swapping the way it is. Especially since the line you're swapping d-s-f ends up discarding.

[peff]

@scottchiefbaker I don't think it would be trivial to change add --patch. It has to show something to the user when it asks about a deletion or mode change. I think your best bet is to have a special mode for diff-so-fancy that specifically keeps those lines at their original spots (or possibly prints some fancy version of them). I think you should be able to make a diff that doesn't look good when fully assembled, because git won't actually show the fully assembled diff to the user.

[scottchiefbaker]

@peff where are we at with:

best bet is to have a special mode for diff-so-fancy that specifically keeps those lines at their original spots (or possibly prints some fancy version of them).

I don't think d-s-f can address this issue with add --patch moving those lines around AFTER they're output from d-s-f.

[peff]

@scottchiefbaker I wasn't planning to work on this. I don't think it's going to be feasible for add --patch to avoid moving the lines around. If you want it to work, I think d-s-f needs to maintain a 1:1 mapping of original and fancy lines (which yes, constricts your fanciness).

[scottchiefbaker]

What I took away from your comment was that a special mode would have to made for git add --patch that does NOT move those lines around. Is that correct understanding? If so I guess we should probably file a formal bug/issue with the git team to request that.

[OJFord]

@samcv interactive.diffFilter has been discussed earlier in this thread (cf. #35 (comment)) - as @SimenB says this is what sparked the discussion and attempts to hack around lines being out of order/missing/duplicated.

[vanniktech]

#296 mentioned that there needs to be some work done upstream by Git. Is there an open issue or something which we can subscribe to? I'd really like to start using diff-so-fancy again when using interactive.

[cosurgi]

yes! it works!

[interactive]
	diffFilter = ~/bin/git-fancy-diff.perl

and then git add --all --patch is awesome!

[jasonmp85]

and then git add --all --patch is awesome!

Wait, I'm missing some context. What do I need to put in ~/bin for this to work?

[cosurgi]

Oh. Sorry, it's this file: https://github.com/so-fancy/diff-so-fancy/blob/v1.1.1/diff-so-fancy I've put it in my local ~/bin directory. An older version actually ;) I should upgrade sometime, I guess :)

[cosurgi]

If it doesn't work for you then maybe some other parts of my .gitconfig play a role. Maybe this one:

[core]
	pager = ~/bin/git-fancy-diff.perl | less --tabs=4 -RFX

I'm not sure.

[cosurgi]

Hmm.. when I look closer I see that my ~/bin/git-fancy-diff.perl is a little different than https://github.com/so-fancy/diff-so-fancy/blob/v1.1.1/diff-so-fancy I have no idea where did this change come from. It is a standalone file in my ~/bin without all the other extra files in https://github.com/so-fancy/diff-so-fancy

Here's my version (attachments work with .txt, so I added extra extension).
git-fancy-diff.perl.txt

btw, I use git version 2.11.0

[sublimino]

@cosurgi as noted in #35 (comment) older git clients work, but there are security vulnerabilities in old git versions.

The only difference in your files is a header that reads:

# This chunk of stuff was generated by App::FatPacker. To find the original 
# file's code, look for the end of this BEGIN block or the string 'FATPACK' 
...
unshift @INC, bless \%fatpacked, $class;
 } # END OF FATPACK CODE

Do you see the same results using that code with the latest version of git?

[alfredbez]

@wren Which git version do you use? I've tried your config, but it gives me the known error:

fatal: mismatched output from interactive.diffFilter
hint: Your filter must maintain a one-to-one correspondence
hint: between its input and output lines.

I use git version 2.17.1.

[wren]

$ git --version
git version 2.24.0
hub version 2.13.0

$ diff-so-fancy --version
Diff-so-fancy: https://github.com/so-fancy/diff-so-fancy
Version      : 1.2.6

[quintrino]

Hmm, @wren I also have the same versions (except for hub) and this works for me for the first chunk, but not for the later ones.

Do you have anything else in your config that would be interacting with this?

I am on the same versions and as above it works fine in one repository but errors in another.

[mvcds]

I am on the same versions and as above it works fine in one repository but errors in another.

Have you checked their .gitconfig to see if some different options could influence the behavior of the lib, @Katsika ? I'm not aware of anything like this, but it could trigger some discovery

@mvcds Can't see any obvious differences between repositories. Also, it does not error for entire repository but single file only (git config file). All other files in repository are OK. I just managed to break that file down to error causing chunk. It is

+
+ [filter "lfs"]
+   process = git-lfs filter-process
+   required = true
+   clean = git-lfs clean -- %f
+   smudge = git-lfs smudge -- %f

at the end of the file. If I remove that chunk completely dsf and add --patch works just fine.

[quintrino]

Hi @Katsika, with the line but single file only (git config file) do you mean you're committing the git config file into git and it's the filter chunk that's breaking it?
I'm not sure I understand.

Or is it some other random file?

@quintrino Yes, you read it as intended. Apologies for not being clear. I have dot files repository and dsf works for all changed files except for git configuration file and in particular for chunk I pasted in previous comment.

[boris-petrov]

I have the same experience as @Katsika - works fine for one repo, for another it doesn't. I tried making the echo command have a -n argument (that removes the new line at the end of the output of echo) which fixed the first chunk in the broken repo, but blows up on the second. Maybe someone has an idea why?

[OJFord]

@HaleTom We discussed it above (#35 (comment)) - there ended up being issues with header lines appearing in different orders iirc.

[HaleTom]

@OJFord I like your suggestion too, but I believe I've just demonstrated above that the issue is the number of lines, rather than the ordering.

[OJFord]

Ah yes, sorry I missed that. That error and hint was introduced as a result, as far as I recall. Whether a line is missing or out of order, the core issue is the same, d-s-f may modify only within lines.

I think this is possible, it's just messy, requires look-ahead up to the end of the header, and will be necessarily inconsistent with non-interactive d-s-f, and between different sorts of patch.

[vassudanagunta]

Can y'all confirm my explanation as to why git add --patch or any interactive git command can't use a pager?

It's the very same reason that getting git add --patch to use diff-so-fancy isn't so simple, and the reason there are the distinct core.pager and interactive.diffFilter settings discussed above... Am I right?

[peff]

@vassudanagunta Certainly you can't use a pager in the naive way of git add -p | less for the reasons you mentioned. But there's no reason Git couldn't show each hunk with a pager, and avoid reading input until the pager finishes. In fact, interactive.diffFilter runs an arbitrary command at just the right time already. The only reason that setting it to less doesn't work is that the output is piped back to git-add--interactive, and less is smart enough to pass data through when stdout isn't a tty. But git -c interactive.diffFilter='less >/dev/tty' add -p works reasonably well (at least in a simple test I just tried). It does strip the color out of what's ultimately shown at the prompt. You could use a script like this to make it a true pass-through filter:

#!/bin/sh
tmp=$(mktemp -t diff.XXXXXX) || exit 1
# you could even run it through diff-so-fancy here if you wanted!
cat >"$tmp"
less "$tmp" </dev/tty >/dev/tty 2>&1
cat "$tmp"
rm -f "$tmp"

But all of that is pretty orthogonal to diff-so-fancy. It can use interactive.diffFilter to run at the right moment. The tricky part is that the lines need to correspond between input and output so that git-add--interactive knows which of the diff lines to pass along to git apply.

[peff]

But git -c interactive.diffFilter='less >/dev/tty' add -p works reasonably well (at least in a simple test I just tried).

To be clear, this is a hack. I wouldn't be opposed to a patch to Git to show any given hunk in a pager (either automatically, or when the user says "show this hunk in a pager" at the interactive prompt), if somebody wants to work one up. But again, this is all orthogonal to diff-so-fancy's complications.

[HaleTom]

But git -c interactive.diffFilter='less >/dev/tty' add -p works reasonably well (at least in a simple test I just tried).

This has less page all the hunks in the file (in colour if using d-s-f also), then print out just the hunk in question without colour. A small improvement could be made with tee to repeat the colouring.

To be clear, this is a hack. I wouldn't be opposed to a patch to Git to show any given hunk in a pager (either automatically, or when the user says "show this hunk in a pager" at the interactive prompt), if somebody wants to work one up. But again, this is all orthogonal to diff-so-fancy's complications.

@peff I've looked into the perl code, and it seems that we could print into a pipe rather than simply to STDOUT. I'll work on a PR.

[peff]

This has less page all the hunks in the file

Doh, of course. I was testing only on a single-hunk example, so I didn't notice. A pager option would clearly need to be triggered per-hunk right before prompting.

I've looked into the perl code, and it seems that we could print into a pipe rather than simply to STDOUT.

Yeah, that would make sense.

[HaleTom]

@OJFord I don't follow your look-ahead logic. d-s-f's headers are short by one line (as demonstrated). If they were increased by one line in all instances, then the line count would match.

If there is only a single file (hence single header), then adding an echo can make git's line count equivalence check pass. But if there are multiple files, there's no easy way of adding a line to the 2nd through last headers.

Why doesn't d-s-f always produce a 5-line header (like diff)? Or if not always, when given something like a --diffFilter argument?

[pascalhpi]

This does no longer work. Right now it's

[interactive]
        diffFilter = delta --color-only

but you should probably just follow the instructions given at the repo (https://github.com/dandavison/delta), as they'll probably be up to date!

[bradly]

Sorry if this is obvious, but is git add -p supported? This issue is almost 5 years old with a ton of comments and discussions about various interactive git commands, so I'm unsure if simple add -p is supported at this time or if there is a successful work around. Thank you all for your work.

[OJFord]

Ah interesting, good find!

So it sounds like the leading (well, I suppose it's also trailing) single whitespace on the final line is getting trimmed or ignored; one way or another that line is being treated as blank, whereas it's actually a semantic part of the diff.

I believe there's a git config option for marking unchanged lines (similarly to +/- for changed) - so that would be a good way to confirm.

(Edit: I seem to be wrong that that's an option, but could do something like s/^ /=/g to test, or that may even be useful internally to avoid trimming it (if that's what's happening), before changing back at the end for presentation.)

cc @scottchiefbaker

[wren]

That's what I thought, too, at first. But the dsf output is actually longer than it's supposed to be in these cases, so I don't think any unneeded trimming is happening. Maybe there's somehow an extra line being inserted somewhere?

[OJFord]

the dsf output is actually longer than it's supposed to be in these cases

I don't follow?

From your post above:

$ cat yes-newline | grep -c ^
11

$ cat yes-newline | diff-so-fancy | grep -c ^
10

In fact, it's even easier to reproduce:

$echo -n ' ' | diff-so-fancy | wc
      0       0       0
$echo -n '+' | diff-so-fancy | wc
      0       1       1

The space on the final line is as semantic as as the + or - on the lines preceding it, but it's getting trimmed. It's OK on lines that have content after them, just not when it's the final line.

[wren]

@OJFord I'm sorry, I don't think I explained very well in my original post.

I meant the following example as dsf doing things right. And it is what happens most of the time (which is why patch mode works on most diffs):

$ cat yes-newline | grep -c ^
11

# good dsf behavior
$ cat yes-newline | diff-so-fancy | grep -c ^
10

And I meant the following example to point out the behavior I believe is faulty:

$ cat no-newline | grep -c ^
11

# bad dsf behavior
$ cat no-newline | diff-so-fancy | grep -c ^
11

Since dsf creates a header that is one line shorter than the git header, the dsf output should reliably be one line shorter (two lines for mode changes and file deletions, but let's stick with the current examples). If the dsf output is a predictable length, then the fix for this issue (fix patch mode) gets much simpler (just pad the extra line or two in the header). I believe that this extra line that is sometimes output is responsible for the various problems that currently make patch mode unstable. I'll do a line-by-line breakdown of the above diffs in a subsequent comment, to show the extra line.

For the echo examples above, though, I'm don't think those are representative examples for this issue. Diff-so-fancy has no output at all if the input is entirely whitespace (no matter how many lines), like so:

$ echo -n '   \n \n\n   \n ' | wc
      4       0      12

$ echo -n '   \n \n\n   \n ' | diff-so-fancy | wc
      0       0       0

But if you have any non-whitespace characters, then the rest of the string is detected just fine. You can see in the following example that the last line isn't being trimmed, even though it's just a space with no newline:

$ echo -n 'hello  \n \n\n   \n ' | wc
      4       1      16

$ echo -n 'hello  \n \n\n   \n ' | diff-so-fancy | wc
      4       1      16

$ echo -n 'hello  \n \n\n   \n ' | grep -c ^
5

So, I really don't think it's a trimming issue, and is actually an extra line issue. Does that make sense?

[scottchiefbaker]

As the main developer I have to admit this hasn't been a huge priority for me because I don't user patch mode very often. Obviously I'm in the minority here as this bug has a lot of traffic on it. I'm glad people stepped up and took the time to figure out the problem, and a possible solution.

I will 100% be willing to merge a fix on this once we have a solid handle on what/how. What you have above looks pretty decent to me. Assuming the test suite passes then I'm probably fine with it.

With this fix in place, I think it would be easy to add a patch mode...

Can you elaborate on this? Would we have a specific flag that is invoked for git add -p aliases? How would that be implemented?

[wren]

Can you elaborate on this?

Yes, of course.

I was thinking of patch mode as a flag (unless git add --patch is auto-detectable somehow?) to pass to tell diff-so-fancy to do additional formatting to the regular output so that git add --patch will work. This additional formatting consists entirely of 1 to 2-ish blank lines in the header. The rest of the output should be identical to current dsf output, and should be unaffected by a patch mode. If you don't want to introduce a new flag or mode, these blank lines could become part of the regular diff output. But I do think the current output looks nicer without them, so I'm suggesting a separate mode.

Also, the rest of the thread already covers this, but to reiterate: the need for the blank lines is due to a git requirement that any output it gets back from a pager should match its original output line-by-line. Padding the header with blank lines is fudging this requirement a little bit, but the rest of the dsf output already meets the requirement, so I think this should be okay (the worst I've observed is that the spacing might look slightly awkward due to extra blank lines, but that seems a small price to get the added readability from dsf).

So, this patch mode (whether it's activated by flag or automatically) would look at the type of diff (file deleted, mode changed, or everything else) and determine how many blank lines to prepend to the header. Conveniently, I already needed to do these comparisons while troubleshooting this issue (I believe the required changes are 2 lines for file deleted and mode changed, and 1 line for everything else, but this needs some tests added, of course), so most of that work is already done.

For implementation, I can't get very detailed since I haven't thoroughly looked through the dsf code, yet. At a glance, though, it looks like there are already sections for the different types of diffs in the do_dsf_stuff subroutine, so maybe something in each of those to keep track of how many lines to add to the header? If there's a better place for this, please let me know, otherwise I can get into more details after digging in a bit more.

A potential problem I foresee is with testing. Bats notoriously has trouble with blank lines (bats-core/bats-core#224, sstephenson/bats#93), so writing tests for this might get a bit awkward. I think the new functionality could be split up in such a way that it still gets some decent coverage, though.

Another potential problem is related to file deletions. Git weirdly reorders some of the output after dsf returns. From my testing, it looks like it always moves the 2nd line and inserts it back as the 5th line. In the context of a new patch mode, the 2nd line is always blank, so git just ends up moving a blank line down a bit which doesn't affect the readability at all. The potential problem is that I'm not sure if git ever moves other lines to other places. It doesn't seem like it from my testing, but this would require some further research to be sure. Realistically, though, this would only ever affect lines from the header (as mentioned above), so it's probably pretty low impact either way.

[wren]

@scottchiefbaker Hi! Is that approach acceptable to you?

I might have some time this weekend to start on a PR if we've nailed down the what and how to do.