`gix clean -xde` deletes whole repo if `.gitignore` lists `*` or `/` about gitoxide

[Byron]

I'll think about how to improve that wording and open a pull request to do so.

Thank you 🙏

But git does not itself treat / as a full path the same way. For example, in the above experiments, listing / did not cause git to regard .gitignore as ignored--when making the / test repository, I didn't have to have !.gitignore, or any other entry appended after /, to successfully stage .gitignore by running git add .. Repeating them now confirms this, and also files with names like a are not ignored by git due to / in .gitignore. For git, I am not sure / as a full entry in .gitignore has any effect at all.

Thanks for researching this. I will add this to the baseline tests (comparing Git results with gitoxide) so that it's clear these match. Once again, I think this case is currently missing.

That makes sense, because if I ignore foo and unignore bar then foo/bar is still ignored due to being in the foo directory.

I mean, this is how gix clean does it - it only looks one dir-entry at a time and asks for whether it's igonred or precious. And I don't know what would happen if /foo\n!bar is used in .gitignore and someone tries to add foo/bar. Probably foo/bar would then be allowed, at least it should be as per the rules. In case you are interested, this should be verifiable with gix exclude query.

I would not consider this to disprove that, but I have found that when the current repository is a submodule with a .git file and its own .gitignore lists .git, deleting the .git file in a submodule does not require -r.

Thank you, that sounds wrong, too, and I will take a look.

[EliahKagan]

I'll think about how to improve that wording and open a pull request to do so.

Thank you 🙏

No problem! I'm holding off for a bit in doing so, since #1464 might affect what ought to be said.

[EliahKagan]

I've edited the description to note that the code I quoted is not necessarily where a change has to be made, since Ignored taking precedence over Pruned does make sense outside of specific cases.

With us being in different time zones, it would have been useful had I contributed some part of the tests or fix between the initial report and now. Unfortunately I was quite distracted and, while I was able to make a bit of progress on other gitoxide-related things, I didn't really come up with anything for this.

[Byron]

Thanks a lot for the incredibly detailed analysis of the problem.

I also feel that ideally, a solution would make such kind of mistakes impossible, but I am also not sure this can happen. More will be known once I dive into the code.

In addition to the above, the effect of -x and -d on actual nested repositories, especially if they are to continue to delete them under some conditions even in the absence of -r, should be documented explicitly, including in the output of gix help clean. But that could be done separately from the fix for this bug.

I'd hope you will be able to submit a PR with the documentation improvements you would like to see - I cannot imagine anyone better suited to make them.

Besides that, I hope that the default messaging around the removal of ignored directories works:

❯ cargo run --bin gix -- -r /Users/byron/dev/github.com/nabijaczleweli/cargo-update clean -xd
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.23s
     Running `target/debug/gix -r /Users/byron/dev/github.com/nabijaczleweli/cargo-update clean -xd`
WOULD remove / (🗑️)

WARNING: would remove repositories hidden inside ignored directories - use --skip-hidden-repositories to skip

Even though the example is taken from the buggy version that is unaware this is the root, it does warn that ignored repositories would be removed. Overall, I like the way it guides the user towards adding flags to change the set of would-be-cleaned files.

Regarding /, I think I should add tests around that to see what happens - it's nothing I really considered which makes it likely to exhibit surprising behaviour. In practice as demonstrated here, it seems to be another way of matching all with * though.

Regarding the !<file> not having any effect, I think the real issue here is that it doesn't care about these at all even though it would probably identify them as tracked. The directory walk probably disables listing these, so it won't see them at all. This leaves the issue at incorrectly classifying the root-directory as something that can be deleted. Additional tests around ignoring .git should probably also be added for good measure.

The key seems to be that the top-level directory is taken to match the entry *, causing its contents all to be deleted even if some of them match ! exclusions.

Agreed.

In addition, combined with the above results, this raises the question of whether gix clean -xde without -r is actually intended to delete any actually nested repositories:

The intended behaviour is that it allows itself to remove ignored git repositories which are contained in ignored directories. That's easy to do as it may not even look deeper unless the --skip-hidden-repositories option is provided.

The root cause is probably that it's possible to declare the top-level directory as ignored, which allows contained repositories to be removed as well.

❯ cargo run --bin gix -- -r /Users/byron/dev/github.com/nabijaczleweli/cargo-update clean -xd --skip-hidden-repositories all -r
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.65s
     Running `target/debug/gix -r /Users/byron/dev/github.com/nabijaczleweli/cargo-update clean -xd --skip-hidden-repositories all -r`
WOULD remove repository .git/ (🗑️)

This invocation shows the behaviour quite nicely, as it now found the 'hidden' repository itself, and flagged it for deletion as requested. Of course, that shouldn't be possible here.
It's notable that this also won't be possible for submodules as these should be identified as 'tracked', but the root of the repository is never tracked.

So I'd think this is a very 'local' bug as it really can only apply to the root of the repository, or so I'd think.
Of course, I encourage you to prove that wrong.

[EliahKagan]

In addition to the above, the effect of -x and -d on actual nested repositories, especially if they are to continue to delete them under some conditions even in the absence of -r, should be documented explicitly, including in the output of gix help clean. But that could be done separately from the fix for this bug.

I'd hope you will be able to submit a PR with the documentation improvements you would like to see - I cannot imagine anyone better suited to make them.

Besides that, I hope that the default messaging around the removal of ignored directories works:

❯ cargo run --bin gix -- -r /Users/byron/dev/github.com/nabijaczleweli/cargo-update clean -xd
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.23s
     Running `target/debug/gix -r /Users/byron/dev/github.com/nabijaczleweli/cargo-update clean -xd`
WOULD remove / (🗑️)

WARNING: would remove repositories hidden inside ignored directories - use --skip-hidden-repositories to skip

Even though the example is taken from the buggy version that is unaware this is the root, it does warn that ignored repositories would be removed. Overall, I like the way it guides the user towards adding flags to change the set of would-be-cleaned files.

Yes. Assuming it is intended that -r not always be required to allow nested repositories to be removed--which I think is confirmed by your subsequent description of its interaction with --skip-hidden-repositories--that output is good. But then the way -r/--repositories is documented, at least in the output of gix help clean (i.e. the documentation here), is misleading:

  -r, --repositories
          Remove nested repositories

This makes it seem not only that -r causes nested repositories to be removed, which is true, but that, without -r, nested repositories are not removed.

I'll think about how to improve that wording and open a pull request to do so.

Regarding /, I think I should add tests around that to see what happens - it's nothing I really considered which makes it likely to exhibit surprising behaviour. In practice as demonstrated here, it seems to be another way of matching all with * though.

As far as how gitoxide currently treats / as a full path, I think so.

But git does not itself treat / as a full path the same way. For example, in the above experiments, listing / did not cause git to regard .gitignore as ignored--when making the / test repository, I didn't have to have !.gitignore, or any other entry appended after /, to successfully stage .gitignore by running git add .. Repeating them now confirms this, and also files with names like a are not ignored by git due to / in .gitignore. For git, I am not sure / as a full entry in .gitignore has any effect at all.

Regarding the !<file> not having any effect, I think the real issue here is that it doesn't care about these at all even though it would probably identify them as tracked. The directory walk probably disables listing these, so it won't see them at all. This leaves the issue at incorrectly classifying the root-directory as something that can be deleted. Additional tests around ignoring .git should probably also be added for good measure.

That makes sense, because if I ignore foo and unignore bar then foo/bar is still ignored due to being in the foo directory.

So I'd think this is a very 'local' bug as it really can only apply to the root of the repository, or so I'd think. Of course, I encourage you to prove that wrong.

I would not consider this to disprove that, but I have found that when the current repository is a submodule with a .git file and its own .gitignore lists .git, deleting the .git file in a submodule does not require -r.

That pertains to "Case 5" in the Expected behavior section of the issue description. I had not tried it at that time. I have tried it out now, by doing a recursive clone of has-submodule and experimenting with .gitignore.

It's notable that this also won't be possible for submodules as these should be identified as 'tracked'

Yes, I do think case 6 is already not a problem, and the experiments I've just done seem to confirm this.

[EliahKagan]

To clarify the part about the .git file, the repository is the submodule, and I mean that deleting the repository's own .git file does not require -r. (I was not trying to describe a situation with nested submodules.)

This is likely what you took me to mean, but I just realized that was extremely ambiguous, so I figured I'd clarify just in case.