gliviu / dir-compare Goto Github PK

Consider the following structure

The library never compare index.t and index.cc. The thing is that I want to ignore the extensions only for pair of extensions. In this example, I want the library to treat index.t and index.cc as the same file by the following code:

if (
  // options.ignoreExtension is ["t", "cc"]
  options.ignoreExtension.some((exts: [string, string]) => {
    return exts.includes(extnameOnly(name1)) && exts.includes(extnameOnly(name2));
  })
) {
  name1 = path.basename(name1, path.extname(name1));
  name2 = path.basename(name2, path.extname(name2));
}

https://github.com/moshfeu/vscode-compare-folders/pull/58/files#diff-b3ec7b5233f0b1a6113fd044710718c983a78f50e5960ddb0c0507fc9738e0dcR15

Test repo: https://github.com/moshfeu/dir-compare-test

buffer-equal uses node 0.11+ buffer.equals when available, buffertools require compilation (problematic on windows)

Hello

I have created some TypeScript type definitions so that I can use this library in a project. Would you be interested in a pull request for them?

Thanks for the great library 👍

If a symlink points to a file that doesn't exist, it should not break the diff.

It looks like symlinks are not handled well in general.

offending line: fileDescriptorQueue:43

fs.close(fd, callback); // <--------- callback is undefined here

Doesn't seem to affect functionality. I only noticed due to node v7's warning

(node:48849) DeprecationWarning: Calling an asynchronous function without callback is deprecated.

To reproduce

git clone https://github.com/olsonpm/dir-compare-deprecation.git
cd dir-compare-deprecation
npm i

# must have node 7 installed
nvm use node

node index

# outputs
# done
# (node:48849) DeprecationWarning: Calling an asynchronous function without callback is deprecated.
# (node:48849) DeprecationWarning: Calling an asynchronous function without callback is deprecated.

Since it doesn't affect functionality, I wasn't sure what the correct fix would be. If you want, I can look into a fix as well.

Thanks for the module - it helped me out a lot :)

I want to know which file is different

#16 inspired me to make a test of the following case:

$ tree -C /tmp/22 /tmp/11
/tmp/22
├── 1
├── 2
└── 3 -> 2
/tmp/11
├── 1
├── 2
└── 3 -> 1

6 directories, 0 files

dir-compare reported that everything is fine:

$ dircompare.js -c -lrd /tmp/11/ /tmp/22
Entries are identical
total: 0, equal: 0, distinct: 0, only left: 0, only right: 0

It's usually not a big deal, however, on the way of a potential implementation of #16 it might be worth to add a check if a synlink points to the same place (both relative and absolute). I don't know how it is implemented in the current version, but I think that dir-compare shouldn't follow the link - just check if a destination is the same.

git diff is able to detect when a file has been renamed, based on a "similarity index":

diff --git a/a/a.txt b/b/b.txt
similarity index 85%
rename from a/a.txt
rename to b/b.txt
index b8ee8a7..8054674 100644
--- a/a/a.txt
+++ b/b/b.txt
@@ -1,3 +1,3 @@
 asdfdsfasdf
 gsdfgsdfgdsfgsdfg
-asdf
\ No newline at end of file
+asdf1
\ No newline at end of file

currently dir-compare treats all renames as a file "missing" on each side

compareSync:119 and compareAsync:147

I'm having a problem with the compareDate option on Windows.
If I copy a file and preserve the timestamp by copying the modification date with utimesSync, it only copies the seconds, not the milliseconds.
The compareDate functionality is using getTime() which returns milliseconds.

E.g. I have two files with two timestamps 1480843824000 and 1480843824030 in which I copied the mtime of the second to the first one. The seconds are the same, but the milliseconds are not, as they are not copied. So compareDate will always return false.

Maybe you could add an option to compare seconds and use getTime() / 1000 to compare the date.

Originally asked in moshfeu/vscode-compare-folders#65

I'm in desperate need of the ability to do some diffs on some project directories that should be very similar but have a lot of whitespace differences scattered around, making it impossible to easily figure out what's meaningfully different between them. Any chance you might be able to add something equivalent to running diff -b or diff -w?

For the record, I have the ignoreWhiteSpaces option set which does ignore some whitespace differences like those at the start or end of a line, but that's it. It still shows differences in the number of newlines or whitespace differences in the middle of a line, like between a bracket and a comment.

@gliviu is it a reasonable feature?
Thanks

Just an idea, not a necessity:

An option could be added to compare the checksums of the two directories, for example using the dirsum package.

If the arguments to compareSync are two files, then the result of comparison is a little odd: it seems they are treated as if they were each contained in a directory, so the files are only considered the same if their names are the same.

But compareSync applied to two directories compares their contents; it should do the same for files.

I came across this issue because I have some code that normally compares directories, but in some cases it may only compare single files, and I had to special case this and not use dir-compare for the single files.

Thanks for dir-compare! A very handy piece of code.

mkdir -p tmp/test1.good
mkdir -p tmp/test2.good
mkdir -p tmp/test3.bad
touch tmp/test1.good/test.txt
touch tmp/test2.good/test.txt
touch tmp/test3.bad/test.txt
sudo chown root:root tmp/test3.bad/ -R
sudo chmod og= tmp/test3.bad/
cd tmp/
npm install dir-compare-cli
./node_modules/.bin/dircompare test1.good test2.good

Entries are identical
total: 1, equal: 1, distinct: 0, only left: 0, only right: 0

./node_modules/.bin/dircompare test1.good test3.bad

Error occurred: Error: EACCES: permission denied, scandir '/home/joseph/tmp/test3.bad'
at Object.readdirSync (fs.js:981:3)
at getEntries (/home/joseph/tmp/node_modules/dir-compare-cli/node_modules/dir-compare/src/compareSync.js:18:26)
at compare (/home/joseph/tmp/node_modules/dir-compare-cli/node_modules/dir-compare/src/compareSync.js:33:20)
at Object.compareSync (/home/joseph/tmp/node_modules/dir-compare-cli/node_modules/dir-compare/src/index.js:27:5)
at /home/joseph/tmp/node_modules/dir-compare-cli/src/cli/dircompare.js:113:44
at new Promise ()
at run (/home/joseph/tmp/node_modules/dir-compare-cli/src/cli/dircompare.js:112:38)
at Object. (/home/joseph/tmp/node_modules/dir-compare-cli/src/cli/dircompare.js:147:1)
at Module._compile (internal/modules/cjs/loader.js:1075:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1096:10)

Copied from moshfeu/vscode-compare-folders#71 (comment)

I want to compare my mounted backup with the real directory to verify it's data integrity (that the backed up data are the same as the local). I started with the classic diff, but it doesn't support ** in the exclusion patterns. dir-compare on the other hand uses minimatch which claims to do that. Unfortunately I wasn't able to make it works.

Simple case. I excluded the internal backup subdirectory (made by the problem itself) from external backup.

$ dircompare.js -c /home/foo/somepath/ /tmp/mounted-backup/
Entries are different
total: 326, equal: 26, distinct: 0, only left: 300, only right: 0

With -a I see that:

[/users/xxx/backup/20191019_151830] tasks.xml->missing

I can add -x xxx and it works, but that directory name could be reused in some other content, so I want to use globstar patterns which are also used by my backup tools (Borg backup). And this doesn't work:

$ dircompare.js -c -x "**/users/xxx/backup/**" /home/foo/somepath/ /tmp/mounted-backup/
Entries are different
total: 326, equal: 26, distinct: 0, only left: 300, only right: 0

How can I configure dircompare.js with **?

Line based comparator ignoreLineEnding doesn't work correctly, when the files to be compared are bigger, than the BUF_SIZE (100000). In that case two files, which are only differs in line endings, such as one has CRLF and the other has LF, are deemed to be different by the algorithm.

I'm basically trying to make a directory compare tool that list all the files that are different between two directories. I can share my code if it helps.

Error: UNKNOWN: unknown error, lstat 'M:\User\xx\Node Projects\next-learn-demo\3-create-dynamic-pages\node_modules\core-js\fn\object\define.js'
    at Object.lstatSync (fs.js:943:3)
    at getStatIgnoreBrokenLink (C:\Node\fileDiff\node_modules\dir-compare\src\entry\entryBuilder.js:50:17)
    at Object.buildEntry (C:\Node\fileDiff\node_modules\dir-compare\src\entry\entryBuilder.js:32:15)
    at Object.buildDirEntries (C:\Node\fileDiff\node_modules\dir-compare\src\entry\entryBuilder.js:19:21)
    at C:\Node\fileDiff\node_modules\dir-compare\src\compareAsync.js:21:37
    at async Promise.all (index 1)
    at async Promise.all (index 9)
    at async Promise.all (index 6)
    at async Promise.all (index 92)
    at async Promise.all (index 3) {
  errno: -2146893818,
  syscall: 'lstat',
  code: 'UNKNOWN',
  path: 'M:\\User\\xx\\Node Projects\\next-learn-demo\\3-create-dynamic-pages\\node_modules\\core-js\\fn\\object\\define.js'
}

We want an option ignoreContentCase.
When we develop the vba extensions, vba kindly does case conversions automatically.
So, we compare files, dir-compare detects the difference.

We think adding the code below is ok, but we can not test.

at function compareLine in compareLines.ts

if (options.ignoreContentCase) {
    line1 = Line1.toLocaleLowerCase()
    line2 = line2.toLocaleLowerCase()
}

Or, someone tell us, how to extend dir-compare to ignore Content Case.
thank you.

i see you are capturing the date in the diff, why don't you have an option to compare those?

Hi,
thanks for the work on the library 🙇

I'm integrating it in a personal app using typescript and noticed that that the Difference interface doesn't currently cover the actual result of the difference.

For example:

  {
    "path2": "/<base-path>/LUTs/ImpulZ Ultimate/ImpulZ Ultimate for Vision Color/29 ImpulZ -  Sony FS700 Cine 4",
    "relativePath": "/LUTs/ImpulZ Ultimate/ImpulZ Ultimate for Vision Color/29 ImpulZ -  Sony FS700 Cine 4",
    "name2": "Kodak Vision3 250D 5207 (NEG)_VS.cube",
    "state": "right",
    "type1": "missing",
    "type2": "file",
    "level": 4,
    "size2": 137798,
    "date2": "2014-05-30T16:54:26.000Z"
  }

Doesn't contain name1, size1, date1 , equally for "state": "right", there isn't a properties called name2, size2, date2.

I propose changing the typescript interface of all possible solutions to a union of a specific type of response based on state.

interface Difference {..} to type Difference = DifferenceEqual | DifferenceLeft | DifferenceRight | DifferenceDistinct

As a user of the library, I had a problem to understand that type of result combinations would I get depending on the state.

I still haven't caught a case when would I get a "state": "distinct" as left & right cover the "distinct" cases.

If using the only state property is a valid way of splitting up the differences, I'm ready to help with a PR.

Update

Going through the library code understood that state:

• left - Entry is only present on the left side
• right - Entry is only present on the right side
• equal - Entry exists on both sides and depending on given options entries are considered "equal"
• distinct - Entry exists on both sides and depending on given option entries aren't considered "equal"

I suggest to Travis CI to automatically run your tests on OS X and Linux and AppVeyor to run your tests x86 and x64 Windows machines automatically.

I'm using this great library in my vscode extension.
There is a feature request to ignore files extension so the it will compare only by files name.
What do you think?

moshfeu/vscode-compare-folders#57

In my test code I am just testing if the non-whitespace is equal. I don't care if line endings are different.
Can we get an option to specify that line endings are ignored in the buffer comparison?

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• Update dependency tar-fs to v3
• Update dependency tsc-watch to v6
• Update dependency typescript to v5
• Update typescript-eslint monorepo to v6 (major) (@typescript-eslint/eslint-plugin, @typescript-eslint/parser)
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency minimatch to v3.0.5 [SECURITY]
• Update dependency @types/node to v16.18.41
• Update dependency semver to v7.5.4
• Update dependency ts-node to v10.9.1
• Update dependency tsc-watch to v4.6.2
• Update dependency typedoc to v0.24.8
• Update dependency typescript to v4.9.5
• Update typescript-eslint monorepo to v5.62.0 (@typescript-eslint/eslint-plugin, @typescript-eslint/parser)
• Update dependency @types/node to v18
• Update dependency eslint to v8
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• Update dependency memory-streams to v0.1.3
• Update dependency p-limit to v4

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

When using the "includeFilter" option, it's impossible to filter folders.

Checking the code (

Could it be possible to replace the line
if ((entry.stat.isFile() && options.includeFilter) && (!match(path, options.includeFilter))) {
by
if (options.includeFilter && !match(path, options.includeFilter)) {

It then will match what is done for the excludeFilter.

What do you think about it ?
Thanks in advance.

Here are my test results:

> node tests/runTests.js

test1: !!!!FAILED!!!!
test2: !!!!FAILED!!!!
test3: !!!!FAILED!!!!
test4: !!!!FAILED!!!!
test5: !!!!FAILED!!!!
test6: !!!!FAILED!!!!
test7: !!!!FAILED!!!!
test8: !!!!FAILED!!!!
test100: !!!!FAILED!!!!
test101: !!!!FAILED!!!!
test102: !!!!FAILED!!!!
test103: !!!!FAILED!!!!
test104: !!!!FAILED!!!!
test105: !!!!FAILED!!!!
test106: !!!!FAILED!!!!
test200: !!!!FAILED!!!!
test201: !!!!FAILED!!!!
test300: Passed
test301: Passed
test302: Passed
test303: Passed
test304: Passed
test400: !!!!FAILED!!!!
test500: !!!!FAILED!!!!
test501: !!!!FAILED!!!!
test600: !!!!FAILED!!!!
test601: !!!!FAILED!!!!

Tests: 27, failed: 22, succeeded: 5

I implemented the gitignoreFilter as specified here https://github.com/gliviu/dir-compare?tab=readme-ov-file#implement-gitignore-filter.

However, I started getting errors like:

Error: Path /private/var/folder/a.txt is not in cwd /var/folder

(I shortened the paths for the example)

I discovered that /var/folder is actually a symlink to /private/var/folder. dir-compare turns the symlink into an absolute path, but the GlobbyFilterFunction in gitignoreFilter.ts uses the original symlinked path, not the absolute path, hence the mismatch.

I fixed this for myself by modifying line 26 of gitignoreFilter.ts to use entry.path instead of entry.absolutePath.
https://github.com/gliviu/dir-compare/blob/master/test/extended/gitignoreSupport/gitignoreFilter.ts#L26

You could probably also resolve the absolute paths before calling isGitIgnoredSync.

You say the actual version is v2.1.0 but the real version in package json is 2.0.1

My use case is looking at whether we need to do git operations by comparing a pair of directories as current and to-be states. Recently ran into a problem where someone had created an empty directory in the to-be version, which caused dir-compare to (correctly) return that the directories were not the same, but then no git operations were needed (because git will ignore empty directories). Bottom line:
is there an option I'm missing that would allow me to tell dir-compare to ignore empty directories, and if not, is this something that would be worth me doing a PR for (vs what I'm going to do right now which is check the result sets after the fact)

Hi, I notice there are some breaking behavior changes when upgrading from 1.4 to 1.5:

• the diff paths are changed from relative to absolute path
• the compare() function will throw instead of returning a rejected promise when the input path does not exist.

Are these intentional?
Especially the second one, my guess is that it is a regression.

Thanks for this very useful library thou. 🌷

Great library. Any chance to get version that supports node 18?

I am getting an error while installing:

npm install dir-compare           
npm ERR! code EBADENGINE
npm ERR! engine Unsupported engine
npm ERR! notsup Required: {"npm":">=8.0.0 <9.0.0","node":">=16.0.0 <17.0.0"}
npm ERR! notsup Actual:   {"npm":"8.19.2","node":"v18.12.1"}