avianflu / ncp Goto Github PK

Hi, I tried to copy all json files in a folder without success.

copyJson = (source, dest, callback) ->
  jsonFilter = (name) ->
    return name[-5..] is '.json'

  options = {filter: jsonFilter}
  ncp source, dest, options, (error)->
    if error
      console.log "Error while copying *.json from #{source} to #{dest}"
    callback()

options.filter - a RegExp instance, against which each file name is tested to determine whether to copy it or not, or a function taking single parameter: copied file name, returning true or false, determining whether to copy file or not.

function startCopy(source) {
    started++;
    if (filter) {
      if (filter instanceof RegExp) {
        if (!filter.test(source)) {
          return cb(true);
        }
      }
      else if (typeof filter === 'function') {
        if (!filter(source)) {
          return cb(true);
        }
      }
    }
    return getStats(source);
  }

As the source folder name doesn't fit the *.json mask, content is not explored. Did I misunderstand the purpose of the filter, or should only filenames be checked, and not folder names ?

Regards

when a file is modified, copy it automatically?

The version of node that landed on Friday is now broken due to the same version check referenced in #79

I think it is worth escalating now as ncp is broken on stable node.js

/cc @AvianFlu

Is this project still maintained?

The pull requests fixing ncp on io.js and Node v0.12 have been opened a month ago and still not merged: #78, #80.

Should we all switch to cpr or something else instead?

Hi,

I just created a node-webkit app on osx (10.9.4) and added ncp 0.6.0.

I'm only calling this code form node-webkit via a button click:

    var fromPath = "/Volumes/6148b320-ed75-4d39-ba7e-d55fe7f5eac0/TestApp.app";
    var toPath = "/var/folders/fp/rt5y2x2x7qvghvsrf03jr6jh0000gn/T/x1.TestApp.app";

    console.log("ncp - pre", fromPath, toPath);
    ncp(fromPath,toPath, function(err){
      console.log("copied files", err);
    });

The app goes immediately to 100% cpu usage and never finishes.
the console output "copied files" is NOT printed out.

When I switch to version 0.5.1 the coping works fine.

Any ideas on what the issue here may be?

Cheers,
Gerwin

First, thanks for the great tool.

I'm using ncp to copy all contents of one folder to another, and have created a gulp task to do so. I would like to include this gulp task in the middle of a sequence of gulp tasks (using runSequence), but considering it is asynchronous, the next gulp task in the sequence begins before ncp is complete, causing problems. An option to make the call synchronous would be great.

It would be nice to be able to pass a symlink as a source and to have it copy the contents of the linked directory.

(The following was discovered during concurrency testing using loader.io.)

Example:

ncp(path, newpath, function(error) {
  if(error) {
    ...
  } else {
    foo(newpath,...);
  }
});

If the above code is run under high concurrency, where multiple calls to ncp are executing in parallel, a given callback function may be invoked multiple times. For example, concurrent calls to:

Copy A -> X
Copy A -> Y
Copy A -> Z

results in three calls to the callback that invokes foo(Z).

This would allow for highly customizable file tree copying.

While copying a folder I noticed the callback never gets called. I'm not getting errors, but I'd like to do work in that callback once ncp is done.

The copy does happen. Just no calling of the cb.

I have version 2.0.0, and node version v0.10.37

Let's say I want to copy the following files: a.txt, b.txt, c.txt
If the target destination has a directory named b.txt ncp will find that as an error even if clobber is set as true.

The other way around is even weirder.
Eg. I want to copy the following folders: a, b, c
If the the target destination has a file named "b" it will silently skip the folder called "b" and finishes the job without overwriting the target or throwing an error.

I am using bunyan in a project I am "browserifying". It all works well, but things really collapse when I use more traditional browsers... E.g. IE9. Any objections to changing const to var? The const modern variable is never really exposed outside the library so risks are basically limited to your code.

This would really help! I will submit a PR if you would like me to.

Test case; previous version:

$ mkdir test-0.6.0
$ cd test-0.6.0/
$ npm install [email protected]
node_modules/.bin/ncp node_modules node_modules.backup
$ echo $?
0
$ ls
node_modules  node_modules.backup

Test case; 1.0.0:

$ mkdir test-1.0.0
$ cd test-1.0.0
$ npm install [email protected]
$ node_modules/.bin/ncp node_modules node_modules.backup
An error has occurred.
Error: can't copy `/aux0/dp/test-1.0.0/node_modules.backup` into `/aux0/dp/test-1.0.0/node_modules`  itself
    at Function.ncp (/aux0/dp/test-1.0.0/node_modules/ncp/lib/ncp.js:34:23)
    at Object.<anonymous> (/aux0/dp/test-1.0.0/node_modules/ncp/bin/ncp:33:5)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:902:3
$ echo $?
1

This is, for us, very severe.

The permissions are not currently respected.

cp -r stops users from doing this. ncp currently continues to recurse, resulting eventually in ENAMETOOLONG after many levels of directory nesting. While an obscure situation, this should be fixed.

Looks like callback fired before all files are copied

console.log(fs.readdirSync(argv["from"]));

ncp(argv["from"], argv["to"], function(err) {
    if(err) { throw err; }

    console.log(fs.readdirSync(argv["to"]));
});

here is output (nw.js v0.12.0)

p.s. recursive copy from fs-extra work like a charm

I think your filter script might be slightly wrong. Tested using the following REGEXP:

/package/i

Against the following /package.json. It still copies it over. After looking in to the NCP script I believe it should be the following, which works:

if (filter.test(source)) {

from line 38 of ncp.js

Unless I'm completely missing the way the filter is supposed to work?

It would be nice to have an option in to only copy files if their modification time is newer than the one in the destination folder.

As of 0.2.4 files are moved instead of copied.

This only occurs when using ncp programmatically, the cli tool seems unaffected.

To reproduce:

ncp = require('ncp').ncp;
ncp('./a', './b', {}, console.log);
ncp('./a', './c', {}, console.log); // will throw Error: ENOENT, no such file or directory

Using node v0.6.7 on OS X 10.7.3

I poked around the code and noticed that it doesn't indeed attempt to preserve file permissions, but I feel that I must be doing something wrong if that's case.

Here's my test program with ncp 0.5.0 and v0.10.25:

var fs = require('fs');
var path = require('path');
var ncp = require('ncp');
var assert = require('assert');
var userid = require('userid'); //npm install --save userid
var rimraf = require('rimraf'); //npm install --save rimraf

var DIR = '/tmp/ncptest';

if (fs.existsSync(DIR))
  rimraf.sync(DIR);

fs.mkdirSync(DIR);


//http://man7.org/linux/man-pages/man2/stat.2.html
var S_IFREG = 0100000 //regular file
var S_IFDIR = 0040000 //directory

var permDir = path.join(DIR, 'perms');
fs.mkdirSync(permDir);

var srcDir = path.join(permDir, 'src');
fs.mkdirSync(srcDir);

var f1 = path.join(srcDir, 'f1.txt');
fs.writeFileSync(f1, '');
fs.chmodSync(f1, 0666);
fs.chownSync(f1, process.getuid(), userid.gid('wheel'));
var f1stats = fs.lstatSync(f1);
assert.equal (f1stats.mode - S_IFREG, 0666);

var d1 = path.join(srcDir, 'somedir');
fs.mkdirSync(d1);
fs.chmodSync(d1, 0777);
fs.chownSync(d1, process.getuid(), userid.gid('staff'));
var d1stats = fs.lstatSync(d1);
assert.equal (d1stats.mode - S_IFDIR, 0777);

var f2 = path.join(d1, 'f2.bin');
fs.writeFileSync(f2, '');
fs.chmodSync(f2, 0777);
fs.chownSync(f2, process.getuid(), userid.gid('staff'));
var f2stats = fs.lstatSync(f2);
assert.equal (f2stats.mode - S_IFREG, 0777);

var d2 = path.join(srcDir, 'crazydir');
fs.mkdirSync(d2);
fs.chmodSync(d2, 0444);
fs.chownSync(d2, process.getuid(), userid.gid('wheel'));
var d2stats = fs.lstatSync(d2);
assert.equal (d2stats.mode - S_IFDIR, 0444);

var destDir = path.join(permDir, 'dest');
ncp(srcDir, destDir, function(err) {
  assert (!err)

  var newf1stats = fs.lstatSync(path.join(permDir, 'dest/f1.txt'));
  var newd1stats = fs.lstatSync(path.join(permDir, 'dest/somedir'));
  var newf2stats = fs.lstatSync(path.join(permDir, 'dest/somedir/f2.bin'));
  var newd2stats = fs.lstatSync(path.join(permDir, 'dest/crazydir'));

  assert.equal (newf1stats.mode, f1stats.mode);
  assert.equal (newd1stats.mode, d1stats.mode);
  assert.equal (newf2stats.mode, f2stats.mode);
  assert.equal (newd2stats.mode, d2stats.mode);
})

Any ideas?

Right now files can get copied into dirs that don't exist (yet), which can lead to FS errors. It would be a good idea to use fs-extra instead of fs, which comes with ensureDir to solve that problem

io.js (https://iojs.org/) has process.version 'v1.0.3'

When it tries to use ncp, ncp runs this check:

const modern = /^v0\.1\d\.\d+/.test(process.version);

https://github.com/AvianFlu/ncp/blob/master/lib/ncp.js#L4

sadly, this identifies io.js as a non-modern version. It picks the wrong 'defer' method, and goes into an infinite loop.

I've just spent several hours debugging an issue which just provided an error message of undefined. I've ultimately tracked it down to an error originating from ncp calling the callback with an err that is an array. Unfortunately this error is then propagated to far away, to code which expects errors to be Error objects.

May I suggest that a better way to handle this would be an Error object with an array of paths in it?

Hello,

I see the callback gets called after

readStream.once('end', cb);

which potentially leads to the situation when callback gets called before all writes are complete. Shouldn't this:

writeStream.once('finish', cb);

be used instead?

The potential problem may never occur with some sane file-system and stream implementation but it still comes as a bit dangerous to me.

I have seen the first construction (wrong in my opinion) used in many projects, so if I am wrong go on and educate me please.

I recently ran into an issue on a127 where the filter callback used to filter out paths not to copy was not matching. When I did a little debugging, I noticed that ncp was passing in a file name with both \ and / in it. Basically, the / started at the copy root path. So if I started copying from C:\Users\Jeremy Whitlock\Projects\a127, all path segments after C:\Users\Jeremy Whitlock\Projects\a127 used / for path separators. For example, when ncp called the filter for the C:\Users\Jeremy Whitlock\Projects\a127\package.json file, the file name passed to the filter is C:\Users\Jeremy Whitlock\Projects\a127/package.json.

Normal people when they think of issuing a command: "copy folder X into folder Y" expect that the result they get is a folder under Y called X with all the contents of folder X in it.

NCP breaks this in two ways:

1. You end up with folder Y with contents of folder X in it
2. If you have more than Z items in the folder it just stops copying, there seems to be no way to disable this limit.

Now, try this out in a shell:

mkdir foo
mkdir bar
touch foo/file
cp -r foo bar

What do you end up with?

Yes, you will have a bar/foo/file. This is the expected behavior from anything that copies things recursively.

If you execute instead:

node
require("ncp").ncp("foo", "bar")

You end up with bar/file, you might expect this result with "foo/*" as the first argument, or something similar, but this is just bad.

Sup dawg:

josh@onix:~/dev/project$ npm install ncp
npm http GET https://registry.npmjs.org/ncp
npm http 200 https://registry.npmjs.org/ncp
npm http GET https://registry.npmjs.org/ncp/-/ncp-0.2.6.tgz
npm http 200 https://registry.npmjs.org/ncp/-/ncp-0.2.6.tgz
npm ERR! Error: EACCES, symlink '../ncp/bin/ncp'
npm ERR!  { [Error: EACCES, symlink '../ncp/bin/ncp'] errno: 3, code: 'EACCES', path: '../ncp/bin/ncp' }
npm ERR! 
npm ERR! Please try running this command again as root/Administrator.

npm ERR! System Linux 3.2.0-34-generic
npm ERR! command "/usr/local/bin/node" "/usr/local/bin/npm" "install" "ncp"
npm ERR! cwd /home/josh/dev/project
npm ERR! node -v v0.8.8
npm ERR! npm -v 1.1.59
npm ERR! path ../ncp/bin/ncp
npm ERR! code EACCES
npm ERR! errno 3
npm ERR! stack Error: EACCES, symlink '../ncp/bin/ncp'
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /home/josh/dev/project/npm-debug.log
npm ERR! not ok code 0

looks like the bin install is bugging out?

bad AvianFlu!

I have a very involved buildscript, with nested buildsteps and callbacks several layers deep. I'd like to rewrite that as a chain of .then().

Are there any plans to support a Promise API with ncp?

In 20dc003, line 22 was terminated with a semicolon, making the subsequent variables global and therefore shared between concurrent calls to ncp. The result is missed callbacks when the check on line 222 incorrectly returns false.

Starting with grunt-docco, we noticed a directory copy sometimes failed (DavidSouther/grunt-docco#40). We tracked that down to node-fs-extra, where @malinges opened jprichardson/node-fs-extra#98; I duplicated that error using ncp directly and am opening this here.

% git clone https://gist.github.com/DavidSouther/77db8d2ce83c98521082
% cd 77db8d2ce83c98521082
% npm install
% node copytest1.js
done: { '0': null }

% node copytest1.js
done: { '0': null }

% node copytest1.js
done: { '0': null }

% node copytest1.js
done: { '0': null }

% node copytest1.js
done: { '0': null }
done: { '0': null }

% node copytest1.js
done: { '0': null }
done: { '0': null }

% node copytest1.js
done: { '0': null }

% node copytest1.js
done: { '0': null }

% node copytest1.js
done: { '0': null }

% node copytest1.js
done: { '0': null }
done: { '0': null }

% node copytest2.js
done: { '0': null }

% node copytest2.js

% node copytest2.js

% node copytest2.js

% node copytest2.js

%

We expect either of these files to always output exactly one done event with no arguments.

Possibly related to #51, #66?

ncp 'a', 'b', (error) => console.log 'Error is', error
# => Error is []

a doesn't exist in this case.

I noticed that NCP is not passing back an error into the callback, but instead is passing an array of errors. Example:

ncp('/somefolder', "/home/vagrant/nonexistantDirectory", function(e) {
    console.log(e)
})

Please fix this so it actually returns an error object, not an array.

Hi.

I encountered the following problem.
If you copy a directory containing a number of other directories ncp stops responding.
This bug works on ubuntu and does not work on os x.

I think this is due to the fact that adding a directory in turn "running" variable increases, but the function of CB may not be started, because "function cb(skipped)" running only with files.

On my computer at work, I rewrote this section so:
before

function getStats(source) {
    var stat = dereference ? fs.stat : fs.lstat;
    if (running >= limit) {
      return defer(function () {
        getStats(source);
      });
    }
    running++;
    stat(source, function (err, stats) {
      var item = {};
      if (err) {
        return onError(err);
      }

after

function getStats(source) {
    var stat = dereference ? fs.stat : fs.lstat;

    stat(source, function (err, stats) {
      var item = {};
      if (err) {
        return onError(err);
      }
      if (stats.isFile() && running >= limit) {
         return defer(function () {
           getStats(source);
         });
      }
      running++;

Your thoughts on this subject?

Like the --parent option in cp

There seems to be an issue copying a directory over another directory using clobber mode. I am finding that intermittently the completion callback is never triggered.

I did a little digging and saw that the "running" count becomes negative in my case.

I think the issue is in the onFile function here:

      if(clobber) {
        rmFile(target, function () {
          copyFile(file, target);
        });
      }
      if (modified) {
        var stat = dereference ? fs.stat : fs.lstat;
        stat(target, function(err, stats) {
            //if souce modified time greater to target modified time copy file
            if (file.mtime.getTime()>stats.mtime.getTime())
                copyFile(file, target);
            else return cb();
        });
      }

If clobber is true, there can be two calls to copyFile, eventually causing two calls to cb, resulting in the started/finished/running counts getting out of sync. I think this is related to this change:
85cd50a

Thanks

I have a situation where I need to copy readonly files into another folder where the copy needs to be read-write. I don't see a way to do this currently. Maybe I'm missing something or doing something wrong?

The command line cp utility can copy files to a directory, but ncp cannot. Example:
cp path/to/file.txt path/to/dir

Replacing onFile with the following code will resolve the issue. I'm just too lazy to fork the project, add test cases, yadda yadda yadda:

function onFile(file) {
var target = file.replace(currentPath, targetPath);
path.exists(target, function (exists) {
if (exists) {
if (fs.statSync(target).isDirectory()) {
target = path.join(target, path.basename(file));
} else {
return rmFile(target, function () {
copyFile(file, target);
});
}
}
copyFile(file, target);
});
}

I'm getting the error "UNKNOWN, unlink" after switching to piping files to copy them. It looks like ncp might vomit if the file is currently already open.

Is there a reason you changed the major version? Are there breaking changes? It sure seems that way. My previously working code (with version 0.6.0) now never get its callback called (for error or otherwise) in version 1.0.0. It seems like something major must be broken in the new version

Looking through the code I noticed that fs.utimesSync(path, atime, mtime) is being used to update timestamps instead of fs.utimes(path, atime, mtime, callback). Is there any particular reason?

I see that right below the usage of the synchronous method the callback is called, so I presume it could just be updated to execute the callback through the async utimes method instead.

This command ncp test/source test/source_final fails with the error:

Error: can't copy `test\source_final` into `test\source` itself

Not sure if this is something I'm doing wrong, but the callback is only called for me when the target directory doesn't exist or is empty. I'd expect it to be called regardless of this.

Duplicate of issue #71 and others. Existing fixes in PRs #77, #88, and others.

A new fix and test are proposed in PR #94.

Duplicate copies: when clobbering an existing file, onFile does both a remove-and-copy (line 94) and possibly copies the file again depending on modified (line 102).

Early callback: both the call to copyFile on line 95 and one of the later calls to copyFile or cb will increment the number of finished callbacks, with the result that the callback passed to ncp is called too early, before the copy is necessarily completed.

I can appreciate the existing maintainers are likely busy, so in the interim I would like to recommend users the similar node_module CPR.

Milage may vary, but for my use-cases it was a drop-in replacement with the added benefit of nearly eliminating all my existing intermittent CI failures.

The maintainers appear active and responsive.

If i myself had more bandwidth I would offer my time to help maintain this project, unfortunately my queue is full and I hope to save others time and energy by making this recommendation.

I hope this doesn't cause any anger or hurt feelings. It absolutely is not my intent.

some relevant issues:

• #71
• #95

Currently ncp does not copy hidden files. I would like the option to copy hidden files.

if i try to copy from folder myfolder to $$$temp$$$\myfolder, it will fail,
because string.replace function interprets second parameter and replaces $$ to $ (for example).
it is easy to fix - just instead of
var target = link.replace(currentPath, targetPath);
call
var target = link.replace(currentPath, function () { return targetPath; });

thank you

I'm unsure as to why ncp is not throwing this error, but node is supposed to throw when writing an recursive function that uses process.nexTick.

(node) warning: Recursive process.nextTick detected. This will break in the next version of node. Please use setImmediate for recursive deferral.

It looks like the current implementation found at --> https://github.com/AvianFlu/ncp/blob/master/lib/ncp.js#L55-L61 is blocking the thread... makes me sad.

I'm looking at ways to fix this, but commenting out the three lines that call to defer does stop ncp form breaking on iojs and node 0.12

/cc @AvianFlu

• Node 5.10.1
• Windows 7

Add some console on that:

  function copyFile(file, target) {
    var readStream = fs.createReadStream(file.name),
        writeStream = fs.createWriteStream(target, { mode: file.mode });

    readStream.on('error', function () {
        console.log("Read Err:", arguments);
    });
    writeStream.on('error', function () {
        console.log("Write Err:", arguments);
    });
    writeStream.on('close', function () {
        console.log("Write Close:", arguments);
    });

    if(transform) {
      transform(readStream, writeStream, file);
    } else {
      writeStream.on('open', function() {
          console.log("Open:", file.mode, file.name, target);
        readStream.pipe(writeStream);
      });
    }
    writeStream.once('finish', function() {
        console.log("done", target);
        if (modified) {
            //target file modified date sync.
            fs.utimesSync(target, file.atime, file.mtime);
            cb();
        }
        else cb();
    });
  }

console.log('open') works and other console not trigger. Create 0 size file. I'm not sure whether is a nodeJS bug?