browserify / detective

Find all calls to require() no matter how deeply nested using a proper walk of the AST

detective

find all calls to require() by walking the AST

example

strings

strings_src.js:

var a = require('a');
var b = require('b');
var c = require('c');

strings.js:

var detective = require('detective');
var fs = require('fs');

var src = fs.readFileSync(__dirname + '/strings_src.js');
var requires = detective(src);
console.dir(requires);

output:

$ node examples/strings.js
[ 'a', 'b', 'c' ]

methods

var detective = require('detective');

detective(src, opts)

Give some source body src, return an array of all the require() calls with string arguments.

The options parameter opts is passed along to detective.find().

var found = detective.find(src, opts)

Give some source body src, return found with:

• found.strings - an array of each string found in a require()
• found.expressions - an array of each stringified expression found in a require() call
• found.nodes (when opts.nodes === true) - an array of AST nodes for each argument found in a require() call

Optionally:

• opts.word - specify a different function name instead of "require"
• opts.nodes - when true, populate found.nodes
• opts.isRequire(node) - a function returning whether an AST CallExpression node is a require call
• opts.parse - supply options directly to acorn with some support for esprima-style options range and loc
• opts.ecmaVersion - default: 9

install

With npm do:

npm install detective

license

MIT

detective's Issues

why not use the Regular Expression to find all params in require function??

why not use the Regular Expression to find all params in require function??

Revert 'require' word optimization

@zertosh What do you think about reverting the regex optimization so that syntax-error can be removed upstream?

Giant libraries with no require calls should be noParse'd anyway, and the performance difference, while significant, is only really marginal. Whereas the performance difference gleaned by removing syntax-error upstream is asymptotic and enormous.

IMO the bundle time of incremental re-builds is more important than one off full rebuilds, since incrementals are executed constantly (and also could be cached by browserify-incremental if you wanted).

I'm requesting this per your comment in this thread:

browserify/browserify#1208 (comment)

If you did this and merged in your sm-fast branch, we could have really awesome rebuild times I think.

pin esprima version

Would be nice for library users if the packages you depend on were also pinned. Not everyone always follows semver and even a ~0.9.9 could be dangerous as there might be some unforeseen side effect in a new point release :/

JS_Prase_Error when a require is inside a function

The following seems to make detective choke with a JS_Parse_Error. It is the only content in my file.

function() {
  require('./foo');
}

detective: 0.1.0

Support ES6 imports

Tried this:

import detective from 'detective';
import fs from 'fs';

let src = fs.readFileSync(__filename);
let requires = detective(src, { parse: { ecmaVersion: 6, sourceType: 'module' } });

console.log(requires);

Prints out [] instead of ['detective', 'fs']. Could we make detective support ES6 imports?

Indication of error for 'require(someStringVariable)'

It would be nice to know that there was a require that could not be parsed rather than it being ignored.

var reqStr = './some-thing'
  ;

require(reqStr);

Upgrade acorn

Will fix browserify/browserify#1146

if require is aliased, detective incorrectly returns uses of it as dependencies

Eg:

var require = function() {
  console.log('test')   
}   
require("this isn't a dependency")

It should be pretty simple to detect scopes in which require is redefined and exclude requires there.

Please include a license

Hi there. It would be awesome from a distribution packaging point of view if you could include the full text of the MIT license in your software, usually in a file called LICENSE.

Thanks!

require('something')()

this dose not work:

require('something')();

binary statement SyntaxError

Burrito -- or possibly detective, I'm not sure which to assign the issue to -- barfs on the following syntax:

++x >= y && okay(null, y)

I'll look into adding a failing unit test so there's more to go on.

Add a note about requires with more than string literals

Is it worth adding a note like browserify/browserify-handbook#21 here?

Or is return an array of all the require() calls with string arguments. enough? (From the readme)

TypeError: Cannot set property 'parent' of null

If you try to walk the source of wysihtml5, detective will throw TypeError: Cannot set property 'parent' of null -- /node-detective/index.js:7:22

Here is a simple example that demonstrates the problem:

var detective = require('detective')
  , request = require('request')

request('https://raw.github.com/xing/wysihtml5/master/dist/wysihtml5-0.4.0pre.js', function (err, response, src) {
  console.dir(detective(src))
})

Here is the node it's looping over. I can't figure out what is causing the node to be null, but it looks like that comes from esprima or escodegen

[ null,
  { type: 'Literal', value: 0 },
  parent: { type: 'ArrayExpression',
    elements: [Circular],
    parent: 
     { type: 'LogicalExpression',
       operator: '||',
       left: [Object],
       right: [Circular],
       parent: [Object] } } ]

I found this issue when upgrading to browserify v2.

Skip commented-out requires

Currently detective parses these as valid requires even though they are not.

require('b')
// require('c')
/*
  require('d')
*/

maybe with https://github.com/jonschlinkert/strip-comments

Upgrade to use Acorn v2

Hey,

I'm getting some parse errors when running dependency-check, which uses detective, on my files. Before digging further, I noticed Acorn is now at v2, but Detective still on v1. Perhaps upgrading to v2 would solve things without debugging. :-)

Cheers

Newer keywords like async/await is not supported

Would be great if you upgraded the acorn package to a newer version that understands async/await so that one can bundle such code in browserify.
I would like to skip transpiling such code to get better debugger experience.

Error behaviour difference between esprima.parse() and detective.find()

While tracking down an inconsistency from upgrading to browserify v2, I noticed a discrepancy between what is thrown by esprima.parse() and detective.find():

> esprima.parse('return a')
Error: Line 1: Illegal return statement
    at throwError (/Users/bengourley/node_modules/module-deps/node_modules/detective/node_modules/esprima/esprima.js:1161:21)
    at throwErrorTolerant (/Users/bengourley/node_modules/module-deps/node_modules/detective/node_modules/esprima/esprima.js:1172:24)
    at parseReturnStatement (/Users/bengourley/node_modules/module-deps/node_modules/detective/node_modules/esprima/esprima.js:2476:13)
    at parseStatement (/Users/bengourley/node_modules/module-deps/node_modules/detective/node_modules/esprima/esprima.js:2752:24)
    at parseSourceElement (/Users/bengourley/node_modules/module-deps/node_modules/detective/node_modules/esprima/esprima.js:3044:24)
    at parseSourceElements (/Users/bengourley/node_modules/module-deps/node_modules/detective/node_modules/esprima/esprima.js:3082:29)
    at parseProgram (/Users/bengourley/node_modules/module-deps/node_modules/detective/node_modules/esprima/esprima.js:3096:19)
    at Object.parse (/Users/bengourley/node_modules/module-deps/node_modules/detective/node_modules/esprima/esprima.js:3843:23)
    at repl:1:10
    at REPLServer.self.eval (repl.js:110:21)

> detective.find('return a')
{ strings: [], expressions: [] } 

I can't see the error being caught anywhere in detective/index.js even though esprima.parse(str) is called directly here.

---

Now to the thing I was looking in to that brought me to this module. I hope this is the correct module on which to discuss the issue.

In browserify v1 (and also in node) it is ok to return from the main module body, eg:

if (condition) {
  // Don't do what you are about to do
  return
}

// Code here

browserify v2 says that this is an illegal return statement, which is presumably in accordance with ES semantics (in that you can't return if you're not in a function), but this introduces a behavioural difference between the browserify environment and the node environment. What do you think?

Cheers

Support array calls

Hello,
interesting project.

It would be nice if the module could detect arrays

require(["a", "b"], function () {});

detective("code above");   //   []

Useful information can still be obtained from

detective.find("code above");
{
   expressions: [  ' [ 'a', 'b' ] '  ]
}

However the expression has to be parsed again to extract the array values.

Insecure dependancy "minimist": "^1.1.1"

Minimist <=1.2.5 is vulnerable to Prototype Pollution via file index.js, function setKey() (lines 69-95).

https://security.snyk.io/vuln/SNYK-JS-MINIMIST-559764

Document how `detective` can be extended

After encountering dependency-check-team/dependency-check#112 I tried looking into how that project uses detective and the various "extensions", such as detective-typescript that enabled foreign syntax. It was not immediately obvious how this worked, seeing no explicit interfaces where detective was wrapped by or passed some "extension"

Could this be documented somehow in the README?

extraneous findit in lib for 0.0.3 release

# mkdir tmp
# cd tmp
# npm install detective
[email protected] ./node_modules/detective
└── [email protected]
# npm ls
npm WARN seq >=0.1.7 Unmet dependency in /Users/naitik/tmp/node_modules/detective/node_modules/findit
/Users/naitik/tmp
└─┬ [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ └── [email protected]
  └─┬ [email protected]  extraneous
    └── UNMET DEPENDENCY seq >=0.1.7

Update to [email protected]

In order to support Logical nullish assignment (??=) we need to upgrade acorn-node to v. 2.0.1

Related Issue: #browserify/acorn-node#17

Related PR: ##85

syntax error is returned when object spread is used

when node-detective is passed the content of this file,
https://github.com/facebook/react-vr/blob/master/Libraries/Pano/Pano.js#L41

the following error message returns here

{ SyntaxError: Unexpected token (41:4)
    at Parser.pp$4.raise (/pathtomyproject/node_modules/depgraph/node_modules/acorn/dist/acorn.js:2488:13)
    at Parser.pp.unexpected (/pathtomyproject/node_modules/depgraph/node_modules/acorn/dist/acorn.js:623:8)
    at Parser.pp$3.parseIdent (/pathtomyproject/node_modules/depgraph/node_modules/acorn/dist/acorn.js:2445:10)
    at Parser.pp$3.parsePropertyName (/pathtomyproject/node_modules/depgraph/node_modules/acorn/dist/acorn.js:2265:99)
    at Parser.pp$3.parseObj (/pathtomyproject/node_modules/depgraph/node_modules/acorn/dist/acorn.js:2188:12)
    at Parser.pp$3.parseExprAtom (/pathtomyproject/node_modules/depgraph/node_modules/acorn/dist/acorn.js:1994:17)
    at Parser.pp$3.parseExprSubscripts (/pathtomyproject/node_modules/depgraph/node_modules/acorn/dist/acorn.js:1872:19)
    at Parser.pp$3.parseMaybeUnary (/pathtomyproject/node_modules/depgraph/node_modules/acorn/dist/acorn.js:1849:17)
    at Parser.pp$3.parseExprOps (/pathtomyproject/node_modules/depgraph/node_modules/acorn/dist/acorn.js:1791:19)
    at Parser.pp$3.parseMaybeConditional (/pathtomyproject/node_modules/depgraph/node_modules/acorn/dist/acorn.js:1774:19)
    at Parser.pp$3.parseMaybeAssign (/pathtomyproject/node_modules/depgraph/node_modules/acorn/dist/acorn.js:1750:19)
  pos: 1722,
  loc: Position { line: 41, column: 4 },
  raisedAt: 1725 }

I updated acorn locally to [email protected] and see the same error

related:

• https://github.com/UXtemple/acorn-object-spread

breaks on deep chaining.

this breaks on cases like:

require('crypto').createHash('sha1').update('abc').digest('hex')

this expression produces an AST that looks like this:

     [ { name: 'call',
          start: 
           { type: 'name',
             value: 'require' },
          end: 
           { type: 'punc',
             value: ')' } },
        [ 'dot',
          [ 'call',
            [ 'dot',
              [ 'call',
                [ 'name', 'require' ],
                [ [ { name: 'string',
                      start: 
                       { type: 'string',
                         value: 'c' },
                      end: 
                       { type: 'string',
                         value: 'c' } },
                    'c' ] ] ],
              'hello' ],
            [] ],

(I have removed irrelevant json)

currently, detective calls the traverse function on each array that starts with an object,
and then checks a few static cases.

really since the actual value passed to require may be chained arbitarily,
it needs to traverse the tree looking for the pattern

['call',
    ['name', 'require'],
    [ [ {name: 'string', ...}, STRING ] ]
]

I have an idea for a simpler way to do this, pull request on it's way.

'Missing yield in generator' should not be an error

Generators without a yield is valid JS; the older version of esprima-fb incorrectly flags this as an error. Upgrading to newer esprima-fb solves the issue.

Similar issue from facebook/react facebook/react#1475

Pass `sourceType` option through to acorn

Acorn throws an error when parsing a source that uses ES6 module syntax unless it has the sourceType: 'module' option (in addition to ecmaVersion: 6).

hash bang choke

It appears that uglify chokes on hash bangs?

SyntaxError: Unexpected character '#'
at line 0:0 in expression:

#!/usr/bin/env node
at /home/wojtek/Project/Node/INS/node_modules/detective/node_modules/burrito/index.js:16:17
at /home/wojtek/Project/Node/INS/node_modules/detective/node_modules/burrito/index.js:41:11
at Function.find (/home/wojtek/Project/Node/INS/node_modules/detective/index.js:12:5)
at /home/wojtek/Project/Node/INS/node_modules/detective/index.js:4:20
at /home/wojtek/Project/Node/INS/index.js:42:12
at /home/wojtek/Project/Node/INS/index.js:52:17
at Array.forEach (native)
at Object.start (/home/wojtek/Project/Node/INS/index.js:50:9)
at Object. (/home/wojtek/Project/Node/INS/bin/ins.js:23:26)
at Module._compile (module.js:402:26)

I'm getting a list of .js files and passing each one through detective. my /bin/file.js causes the error here as it begins with #!/usr/bin/env node

Replace `acorn` parser for `babylon`?

Hi,

I came across a problem when node-detective cannot operate on ES2015+ source code.

This is not a problem solely related to this package. The same problem applies to node-syntax-error package which is used in browserify pipeline.

I can see a clear problem in the fact, that some packages used by browserify internally depend on acorn parser.

This parser dependency is limiting e.g. for using browserify in conjuction with babelify in such situations when you don't want to transpile all the source code down to ES5.

For instance, let's say you don't want to transpile those syntax features, which are already natively supported by browsers, e.g. object spread operator in current Chrome. And now comes the problem with browserify syntax check and module dependecies parsing via module-deps -> node-detective dependency, all caused by code containing syntax features not parseable using acorn parser. It this case with ... operator.

Don't you think it would make sense to replace the parser and be more aligned with babel ecosystem?

Cheers,

Ondrej

PS: The same issue is being solved in these issues in Webpack project:

webpack/webpack#4308
webpack/webpack#2872

Support AMD syntax for finding dependencies?

Is there any interest in allowing detective to find the dependencies within the variations of AMD module syntax? Or is that outside of the scope of this module?

I'd whip up a PR if you're interested.

i compressed this project into 10 lines :

var regex = {
    all: /require\s*\('\s*([^'])*'\s*\)|require\s*\("\s*([^"])*"\s*\)/g,
    start: /require\s*\(\s*["']/,
    end: /["']\s*\)$/,
    comments: /\/\*.+?\*\/|\/\/.*(?=[\n\r])/g   // or /(\/\*([\s\S]*?)\*\/)|(\/\/(.*)$)/gm
  }

  function extract_dependencies(text) {
    text = text.replace(regex.comments, '')
    var requires = text.match(regex.all) || []
    for(var i=0; i< requires.length; i++) {
      requires[i] = requires[i].replace(regex.start, "").replace(regex.end, "")
    }
    return requires
  }

It ignores calculated require's which are not string calls such as require(a + b), but catches all others. It's also very fast -

['a', 'b', 'c'].forEach(require)...

The above sorta thing doesn't work. not sure if you care about that, but just thought i'd point it out.

Ability to handle cases where require is called with a non-literal parameter

Eg: require("dep"+"endency");

In the general case, it would be very nice if the value of statements passed into require could be statically determined (where possible), allowing detective to find dependencies described via function parameters, variables, or other non-literal sources.

It may be out of scope for this module, specifically, to have more arbitrary static analysis that can statically determine the value of statements, but it would certainly make this module more robust, and would be a pretty useful module in its own right. One case I'm thinking about is supporting something like this: https://npmjs.org/package/use

Add support for ES6 template functions

I've tried to use installify (it's based on detective) with hyperx and it fails at the template string function from this example https://github.com/substack/hyperx:

var vdom = require('virtual-dom')
var hyperx = require('hyperx')
var hx = hyperx(vdom.h)
...
return hx`<div>
    <h1>clicked ${state.times} times</h1>
    <button onclick=${onclick}>click me!</button>
  </div>`