checking dependencies recursively about citgm

[wraithan]

Last I checked the memcached module wasn't building on iojs 3. But it isn't the memcached module, but instead the hashring module. If hashring were more popular, but not popular enough to make the list of that smoke runs with any frequency... it would be breaking more of the ecosystem, so saying memcached is busted is one problem, but if all things that dep on hashring are busted then it is a bigger problem than the singular module.

The other case is where the tests don't exercise the dep sufficiently for its own use case and so you end up breaking users but not test suites, deeper dep testing could catch more of those cases.

All that said, a human going through the output could catch the first case pretty easily. The second case could be pretty well mitigated by instead of running just most popular, we use something like @chrisdickinson's whole registry scanning stuff to find modules using the specific things that changed to augment running the most popular stuff.

I'm +0 on it. For automated background stuff, totally fine, but as a tool being used in real time by humans it would likely add too much time and humans wouldn't run it. Lazy humans...

[jasnell]

If it is added, it would definitely be behind a switch that would also allow limiting the recursion depth. We would have to figure out how to output the results so that they are actually usable. Would take some playing around to get right I suspect.

[rvagg]

this would be neat coupled with a feature where we can log success/failure of a module+module-version+node-version, caching those so when we run trough a batch of top-level modules we don't end up repeating the same tests, we could churn through a large number of packages in an automated way like that

[thefourtheye]

I am +1 on this. More modules we cover better for us. But most of the dependencies may not be the latest. Would it be worth testing against them?

[mhdawson]

I'm + 1 on this as well. If you want to support version X of a module it would be nice to easily be able to validate the modules it depends on as well. An alternative to integrating directly might be a script that would take a list of 1 or more modules and generate a file that could be fed in to citgm to test those modules and all of the dependencies and we could do the de-duplication at this point.

[jasnell]

Ok, I have a few ideas on how we can proceed with usable output. Once back from nodeconfeu I'll start working on it.

[jasnell]

/cc @thealphanerd

[MylesBorins]

The more I think about this the more I wonder if there should be some sort of backing service for the caching layer. If dep X at version Y has already been tested at some point against a specific version of Node, then it doesn't really need to be run again, now does it?

[MylesBorins]

Now that I have worked on this more I'm going to suggest that we do not implement this feature unless we can find a safe way to parallelize the build jobs.

We should be able to rely on npm to create a list of all modules to be tested

[MylesBorins]

I'm closing this for now. I think the idea of recursively testing modules is nice, but tbqh using a lookup table has solved all sorts of nasty technical edge cases. I think we should be focusing on expanding that list.

Please feel free to re open if you disagree