Watchman goes nuts and consumes lots of CPU about watchman

[wez]

Based on that log, I think 8ff5d80 and b9e6899 will help resolve one error case.

More troubling though is kFSEventStreamEventFlagKernelDropped, which is equivalent to the linux IN_Q_OVERFLOW. I haven't been able to find information on bumping up the equivalent limits for the mac (not even sure if they exist).

If the system is unable to keep on top of the notifications then watchman will spend a bunch of time and resources re-crawling and re-examining the tree to try to keep current.

How many discrete watches are in use on that system? Is .watchmanconfig present in those watches, or is it just running with the defaults?

[wez]

... and are those repos big (lot of files and dirs)? Roughly how many?

[dturner-tw]

If by discrete watches, you mean the length of the list watchman watch-list returns, I'm not sure (I'll ask him to report that as well next time he has issues) -- but I see maybe a couple dozen listed in the log. The watchman stuff I wrote for git does one watch per git repo.

No .watchmanconfig -- just the defaults.

None of the repos are huge -- the largest two are each roughly 25k dirs, 70k files (the rest are much smaller).

[wez]

an interesting data point might also be whether any of those watches are recursive (eg: watching both /foo/bar and /foo) as this would double the number of updates that need to be processed. We have configuration options to restrict watchman initiated watches to repo roots if it looks like this is happening.

[dturner-tw]

I would be surprised if any of them were recursive; there is never a reason for anyone to do that with how we're using watchman, and as you can see from the log, none of the listed ones (which include the largest directory trees) appear to do it.

[wez]

I'm going to close this out; I think the main contributor here was a couple of bugs in deciding to recrawl and then tripping over ourselves in the overflow-and-recrawl case. If you're still impacted by this on a more recent build, please re-open.