Trigger added immediately after watch added doesn't work about watchman

[charleseff]

Actually it's not the fault of the wrapping Ruby program - the problem seems to be about timing. I've boiled down the Ruby script to a simple Bash script:

#!/bin/sh
set -x
watchman trigger-list /Users/cf/foo
watchman watch-del /Users/cf/foo
watchman watch /Users/cf/foo
#sleep 2
watchman -- trigger /Users/cf/foo test '*.py' -- echo WORKING

When run, triggers won't work, but when uncommenting the sleep command, the trigger does work. I guess I can just add a sleep into my script, but will this always reliably work? Is there any better way without relying upon sleeps?

[wez]

Ah, this is probably a race with the initial crawl of the tree.
For now, you'll need to sleep before adding the triggers (you might find you need to sleep longer if you have more files and/or have more io load). I'll see about getting this fixed sometime this week.

Watches and triggers persist across a watchman process restart, so hopefully you wouldn't hit this problem especially frequently.

[wez]

can you clarify: "the trigger never fires" Does this mean that it never ever fires, or that it doesn't fire while your script is running, but does when you later touch a .py file?

[charleseff]

Ah thanks. I may add a 5 second sleep just to be extra safe. Would love to know when this is fixed.

I meant that it never fires, meaning I can touch *.py files at any time and they will never fire the trigger, even though the trigger shows up just fine in the trigger-list command.

[wez]

I can't reproduce this. Where are you looking for the echo output? It goes to the watchman log, which is typically /var/tmp/.watchman.$USER.log on OS/X.

[charleseff]

I've been tailing $TMPDIR/.watchman.$USER.log.
Adding the 5 second sleep seems to have fixed the problem. I will see the output in the log when adding the trigger after a 5 second sleep after adding the watch, but will not see the output when adding the trigger immediately after adding the watch.

[wez]

Can you attach your log file or otherwise get me a copy?
Also, how large is the tree that you're watching? Can you find /Users/cf/foo | wc -l and do the same but exclude any .git, .hg or .svn dirs?

[charleseff]

The log file is not interesting, it only outputs 'WORKING' when I've slept a bit before adding the trigger, then touched a .py file, and outputs nothing when I don't sleep at all before adding the trigger.

Here are the word counts you requested:

$ find . ! -path "./misc/*" | wc -l
   64501
$ find . ! -path "./.git/*" | wc -l
   23475

and for .py files:

$ find . -type f -name "*.py" | wc -l
    4556

[charleseff]

Actually every time I run the script I also get at the beginning of $TMPDIR/.watchman.$USER.log:

1400036743: tid=2017260304 file limit is 2560 kern.maxfilesperproc=10240
1400036743: tid=2017260304 raised file limit to 10240

[charleseff]

I'm actually still seeing the behavior of the trigger not firing, even when adding it with a 5 second delay after the watch add. I don't see the behavior immediately after adding the trigger but I do see after watchman's been running for some time. Maybe this is due to a host suspend/resume? I am running watchman on Mac OS X.

[wez]

Can you send me a complete log file when this happens? I wonder if there is evidence of something going awry in there

[charleseff]

I've been able to reproduce this every time:

• I've got a watchman with trigger working (by using the 5-second delay)
• I put my Mac to sleep, and wake it up
• the watchman pid is now different
• now watchman's triggers don't work.

So looks like Watchman is being restart after a sleep/resume, and somehow getting itself into the same bad state as if I didn't use the 5-second delay. Any insight into this?

[charleseff]

Ok I'll attempt to outline the commands that I run and the subsequent output in the log:

$ ../vmsetup/watchman/watchman watch-del /Users/cf/foo && pkill watchman && {command that runs watchman with trigger} && touch foo.py

output in log:
1400280846: tid=2104419088 file limit is 2560 kern.maxfilesperproc=10240
1400280846: tid=2104419088 raised file limit to 10240
FOOBAR foo.py

$ ps -ef | grep watchma[n]
1611786025 61822 1 0 3:54PM ?? 0:00.36 watchman watch-list

(then i sleep and resume my computer)

after a few seconds, new output in log:
1400280989: tid=2104419088 file limit is 2560 kern.maxfilesperproc=10240
1400280989: tid=2104419088 raised file limit to 10240

$ ps -ef | grep watchma[n]
1611786025 62666 1 0 3:56PM ?? 0:00.44 watchman watch-list

$ touch foo.py

(nothing in log)

[wez]

I think something weird is going on, because it looks like your watchman process got restarted.

If you'd like to try to debug this more in realtime, I'm hanging out in #watchman on freenode.

Stuff I'd like to know:

• How you configured watchman (are you using gimli?)
• Which version you're running (version number and/or git hash)
• which OS version you're on (I'm assuming OS/X, if so which version?)
• Are you launching the watchman server process via launchd or something?
• I'd really like to see a complete log to see what's going on

If you re-run your test, please also run watchman get-pid ; pgrep watchman at the points where you're doing the ps | grep stuff

[charleseff]

Hey Wez, sorry for the delay (vacation). Here's the answers to your questions:

• Essentially what i do is:
  • git clone watchman
  • git checkout v2.9.5 to get to the release commit
  • run ./autogen.sh && ./configure && make
    I don't know what gimli is so maybe I'm not using it
• I am at version 2.9.5, or specifically, git hash 83eff81
• OSX version is 10.9.3
• I am not launching the watchman server process via launchd. Just running a watchman command which self-daemonizes

I'll try pinging you on freenode

[wez]

Did you get to the bottom of this?

[charleseff]

Wez,
As it turns out it's totally software on my side that's restarting the process.
After fixing that, I believe I can make things reliably work by just using that 5 second sleep.
Have you found the fix to not require the sleep?

[wez]

Now that the restart thing is cleared up, can you get me a reliable repro for the sleep thing? I can't see anything obvious that would cause it to be required