Error in process_kill(self, private, grace) : process_kill: Operation not permitted about callr

[wch]

It might be the case that it's impossible to avoid a race here. If so, I think it would be best not to throw an R error.

[gaborcsardi]

Why can't I reproduce this? Can you please show session info?

[wch]

─ Session info ───────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 3.4.3 (2017-11-30)
 os       macOS High Sierra 10.13.3   
 system   x86_64, darwin15.6.0        
 ui       RStudio                     
 language (EN)                        
 collate  en_US.UTF-8                 
 tz       America/Chicago             
 date     2018-03-28                  

─ Packages ───────────────────────────────────────────────────────────────────────
 package     * version    date       source                              
 assertthat    0.2.0      2017-04-11 CRAN (R 3.4.0)                      
 callr       * 2.0.2.9002 2018-03-26 Github (r-lib/callr@f981a0b)        
 clisymbols    1.2.0      2017-05-21 CRAN (R 3.4.0)                      
 crayon        1.3.4      2017-09-16 CRAN (R 3.4.1)                      
 debugme       1.1.0.9000 2018-01-15 Github (gaborcsardi/debugme@f4a116a)
 R6            2.2.2      2017-06-17 CRAN (R 3.4.0)                      
 rstudioapi    0.7        2017-09-07 CRAN (R 3.4.1)                      
 sessioninfo   1.0.0      2017-06-21 CRAN (R 3.4.1)                      
 withr         2.1.2      2018-03-15 CRAN (R 3.4.4)                      
 yaml          2.1.18     2018-03-08 CRAN (R 3.4.4)                      

[gaborcsardi]

So this does not fail for me in a new session:

for (i in 1:1000) { p <- process$new('ls'); p$kill() }

[gaborcsardi]

Does this fail for you? If not, is there another good way to make it fail?

[gaborcsardi]

So it does fail for you both in Linux and macOS, right?

[gaborcsardi]

Aha, got it:

❯ sapply(1:100, function(x) { p <- process$new("ls", "/"); Sys.sleep(0.002); p$kill() })
Error in process_kill(self, private, grace) :
  process_kill: Operation not permitted

[gaborcsardi]

So this is coming from here:

callr/src/unix/callr.c

Line 701 in f981a0b

error("process_kill: %s", strerror(errno));

According to the manual:

kill() will fail and no signal will be sent if:

[EINVAL] Sig is not a valid, supported signal number.

[EPERM]  The sending process is not the super-user and its effective user id does not match the effec-
         tive user-id of the receiving process.  When signaling a process group, this error is
         returned if any members of the group could not be signaled.

[ESRCH]  No process or process group can be found corresponding to that specified by pid.

[ESRCH]  The process id was given as 0, but the sending process does not have a process group.

so this why I handle ESRCH specially, to avoid throwing an R error. The EPERM return value is bit unexpected, to be honest, but it is what it is.... I'll just handle that specially as well...

[gaborcsardi]

Please check if the fix works for you, it should, it looks good here:

❯ sapply(1:100, function(x) { p <- process$new("ls", "/"); Sys.sleep(0.0015); p$kill() })
  [1]  TRUE  TRUE  TRUE  TRUE FALSE  TRUE  TRUE FALSE  TRUE  TRUE FALSE  TRUE
 [13] FALSE FALSE FALSE  TRUE  TRUE FALSE  TRUE FALSE  TRUE  TRUE FALSE  TRUE
 [25] FALSE  TRUE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE  TRUE  TRUE FALSE
 [37]  TRUE  TRUE FALSE  TRUE  TRUE FALSE  TRUE FALSE  TRUE  TRUE FALSE  TRUE
 [49]  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE  TRUE FALSE  TRUE  TRUE  TRUE
 [61]  TRUE FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE FALSE FALSE  TRUE FALSE
 [73]  TRUE  TRUE FALSE FALSE  TRUE  TRUE FALSE FALSE  TRUE  TRUE  TRUE  TRUE
 [85]  TRUE FALSE FALSE  TRUE  TRUE FALSE  TRUE FALSE  TRUE  TRUE FALSE  TRUE
 [97] FALSE  TRUE FALSE  TRUE

[wch]

Your computer must be faster than mine. :)

The fix does look like it worked -- I no longer see the error message.

This doesn't fix the other error I was seeing from Travis, which I believe was from here:

callr/src/unix/callr.c

Line 690 in 9c38827

error("callr_kill: %s", strerror(errno));

[wch]

From what I can find, the problem is that the process is reaped before waitpid() is called. https://stackoverflow.com/a/19249944/412655

My guess is that this is more likely to happen on Travis because the machines tend to be slow and bogged down with a lot other processes.

I've tried to repro the error with the following, but haven't succeeded:

p <- process$new('ls')
# Wait for a long while
.Call(callr:::c_callr_kill, p$.__enclos_env__$private$status, 0.1)

[gaborcsardi]

Oops, sorry I missed that it was a different issue.

Good thing I forgot to update the error message, now we know which one is which..... I'll fix that in a sec as well....

[gaborcsardi]

Yeah, so this one is more difficult, because it seems to me that that error can only happen if another package overwrites our signal handler.

The parallel package does that for fork clusters for example. httpuv has a sigaction call: https://github.com/rstudio/httpuv/blob/2bf07cce242253097745ba6d084fd215627c88e5/src/libuv/src/unix/signal.c#L222
Is that ever executed? Probably not, because it would lead to more severe problems...

[gaborcsardi]

Looks like it is not executed, good. I wonder which package it could be then..... but anyway, this is something that callr needs to sort out somehow.....

[gaborcsardi]

Btw. can you reproduce the Travis error somehow?

[wch]

I still haven't been able to reproduce the Travis error locally.

Here's another test log that shows it. I noticed that there's also callr_get_exit_status: No child processes :
https://travis-ci.org/rstudio/shinytest/jobs/359456469#L3078-L3083

 Error in process_signal(self, private, signal) : 
    callr_get_exit_status: No child processes
  Calls: <Anonymous> ... <Anonymous> -> sd_stop -> <Anonymous> -> process_signal -> .Call
  Error in process_kill(self, private, grace) : 
    callr_kill: No child processes
  Calls: <Anonymous> ... <Anonymous> -> sd_stop -> <Anonymous> -> process_kill -> .Call

[gaborcsardi]

Yeah, both of these functions assume that the exit code was either not collected yet (via wait or waitpid), or it was collected by us.

It is interesting, though that they seem to come in pairs... I think I can fix this, at some level. The exit code might be lost forever in this case, but we don't have to error, I'll just set it to NA.

[wch]

FWIW, I think the pairs were just a coincidence. Here's a case where it was just one:
https://travis-ci.org/rstudio/shinytest/jobs/359576507#L1225-L1227

[gaborcsardi]

OK, that makes sense.