pbmclapply hangs if cores is undefined about pbmcapply

[kvnkuang]

Hi Lauren and Scott,

Thanks again for providing the extra information. Sorry about the delayed response. It has been a super busy summer for me.

I am looking into this and will update here when I gain more understanding of the issue.

Regards,
Kevin

[kvnkuang]

Hi Lauren and Scott,

I was able to reproduce the bug in Ubuntu 18.01 with R version 3.6.0.

After some debugging, I realized that it was caused by errors (in Lauren's case, missing the cores variable) from mclapply not passing back to the main process. Since the actual mclapply function is running in a separate process, without signaling the main process that it has failed with an error, the main process will hang indefinitely.

I have implemented a better error capture scheme to solve this issue (cf1f924). Could you please let me know if it solves the hanging issue on your platform? You will just need to install a dev build on your machine:

library(devtools)
install_github("kvnkuang/pbmcapply", ref = "dev")

Thanks,
Kevin

[scottkosty]

Just tested and it works well. I now get the following error:

Error in mclapply(X, function(...) { : object 'cores' not found

Thanks for the fix, Kevin! And thanks for the interesting technical explanation.

[laurenfitch]

This came to my attention because I had the following bad code in my script:

num.cores <- detectCores
results <- pbmclapply(1:length(gal), repairDeletions, mc.cores = floor(num.cores*0.95))

Thanks!

[kvnkuang]

Hi Lauren,

Thanks for reporting the issue. I tried to replicate the bug on my machine but was not able to.

Here's what I got when I don't have cores defined:

> lazySqrt <- function(num) {
+     # Sleep randomly between 0 to 0.5 second
+     Sys.sleep(runif(1, 0, 0.5))
+     return(sqrt(num))
+ }
> result <- pbmclapply(1:3, lazySqrt, mc.cores = cores)
Error in pbmclapply(1:3, lazySqrt, mc.cores = cores) : 
  object 'cores' not found

I also tried your second example, in which you passed a function detectCores instead of its return to num.cores. I still wasn't able to replicate the bug.

> num.cores <- detectCores
> lazySqrt <- function(num) {
+     # Sleep randomly between 0 to 0.5 second
+     Sys.sleep(runif(1, 0, 0.5))
+     return(sqrt(num))
+ }
> results <- pbmclapply(1:3, lazySqrt, mc.cores = floor(num.cores*0.95))
Error in num.cores * 0.95 : non-numeric argument to binary operator

Could you please run the above code blocks and let me know if it runs as expected? If not, could you please tell me what's the version of pbmcapply you're using and which OS you're on?

Thanks again,
Kevin

[laurenfitch]

Hi Kevin,
Running the first code block on my local machine (macOS) I got the same error as last time:

> lazySqrt <- function(num) {
+   # Sleep randomly between 0 to 0.5 second
+   Sys.sleep(runif(1, 0, 0.5))
+   return(sqrt(num))
+ }
> result <- pbmclapply(1:3, lazySqrt, mc.cores = cores)
  |                                                                                                       |   0%, ETA NA

Same for the second block:

> num.cores <- detectCores
> lazySqrt <- function(num) {
+ # Sleep randomly between 0 to 0.5 second
+   Sys.sleep(runif(1, 0, 0.5))
+   return(sqrt(num))
+   }
> results <- pbmclapply(1:3, lazySqrt, mc.cores = floor(num.cores*0.95))
  |                                                                                                       |   0%, ETA NA

This is on macOS, session info is below. pbmcapply is version 1.4.1. I did get the same on Linux in RStudio Server and I'll post the info for that shortly.

> sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.5

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] pbmcapply_1.4.1

loaded via a namespace (and not attached):
[1] compiler_3.5.2 tools_3.5.2    yaml_2.2.0   

[scottkosty]

Hi Lauren and Kevin. First of all, thanks for reporting this type of bug, Lauren! A lot of users would just fix their own code and move on, but I appreciate your time to report this and add further details and help reduce the frustration of future users (indeed this seems like a not-so-rare situation). Seeing efforts like this remind me to do the same when I come across a similar situation. Go R community!

I can reproduce the behavior that Lauren sees by running the first block in #38 (comment). Here is my session info:

> sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 19.04

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
LAPACK: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8      
 [8] LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] colorout_1.2-0

loaded via a namespace (and not attached):
[1] compiler_3.5.2 parallel_3.5.2
> 

I added debug(pbmclapply) to the code and stepped through it. Here is the log: debug_output.log. Note that at the end, I did ctrl + \ to end the process. Looking at the log, it looks like an error is indeed correctly given but something gets stuck during the exit hook? Kevin, is there any other information I could give you that would help?

[kvnkuang]

Perfect! I am glad it works now. Thanks again for your feedback.