Comments (8)
Hi Lauren and Scott,
Thanks again for providing the extra information. Sorry about the delayed response. It has been a super busy summer for me.
I am looking into this and will update here when I gain more understanding of the issue.
Regards,
Kevin
from pbmcapply.
Hi Lauren and Scott,
I was able to reproduce the bug in Ubuntu 18.01 with R version 3.6.0
.
After some debugging, I realized that it was caused by errors (in Lauren's case, missing the cores
variable) from mclapply
not passing back to the main process. Since the actual mclapply
function is running in a separate process, without signaling the main process that it has failed with an error, the main process will hang indefinitely.
I have implemented a better error capture scheme to solve this issue (cf1f924). Could you please let me know if it solves the hanging issue on your platform? You will just need to install a dev build on your machine:
library(devtools)
install_github("kvnkuang/pbmcapply", ref = "dev")
Thanks,
Kevin
from pbmcapply.
Just tested and it works well. I now get the following error:
Error in mclapply(X, function(...) { : object 'cores' not found
Thanks for the fix, Kevin! And thanks for the interesting technical explanation.
from pbmcapply.
This came to my attention because I had the following bad code in my script:
num.cores <- detectCores
results <- pbmclapply(1:length(gal), repairDeletions, mc.cores = floor(num.cores*0.95))
Thanks!
from pbmcapply.
Hi Lauren,
Thanks for reporting the issue. I tried to replicate the bug on my machine but was not able to.
Here's what I got when I don't have cores
defined:
> lazySqrt <- function(num) {
+ # Sleep randomly between 0 to 0.5 second
+ Sys.sleep(runif(1, 0, 0.5))
+ return(sqrt(num))
+ }
> result <- pbmclapply(1:3, lazySqrt, mc.cores = cores)
Error in pbmclapply(1:3, lazySqrt, mc.cores = cores) :
object 'cores' not found
I also tried your second example, in which you passed a function detectCores
instead of its return to num.cores
. I still wasn't able to replicate the bug.
> num.cores <- detectCores
> lazySqrt <- function(num) {
+ # Sleep randomly between 0 to 0.5 second
+ Sys.sleep(runif(1, 0, 0.5))
+ return(sqrt(num))
+ }
> results <- pbmclapply(1:3, lazySqrt, mc.cores = floor(num.cores*0.95))
Error in num.cores * 0.95 : non-numeric argument to binary operator
Could you please run the above code blocks and let me know if it runs as expected? If not, could you please tell me what's the version of pbmcapply
you're using and which OS you're on?
Thanks again,
Kevin
from pbmcapply.
Hi Kevin,
Running the first code block on my local machine (macOS) I got the same error as last time:
> lazySqrt <- function(num) {
+ # Sleep randomly between 0 to 0.5 second
+ Sys.sleep(runif(1, 0, 0.5))
+ return(sqrt(num))
+ }
> result <- pbmclapply(1:3, lazySqrt, mc.cores = cores)
| | 0%, ETA NA
Same for the second block:
> num.cores <- detectCores
> lazySqrt <- function(num) {
+ # Sleep randomly between 0 to 0.5 second
+ Sys.sleep(runif(1, 0, 0.5))
+ return(sqrt(num))
+ }
> results <- pbmclapply(1:3, lazySqrt, mc.cores = floor(num.cores*0.95))
| | 0%, ETA NA
This is on macOS, session info is below. pbmcapply is version 1.4.1. I did get the same on Linux in RStudio Server and I'll post the info for that shortly.
> sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.5
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods base
other attached packages:
[1] pbmcapply_1.4.1
loaded via a namespace (and not attached):
[1] compiler_3.5.2 tools_3.5.2 yaml_2.2.0
from pbmcapply.
Hi Lauren and Kevin. First of all, thanks for reporting this type of bug, Lauren! A lot of users would just fix their own code and move on, but I appreciate your time to report this and add further details and help reduce the frustration of future users (indeed this seems like a not-so-rare situation). Seeing efforts like this remind me to do the same when I come across a similar situation. Go R community!
I can reproduce the behavior that Lauren sees by running the first block in #38 (comment). Here is my session info:
> sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 19.04
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
LAPACK: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8
[8] LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] colorout_1.2-0
loaded via a namespace (and not attached):
[1] compiler_3.5.2 parallel_3.5.2
>
I added debug(pbmclapply)
to the code and stepped through it. Here is the log: debug_output.log. Note that at the end, I did ctrl + \
to end the process. Looking at the log, it looks like an error is indeed correctly given but something gets stuck during the exit hook? Kevin, is there any other information I could give you that would help?
from pbmcapply.
Perfect! I am glad it works now. Thanks again for your feedback.
from pbmcapply.
Related Issues (20)
- mc.preschedule HOT 2
- progressBar() error after the 5/11/2018 update HOT 2
- Width calculation problems for short and long durations HOT 8
- Execution cannot be interrupted with Ctrl-C HOT 8
- provide a pbmcapply version of apply (not only mapply and lapply) HOT 3
- Object 'mcinteractive' not found HOT 9
- Address "cannot wait for child %d as it does not exist" warning HOT 4
- pbmclapply slow down HOT 5
- not supported by MRO? HOT 1
- pbmcapply not showing in Rscript mode HOT 3
- Execution cannot be interrupted with Ctrl-C since ignore.interactive was added HOT 1
- reproducibility
- progressBar(max=0): throws error on 'missing value where TRUE/FALSE needed'
- Custom progress bar?
- If warning, return of pbmclapply has different return format if length of X is 1 HOT 1
- pbmclapply fails to return list where lapply, pbapply, and mclapply have no issue
- learning resources for possible contributor
- Issue with differing behaviour from mcapply
- WISH: Option for outputting progress bar to standard error (stderr)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pbmcapply.