console being held on call to sge about furrr

[DavisVaughan]

I think I'm going to keep furrr as is for now. I would use future() directly if you need more control over the lazy resolution of values

[DavisVaughan]

So you want to just use furrr as a way to send off jobs to the grid, get the future objects back immediately, and then manually call values() on that yourself when you are ready to try and collect from the grid? I could see that being useful.

I think it would just require an extra wait argument in future_options() and then a change to this line that does or does not try and resolve based on the value of wait. https://github.com/DavisVaughan/furrr/blob/master/R/future_map_template.R#L188

[yonicd]

I think it can be controlled in the tweak you define internally, can that be made into a functional expr for user to pass in?

[DavisVaughan]

What do you mean? What else would it do?

[yonicd]

i use this (from henrik's suggestion), i use sge in this case

my_sge <- future::tweak(future.batchtools::batchtools_sge, template = PATH_TO_TMPL)

future::plan(list(multiprocess, my_sge))

then i run

Y1 %<-% future_lapply(rep(300, 20),
                      FUN = function(nr){
                        solve( matrix(rnorm(nr^2), nrow=nr, ncol=nr))
                        },
                      future.scheduling = 5)

and the grid is governed by another thread and frees the console to continue working

i think you can wrap that around a functional and have furrr eval internally to what the user gives you... having a default in place in case they don't.