Comments (5)
furrr cannot work with remote clusters + progress bars with the current implementation. It needs to be able to write to a local file on Master from each Node. I've tried to state this by saying in the docs somewhere that the progress bar only works for multicore multiprocess and multisession
from furrr.
This sounds feasible for batchtools then? batchtools writes stdout to text files, for example, and the host process has access to the same file system as the nodes. Perhaps I'm confused by what Master means here.
from furrr.
Master in this case means your local computer, sorry. Where you call furrr from. I don't think in ever case you'd have access to that from your local computer. I also don't have enough experience to write or test that kind of thing
from furrr.
So yes in this case I'm calling furrr from an interactive session on the cluster, connected via ssh.
from furrr.
My best guess: The registry is created in a directory (see future.batchtools:::future_cache_path()
) which is not shared across nodes. Can you provide a traceback()
?
from furrr.
Related Issues (20)
- Error indices from purrr() are misleading in furrr()
- Getting the same random result across `purrr::map()` and `furrr::future_map()` HOT 3
- `furrr` much slower than `purrr` (on Windows) HOT 2
- Export a wrapper around purrr `map()` to workers rather than `map()` itself
- Setting `seed = TRUE` / `.progress=TRUE` globally in the R session HOT 1
- Carrier HOT 2
- mgcv::gam(~s(pc = object not found)) when future::plan(multisession, workers > 1) HOT 1
- when large matrix or df split into list, does furrr workers only access list element? HOT 1
- error: external pointer is not valid is not a particularly user actionable message HOT 5
- terra::rast() doesn't return a variable with workers > 1 HOT 3
- Understanding memory usage and performance of `furrr::future_apply` HOT 5
- Implement map_vec
- Use of variables in glue() inside furrr loops HOT 1
- Progress bar showing 100% even if not completed
- Error when plotting a {sf} object
- Identical RNG state for each task, despite setting different seeds in different tasks HOT 10
- How to pass to .f, character vectors in .x that reference global values?
- `future_map` not obviously faster than `map` in simple linear regression setting HOT 1
- Multiple calls of future_map() within a single plan() or script result in massive slowdown.
- furrr_options documentation doesn't say what to do with it
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from furrr.