Comments (6)
Should wrap schedulers in classes so external folks can write their own.
from targets.
Ideally we should use the same master process, just have an hpc abstract class to submit and conclude targets.
from targets.
First order of business: add an HPC option to settings to let users decide whether to run on workers or master.
from targets.
-
Make a new "preclustermq" algorithm to pre-run all the local targets before deployment to hpc.Nope. Just repurpose local algo. Implement an option to throw a custom condition to quit early fromrun_target()
if thedeployment
setting is"remote"
. - In the clustermq algorithm itself, implement a
local_first
argument ofrun()
to optionally invoke the above. Test skipping targets with and without that option. - In the first test (all local targets) do not set any clustermq options.
- Test on a plan where the first few targets are up to date and the last ones are not. (Get going first with local algo.)
- Test on a plan with nontrivial common data (e.g global functions)
- Repeat existing clustermq tests on a real cluster. Test with tests in
tests/hpc/clustermq.R
. Includes Keras model serialization.
from targets.
Okay, just need to run the last couple tests on SGE.
from targets.
So close yet so far. It's hanging on receive_data() when there is more than one worker.
from targets.
Related Issues (20)
- Avoid polling in crew-powered pipelines HOT 2
- Allow following the best practice of not using library()
- Change progress and reporter labels HOT 1
- [BUG] tar_watch times out after a while HOT 1
- Out of memory messaging HOT 1
- `crayon.enabled` is never set back HOT 2
- targets' use of warnings is redundant, prone to false positives and negatives, and risks (causing other packages to) violating CRAN policy in normal use HOT 2
- High memory usage when dispatching many targets HOT 9
- Cleaner "skipped branch" printing strategy for large (10000+ targets) pipelines HOT 3
- Wrong debugging suggestions in debug. HOT 1
- Best Practice for Pipelines On Top of Packages
- Targets woudn't run due to error "run more than one pipeline on the same data store"
- [ideas] `description` field for targets HOT 1
- Portability issue in hashes HOT 25
- Let `targets::tar_make(reporter = "timestamp_positives")` report timestamps in local time zone HOT 4
- Patterns do not marshal properly HOT 3
- the covr badge in README does not point to main HOT 2
- [Trouble] Targets and list objects HOT 2
- Parquet targets do not support list-columns of ordered factors HOT 1
- allow reporter = "silent" for specific `tar_targets` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from targets.