GithubHelp home page GithubHelp logo

bevry / taskgroup Goto Github PK

View Code? Open in Web Editor NEW
49.0 4.0 13.0 4.09 MB

Group together synchronous and asynchronous tasks and execute them with support for concurrency, naming, and nesting.

Home Page: https://taskgroup.bevry.me

License: Other

JavaScript 98.55% HTML 1.45%
concurrency asynchronous-tasks flow-control javascript builtwith-taskgroup nodejs client-side

taskgroup's Introduction

TaskGroup

Status of the GitHub Workflow: bevry NPM version NPM downloads
GitHub Sponsors donate button ThanksDev donate button Patreon donate button Liberapay donate button Buy Me A Coffee donate button Open Collective donate button crypto donate button PayPal donate button
Discord server badge Twitch community badge

Group together synchronous and asynchronous tasks and execute them with support for concurrency, naming, and nesting.

Usage

Complete API Documentation.

Tutorials & Guides.

Web Browser Demonstration.

Install

  • Install: npm install --save taskgroup
  • Import: import * as pkg from ('taskgroup')
  • Require: const pkg = require('taskgroup')
<script type="module">
    import * as pkg from '//dev.jspm.io/[email protected]'
</script>

This package is published with the following editions:

  • taskgroup aliases taskgroup/index.cjs which uses the Editions Autoloader to automatically select the correct edition for the consumer's environment
  • taskgroup/source/index.js is ESNext source code for Node.js 6 || 8 || 10 || 12 || 14 || 16 || 18 || 20 || 21 with Require for modules
  • taskgroup/edition-browsers/index.js is ESNext compiled for web browsers with Require for modules
  • taskgroup/edition-node-4/index.js is ESNext compiled for Node.js 4 || 6 || 8 || 10 || 12 || 14 || 16 || 18 || 20 || 21 with Require for modules

This project provides its type information via inline JSDoc Comments. To make use of this in TypeScript, set your maxNodeModuleJsDepth compiler option to 5 or thereabouts. You can accomplish this via your tsconfig.json file like so:

{
  "compilerOptions": {
    "maxNodeModuleJsDepth": 5
  }
}

History

Discover the release history by heading on over to the HISTORY.md file.

Backers

Code

Discover how to contribute via the CONTRIBUTING.md file.

Authors

Maintainers

Contributors

Finances

GitHub Sponsors donate button ThanksDev donate button Patreon donate button Liberapay donate button Buy Me A Coffee donate button Open Collective donate button crypto donate button PayPal donate button

Sponsors

  • Andrew Nesbitt — Software engineer and researcher
  • Balsa — We're Balsa, and we're building tools for builders.
  • Codecov — Empower developers with tools to improve code quality and testing.
  • Poonacha Medappa
  • Rob Morris
  • Sentry — Real-time crash reporting for your web apps, mobile apps, and games.
  • Syntax — Syntax Podcast

Donors

License

Unless stated otherwise all works are:

and licensed under:

taskgroup's People

Contributors

balupton avatar crito avatar dependabot-preview[bot] avatar dependabot[bot] avatar github-actions[bot] avatar pflannery avatar sfrdmn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

taskgroup's Issues

Support for returning promises, and async/await functions.

Promises run immediately, and thus all at once, do not catch/isolate uncaught async errors, and can lose errors if .catch was not specified pedantically. Tasks run when you tell them too and using domains can catch/isolate uncaught errors within a task, and will throw errors if not handled. TaskGroup can thus control the concurrency, and be configured on how to deal with error situations - continue, pause, or by default abort and clear - and again, will throw errors if not handled. These factors alone provide why TaskGroup is superior to Promises, however, nevertheless, Promises are popular, and we should provide ways of supporting them to make integration, and perhaps conversion, with that scene easier.

The relevant layers of TaskGroup are:

  • ambi for async/sync method signature normalisation, used by Task, to wrap execute methods
  • Task and TaskGroup

Here are the options I can determine.

Consumption

Add promise consumption to ambi, such that one can do this:

// note the promise runs when the task runs
Task.create(function () {
    return new Promise(function(resolve, reject) {
        setTimeout(function () {
            const result = Math.floor(Math.random() * 1000)
                if ( result % 2 ) {
                    resolve(result)
            }
            else {
                reject(new Error('result did not become an even number'))
            }
        }, 1000)
    })
})

Add promise consumption support to Task, such that one can do this:

// note the promise runs immediately, and will not support things such as task arguments
Task.create(new Promise(function(resolve, reject) {
    setTimeout(function () {
        const result = Math.floor(Math.random() * 1000)
        if ( result % 2 ) {
            resolve(result)
        }
        else {
            reject(new Error('result did not become an even number'))
        }
    }, 1000)
})

Add promise consumption support to TaskGroup, such that one can do this:

// note the promise runs immediately, and will not support things such as task arguments
tasks.addPromise(new Promise(function(resolve, reject) {
    setTimeout(function () {
        const result = Math.floor(Math.random() * 1000)
        if ( result % 2 ) {
            resolve(result)
        }
        else {
            reject(new Error('result did not become an even number'))
        }
    }, 1000)
})

Production

Exposing promises could be done via providing a .promise getter instead of .done(callback):

// Task
Task.create(someTaskMethod).run().promise
    .then(console.log)
    .catch(console.error)

// TaskGroup
TaskGroup.create({storeResult: true}).addTasks(someTasks).run().promise
    .then(console.log)
    .catch(console.error)

I lean towards the above over having Task and TaskGroup extend the Promise class, as the promise API is an ever expanding glob or crap trying to workaround their inherent shortcomings - hence the introduction of their proposed done method and other official and non-official spec modifications implemented by the vast array of promise scene implementations.

However, the above requires the consumer to be aware that what they are consuming is a Task or TaskGroup, rather than just a promise, which if you are writing a simple library, may not be ideal. Once could write a wrapper around Task and TaskGroup, that could swap out their excellent API for the wrose Promise API. Such as:

// Include special versions
const {Task, TaskGroup} = require('taskgroup-promise')

// Task
Task.create((x, y) => x * y).run().promise
    .then(console.log)
    .catch(console.error)

// TaskGroup
TaskGroup.create({storeResult: true}).addTasks(someTasks).run().promise
    .then(console.log)
    .catch(console.error)

However, I am not sure fragmenting the TaskGroup ecosystem makes sense in this way, hence why I still lean towards a .promise getter.

Conclusions

Other ideas and discussion are welcome.

For consumption, seems adding promise consumption to ambi makes the most sense, as adding support to Task and TaskGroup means promises fire immediately, and there will be no support for things like task args. To make things easier, such that Task.create(new Promise(...)) is supported, ambi could check if the method is already a promise, before executing it and checking the return result. A question here, is what if a method returns a promise and accepts a callback as many interop libraries do.

For production, seems doing .promise is the best idea. Perhaps even with a getter for .then and .catch to alias .promise.then and .promise.catch for easier interop - however I am not sure if implementing such things will pass the promise scene's isPromise checks:

// note the promise runs when the task runs
Task.create(function (complete) {
    const p = new Promise(function(resolve, reject) {
        setTimeout(function () {
            const result = Math.floor(Math.random() * 1000)
                if ( result % 2 ) {
                    resolve(result)
            }
            else {
                reject(new Error('result did not become an even number'))
            }
        }, 1000)
    })
        if ( complete ) p.then((result) => complete(null, result)).catch(complete)
        return p  // could be `else return p` but that would not have any problem
})

Would be interesting to see how ambi or Task handles the above case, in terms of what error or enforcement it should produce, or whether it should discard the callback or the promise.

It is also worth mentioning our Chainy project, that provides the chaining abilities of promises to the TaskGroup ecosystem, in a microjs way.

An unusually large amount of memory usage

When creating 10000 small tasks I find that the process memory peeks around 170MB. When I run the same 10000 tasks with setImmediate it's doesn't go over 40MB.
So its seems that task group is generating an unusually large amount of memory overhead.

Here the code I was testing with.

var TaskGroup = require("taskgroup").TaskGroup;
var testArray = Array(10000).join().split(',');
var grp = TaskGroup.create();

testArray.forEach(function(value, index) {
  grp.addTask(function() {
    console.log("task"+index);
  });
  /*
  setImmediate(function() {
    console.log("task"+index);
  });
  */
});

grp.run();

I only tested on Windows 8.1 so not sure what a Linux box will output

Optional sorting of async callback

Hi,

could you please insert a sortable option for async callbacks, so that they will sort in order i added them to the task, not sort by firering the callback. I think it could be a good option like for data in arrays, someone add in a task, especially with parallel tasks.

Added support for nested configuration

Provide a nice API for balUtil.flow:

# Old
balUtil.flow(
    object: docpad
    action: 'generatePrepare generateLoad generateRender generatePostpare'
    args: [opts]
    next: finish
)

# Not possible as addTasks returns the tasks, not the task group chain
new TaskGroup()
    .once('complete', finish)
    .addTasks(
        [@generatePrepare, @generateLoad, @generateRender, @generatePostpare].map (i) -> i.bind(@)
        args: [opts]
    )
    .run()

# Not possible as addTasks returns the tasks, not the task group chain
new TaskGroup(next: finish).addTasks(
    [@generatePrepare, @generateLoad, @generateRender, @generatePostpare].map (i) -> i.bind(@)
    {args: [opts]}
).run()

# Not possible as we don't have tasksConfig
new TaskGroup(
    next: finish
    tasks: [@generatePrepare, @generateLoad, @generateRender, @generatePostpare].map (i) -> i.bind(@)
    tasksConfig: args: [opts]
).run()

# Not possible as tasks is only for tasks
new TaskGroup(
    next: finish
    tasks: {args:[opts}}, [@generatePrepare, @generateLoad, @generateRender, @generatePostpare].map (i) -> i.bind(@)
).run()

(node) warning: Recursive process.nextTick detected

When I use taskgroups 3.3.1 I get lots of the following output to the console

(node) warning: Recursive process.nextTick detected. 
This will break in the next version of node. 
Please use setImmediate for recursive deferral.

and I dont know if it's because of the above reason but running the code below runs up a process memory of around 600-800mb which is huge!!

# tests
outerScope = ->

    innerScope = ->
        list = []

        tasks = new TaskGroup()
        tasks.once 'complete', (err) ->
            y=1

        # add the tasks
        for indexIndex in [0...1000]
            tasks.addTask (taskComplete) ->
                list.push Date.now()
                taskComplete()

        # run the tasks
        tasks.run()

    # run 100 task groups
    for outerIndex in [0...100]
        innerScope();

outerScope()

TaskGroup doesnt work well with assertion libraries like assert and should

Mainly because it suppresses errors from callback.

Code:

var TaskGroup = require('taskgroup').TaskGroup;
var Task = require('taskgroup').Task;
var assert = require('assert');

function toTest(a, next) {
    var tasks = new TaskGroup();
    tasks.addTask('upper case', function(next) {
        a = a.toUpperCase();
        next();
    });
    tasks.done(function(err) {
        next(err, a);
    }).run();
}
function toTestWithoutTaskGroup(a, next) {
    a = a.toUpperCase();
    setTimeout(function() {
        next(null, a);
    }, 10);
}

toTest("abc", function(err, r) {
    assert.equal(r, "lmn");
});

toTestWithoutTaskGroup("abc", function(err, r) {
    assert.equal(r, "efg");
});

In above code, only the second assert (toTestWithoutTaskGroup) will report the failure. First assert will still throw the error but TaskGroup suppresses that. Maybe it shouldnt suppress errors in final callback?

Attach complete callback after the task finished.

Taskgroup currently runs the complete callback only if it got attached before the tasks finish. So the next example runs the completion callback as expected:

    var task = new Task(function(complete){
        setTimeout(function(){
            return complete(null, "an asychronous result");
        }, 5000);
    });
    task.once('complete', function(err, result){
        console.log([err, result]);
    });
    task.run();

If I attach the callback after the task already finished, it never gets called.

    var task = new Task(function(complete){
        setTimeout(function(){
            return complete(null, "an asychronous result");
        }, 1000);
    });
    task.run();
    setTimeout(function () {
        task.once('complete', function(err, result){
            console.log([err, result]);
        });
    }, 10000);

That took me a bit by surprise. Using jQuery promises I can hand a promise around and attach callbacks after the promise resolves. I find that a nice feature since it decouples running tasks from evaluating its results. You could pass taskgroups around, aggregate them, etc. In the past I found this a handy feature in long living single page apps.

I understand that taskgroup.once is implemented using EventEmitter, and therefore listeners are only run in the moment the event is triggered. But I wanted to raise this point, since I came across it.

Preinstall error when installing

Since version 3.2.2 I get to following error while installing DocPad:

> [email protected] preinstall D:\2G\quickit-schouten-machines\node_modules\docpad-plugin-cleanurls\node_modules\taskgroup
> node -e "if(require('fs').existsSync('./.git')){ require('child_process').spawn('npm', ['install','--force',require('./package.json').name], {env:process.env,cwd:process.cwd(),stdio:'inherit'}); }"


[eval]:1
"if(require('fs').existsSync('./.git')){
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: Unexpected token ILLEGAL
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:456:26)
    at evalScript (node.js:532:25)
    at startup (node.js:80:7)
    at node.js:901:3

Looks like the "preinstall" script is breaking for me.

I'm using Windows 7, Node v0.10.17 and npm v1.3.8

Your .dependabot/config.yml contained invalid details

Dependabot encountered the following error when parsing your .dependabot/config.yml:

The property '#/update_configs/0/automerged_updates/0/match/update_type' value "security" did not match one of the following values: all, security:patch, semver:patch, semver:minor, in_range

Please update the config file to conform with Dependabot's specification using our docs and online validator.

Just completed but it already completed earlier.

Hi, I keep getting this error

The task [x➞ task 2 for [y]] just completed, but it had already completed earlier, this is unexpected.

I's happening as I'm calling the same task twice however I'm creating it as a new task each time. I'd just like to check that this should be happening and isn't an issue with the library.

If it is a feature could we please have it changed to a warning level log as it's currently being caught as an error by my error handler.

Have `addTask` return the group instead of the added task

The readme says you can do this:

new TaskGroup().once('complete', next)
    .addTask(function(){})
    .addTask(function(callback){callback();})
    .run();

But doesn't addTask return the task... not the task group.

I get this error:

Object #<Task> has no method 'addTask'

Add a warning about unsupported `complete` event.

I've been bitten more than once by copying (presumably) working code that uses taskgroup (most recently https://github.com/docpad/docpad-plugin-multiplelayouts/blob/master/src/multiplelayouts.plugin.coffee#L31) and then pulling my hair out trying to figure out why it never completes.

Something like console.warn(new Error("TaskGroup 'complete' event is deprecated, please use 'completed' instead")) would save the day (or at least what's left of my hair ;)

Something similar to the async.waterfall method or async.compose?

I'm evaluating libraries to be able to do tasks serially at times in an app and I've narrowed it down to either async or taskgroup. I like the maturity of async and all the collection functions it offers, not that I absolutely need them but you know how it is. However I find taskgroup much more reader-friendly and intuitive to use. Which is a big thing right? One thing that I use and need is something similar to async's waterfall method (and to some degree the compose method) which offers you the ability to pass the result from one function in the series down to the next and so on and so on. How would I do this is taskgroup?

I looked at the code and thought maybe setConfig might allow me to do that but it seems it does check if it is a property of the object in order to set it. I could always simply set a variable in the object to the data I want to carry along but that seems less than elegant. Thoughts?

disabled optimisation issue: "bad value context for arguments value"

I've found something interesting that you may already be aware of.
The following functions all get immediately disabled for optimisation by the V8 compiler when arguments are passed to them.

TaskGroup.addTask
TaskGroup.createTask
TaskGroup.itemCompletionCallback
Task
Task.completionCallback
ambi

and others like
b (minification rocks lol)
setImmediate gets kicked as well

(On top of testing with node 0.10.22, v8 3.14.5.9, I've also tested the code with V8 version 3.23.6 using the d8 compiler and it still disables optimisation for this scenario.)

Pretty much everywhere that has args... causes V8 to immediately disable the functions from optimisation. It seems to be something to do with slice and\or passing the arguments variable to another method inside the calling method.

There is a way around this where by making a copy of the arguments manually which lets us get V8 optimization.

Workaround example:

args=[]
args.push arg for arg in arguments

or when their are static items on the stack too i.e. (item1, item2, args...) then

args=[]
args.push arg for arg in [2...arguments.length]

Love to hear your thoughts..

Task doesn't fire when changing concurrencey config between taskgroups and sub taskgroups.

In the following example:

var group = new TaskGroup();

group.setConfig({concurrency: 1});

group.addTask(function () {
  console.log('Running first task outside of group.');
});

// Add a sub-group to our exiting group
group.addGroup(function(){
  // Tell this sub-group to execute in parallel (all at once) by setting its
  // concurrency to unlimited by default the concurrency for all groups is set
  // to 1 which means that they execute in serial fashion (one after the other,
  // instead of all at once)
  this.setConfig({concurrency: 0});

  // Add an asynchronous task that gives its result to the completion callback
  this.addTask(function(complete){
    setTimeout(function(){
      console.log('Running asynchronous task in group.')
    },500);
  });

  // Add a synchronous task that returns its result
  this.addTask(function(){
    console.log("Running synchronous task in group.")
  });
});

group.addTask(function () {
  console.log('Running second task outside of group.');
});

group.run()

I get the following result:

Running first task outside of group.
Running synchronous task in group.
Running asynchronous task in group.

The second task outside of the group is not fired. If I configure the concurrency of the task group to 0 instead of 1, it does fire as expected.

Detect when a completion callback hasn't been called

Sometimes people simply forget, or programs enter in a particular state, where a completion callback is not called. This can cause unexpected problems, that unfortunately should happen. We should do something to avoid this situation.

My current thoughts are we could detect when we are idling and we are still waiting on a task to complete, if so, then we can output a warning message that lets you know which task we are still waiting for. This should be able to be turned off via a configuration option.

Would help realise problems such as #6

Pause/Resume support

I felt great when found out this awesome lib, which fit my needs when there's support for nested/grouping. Thank you very much.

But when being hooked into it, I think that it lacks the ability to pause/resume the task/group (on purpose).

I have been able to pause/resume the taskgroup (the running task still run), by:

  • Add a new status: paused
  • Add 2 methods:
    • pause(): Change this.state.status to paused
    • resume(): Change this.state.status to false & run the queue again (this.run())
  • Update some status checks

Not sure if there's any official guide for pause/resume, or someone already working on it. If my changes are going to the right direction, I can create a PR for it so you can take a look :) I guess there must be other places need to be updated accordingly :)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.