danielgerlag / workflow-es

Workflow / durable task library for Node.js (or modern browsers)

Workflow ES

Workflow ES is a workflow / saga library for Node.js (or modern browsers). It supports pluggable persistence and concurrency providers to allow for multi-node clusters.

Installing

Install the core npm package "workflow-es"

npm install workflow-es --save

Guides

• Javascript (ES6)
• Typescript

Persistence

Since workflows are typically long running processes, they will need to be persisted to storage between steps. There are several persistence providers available as seperate npm packages.

• Memory Persistence Provider (Default provider, for demo and testing purposes)
• MongoDB
• (more to come soon...)

Multi-node clusters

By default, the WorkflowHost service will run as a single node using the built-in queue and locking providers for a single node configuration. Should you wish to run a multi-node cluster, you will need to configure an external queueing mechanism and a distributed lock manager to co-ordinate the cluster. These are the providers that are currently available.

Queue Providers

• SingleNodeQueueProvider (Default built-in provider)
• Azure
• Redis

Distributed lock managers

• SingleNodeLockProvider (Default built-in provider)
• Azure
• Redis Redlock

Authors

• Daniel Gerlag - Initial work

License

This project is licensed under the MIT License - see the LICENSE.md file for details

workflow-es's Issues

Workflow Engine stops after sometime due to workflow-es-mongodb (getRunnableEvents)

Sometimes we are receiving null data on getRunnableEvents due to some other reason. I have no idea why that is happening. Probably some check inside the function will help the engine running for other workflow execution requests.

The data variable becomes null sometimes. Refer:

workflow-es/providers/workflow-es-mongodb/src/mongodb-provider.ts

Line 162 in 15f5e57

for (let item of data)

Questions regarding .when(...)

I have pretty much worked out all the sample code now but am still struggling with the .when(...) element. What I (think I) do understand....

• .when(...) provides alternatives for outcome of the step they are attached to.
• each .then(..) attached to the .when(...) provides the sequence of steps that will take place if the condition for the .when(...) is met.

What I don't understand is how to trigger the condition. The sample has constants set in it and the "outcome" step sets the .outcome(...) on the ExecutionResult BUT, it seems like both .when(...) steps are always being triggered as the workflow runs (regardless of the value in the outcome).

From the sample, I would have assumed (given the input of 7) that the outcome would be set to 1 and subsequently trigger TaskC and TaskD. However, the source code implies that .when(...) takes a function.

Additionally, I assume the .name(...) important with regard to this process as it seems related to the .end(...) after each .when(...).

I feel like I am missing something....

Regards,
Gary

Limit retries based on maxRetryCount

How do set max retry count? I have a requirement that I need to do 3 reties before concluding the step as error.

Additional primitives

Add Recur, Delay and Schedule primitives as per workflow-core

How to resume a workflow execution?

I'm pretty new to workflow concepts and trying to implement few long running processes. I'm trying to understand two things.

Resuming the workflow
I want the workflow to be triggered by an event or a set of events. The events are received from the messaging system. I can read the messages and feed it into the workflow to start it but once it is waiting (like stored the state on the provider). At this stage I want certain conditions to be set in the storage to be able trigger the next stage:

a. Does a workflow constantly monitors the intermediate saved state for the changes?
b. Should I insert some event and trigger the event to resume the workflow?
c. Or there is some other way to handle this case ?

Scalability in cloud
My Second problem is related to scalability in cloud based system:
Can I run a group of workflow (think of a cluster) in kubernetes to scale it or a provider works with a single workflow engine?

Thanks and feature request

Thank you for this great work - will be downloading and using soon. Appreciate the effort, docs, and clean code.

FWIW: +1 on a generic RDBMS (e.g., JDBC) connector for a SQL persistence option.

Repeated runnable instances on the same queue

Hi Daniel, great library and thanks for all support!

I've faced before multiple process running the same workflow, which was solved using a lock manager.
But recently I've noticed that in very rare occasions, a workflow is ran twice with the same process. One thing that I noticed from the logs is that they were dequeued at the exact same time, which makes me think that it's possible to queue the same workflow more than one time.

I didn't manage to create a reproducible test for this, so I was wondering if you have any thoughts or possible reasons that might lead to this issue.

Thanks

Load workflow definitions from json

Add a feature to load workflow definitions from a json file instead of defining them in code.

Can't make make sequential for loop

I need to run few set of steps in sequentially inside for-each. Current ForEach step is parallel step. Sometimes I might need to load the collection dynamically (based on external web service).

Is it possible to make for-each as sequential?

Output for "waitFor" is executed twice

Hi,

It seems that when I set the output for a "waitFor" step, the output function is being called twice;

Execution log:

Started workflow: c41127b43cef0
Waiting for event...
event function
Publishing event...
Published event
event function
The event data is hi!

The sample code:

const workflow_es = require('workflow-es');

class LogMessage extends workflow_es.StepBody {
  run(context) {
    console.log(this.message);
    return workflow_es.ExecutionResult.next();
  }
}


class EventSample_Workflow {
  constructor() {
    this.id = 'event-sample';
    this.version = 1;
  }

  build(builder) {
    builder
      .startWith(LogMessage)
      .input((step, data) => step.message = 'Waiting for event...')
      .waitFor('myEvent', (data) => '0')
      .output((step, data) => {
        console.log('event function');
        data.externalValue = step.eventData;
      })
      .then(LogMessage)
      .input((step, data) => step.message = 'The event data is ' + data.externalValue);
  }
}

async function main() {
  const config = workflow_es.configureWorkflow();
  const host = config.getHost();

  host.registerWorkflow(EventSample_Workflow);
  await host.start();

  let id = await host.startWorkflow('event-sample', 1, {});
  console.log('Started workflow: ' + id);

  await new Promise(resolve => setTimeout(() => resolve(), 5000));

  console.log('Publishing event...');
  await host.publishEvent('myEvent', '0', 'hi!', new Date());
  console.log('Published event');
}

main();

Thanks

waitFor - multiple events

Hello,

This is more of a question than a bug. I've just started looking at using workflow-es and it looks straightforward for my needs.

However, I've been trying to play around with the idea of a workflow that would have multiple waitFor steps but that are not sequential (that is, think of 3 people having to "approve" the workflow before it could enter the step - those approvals could be in any order).

While I can put multiple waitFor steps in, the engine seems to get confused.

I have thought about using a "while" but I would still need to persist some data to know when all of the previous events had been triggered.

[sorry if you prefer this type of question via a different forum]

NestJS dependency injection

Hi,

I'm trying to use workflow-es with nestJs.

I can't figure how to inject services into a Step.

Thanks

Will it be updated ？

I really like this open source project, will it be updated?

Error handling

Hi,

Is there a way to execute some code if workflow fails?

Thanks

No workflow-es-mysql

Under providers, under workflow-es-mysql, the readme states:

`Installing
Install the npm package "workflow-es-mysql"

npm install workflow-es-mysql --save
`

This npm package does not exist.

workflow-es-mongodb 2.1.1 was not built before it was published to NPM!

So the costructor in src/mongodb-provider.ts in the source code looks like this:

    constructor(connectionString: string) {
        var self = this;
        this.connect = new Promise<void>((resolve, reject) => {
            const options =  { useNewUrlParser: true,  useUnifiedTopology: true };
            MongoClient.connect(connectionString, options, (err, client) => {
                if (err)
                    reject(err);
                self.client = client;
                const db = self.client.db();
                self.workflowCollection = db.collection("workflows");
                self.subscriptionCollection = db.collection("subscriptions");
                self.eventCollection = db.collection("events");
                resolve();
            });
        });
    }

But the costructor in build/mongodb-provider.js in the built code from the npm package looks like this

    function MongoDBPersistence(connectionString) {
        this.retryCount = 0;
        var self = this;
        this.connect = new Promise(function (resolve, reject) {
            console.log("Connecting to mongo...");
            mongodb_1.MongoClient.connect(connectionString, function (err, db) {
                console.log("Connected to mongo");
                self.db = db;
                self.workflowCollection = self.db.collection("workflows");
                self.subscriptionCollection = self.db.collection("subscriptions");
                self.publishCollection = self.db.collection("unpublished");
            });
        });
    }

Note that this built code makes no reference to self.client. I believe that the source code was not rebuilt before it was published to NPM.

If you can fix this ASAP it would be much appreciated!

Ever been used in an Angular app?

Do you know of / can you point me to any examples of workflow-es being used in an Angular app? Clearly the TS would enable that, but we are about to make a significant architectural decision about worklfow, and I'd like to use this library. But I'd sure love to see it being used in an Angular app before we make that commitment. (We're using Angular 6)

Any suggestions?

Thanks!

Get number of steps inside a StepBody

Is there a way to get the count of steps of a workflow inside the StepBody?

Self Transitions and Feedback loops

Could you please provide guidance on how best to implement a self-transition? That is a state that transitions back to itself? Ideally, it would do this conditionally. So, maybe it transitions to an If that can transition back to the original state?

Geneally, the framework seems to only support acyclic paths through the state machine. Surely you can have a return transition that goes back to a previous state? Would you just 'reinstantiate' that step again in the WF definition?

I suspect it would involve ExecutionResult.branch() but there is apparently no documentation of that nor any samples that use it. The If example also appears to be a once-through with no feedback loop.

Any help would be greatly appreciated. If all workflows have to be acyclical, that would be a dealbreaker for us using it.

Workflow introspection

I was wondering if this lib exposes any clean way to introspect workflow steps and workflow states.

1. Introspect workflow steps: Because I would like to create a graph of my workflow that will be displayed on our admin.

2. Introspect workflow states: Im seeking a way to list workflow instances and their states, because I would like to show which step a given instance of a workflow is on.

Thanks

Restart workflow

Hi,

Thanks for the great project.

I'm looking for an option to restart a workflow from within a step.

Is there a way to do so?

Thanks

waitFor - multiple events

Hi,

I'm trying to do as the following:

I want my workflow to have multiple paths.
for example, I want to wait for 2 events. 1 named "manager" and the other named "client"

if the manager event happened then continue in 1 path and if the "client" event happened then continue in the second path.

But I need the workflow not to wait until both events happen.

I need it to continue and complete when the first (of 2) event happens

This is my example code:

export class HelloWorldWorkflow implements WorkflowBase<any> {
  public id: string = 'hello-world';
  public version: number = 1;

  public build(builder: WorkflowBuilder<any>) {
    builder
      .startWith(HelloWorld)
      .parallel()
      .do(branch1 => branch1
        .startWith(HelloWorld1)
        .waitFor('manager', data => '0')
        .output((step, data) => step.eventData && console.log(`Event manager`))

        .then(HelloWorld2),
      )
      .do(branch2 => branch2
        .startWith(HelloWorld3)
        .waitFor('client', data => '0')
        .output((step, data) => step.eventData && console.log(`Event client`))
        .then(HelloWorld4),
      )
      .join()
      .then(GoodbyeWorld);
  }
}


But in my example, the workflow will end only after both events happen.

Thanks

 

load work flow json please

Hi, do not be tired. Be the champ. Please register with Json. Thanks as soon as possible.
Your son is so beautiful and like you

A complete sample of a workflow back to the previous step

pleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleasepleaseplease

-

support typescript 2.4+

Sounds like it's related to https://github.com/Microsoft/TypeScript/wiki/Breaking-Changes#typescript-24 without digging into it much.

Async steps body

Hi,
How to implement async step body... the system crashes when i add await to step body....

Workflow never resumed after waitFor

Workflow never resumed after waitFor
Based on an example in https://github.com/danielgerlag/workflow-es/blob/master/samples/node.js/typescript/04-events.ts

Node v12.11.1
Typescript: 3.5.1
workflow-es: "^2.3.5"
Source code: https://github.com/huksley/workflow-runner

Reproducing:

yarn install
yarn build
node build/src/main.js

This results in hanging because invoking publishEvent does not result in resuming the execution, here are the log:

Started workflow: d63f961b0d715
LogMessage: Waiting for event... d63f961b0d715
Pinging d63f961b0d715
Got ping, sending event
<node hangs here...>

Workflow Persistence with MySQL

Hi Daniel, is there a way to persist the workflow with a MySQL Database? I was looking into providers folder and it has only the mongodb-provider.

FYI - imports missing in 06-deferred-steps.ts example

configureWorkflow and ConsoleLogger missing in the sample code

Error handling question

Hey Daniel (fellow Canadian here 🇨🇦),

Great work on the package. Looks very thought-out and I think has a big future and potential!

One question I have is about error handling.

There are no docs about what happens when a step fails.

Is that by design? Do you think this is outside the scope of this package and that business logic should handle the errors?

E.g. if error -> branch to an error handling step, which then waits & re-queues or marks as failed in the app in some way. Is this how the app should handle it?

Thanks.

Poll Worker Interval Override

Hi Daniel,
I'd like to set processTimer interval of PollWorker to 1 000 milliseconds.

Is there any strong reason to be set to default 10 000 milliseconds without possibility to override it programmatically? Is there any other risk than increased consumption of processor's resources?

Thanks and regards,,
Jan

PS: I used your workflow-core before on .NET projects and I'm glad I can continue with the same solution in Javascript as wel. Big thanks!

Remove underscore

Remove dependence on underscore library by replacing with new built-in ECMA script .find() method.

Issue passing data to parallel

I'm trying to pass the same object through my parallel branches but the data seems to be coming up undefined in the branch. Couldn't find a related examples, is there something I need to add for the parallel case? Thanks for the help!

.startWith(Action1)
.input((step, data) => step.var = data.var)
.output((step, data) => data.var = step.var) <--- defined
.parallel()
.do(branch1 => branch1
.startWith(Action2)
.input((step, data) => step.var = data.var) <--- undefined
.output((step, data) => data.var = step.var)
)
...

Apparent Bug in 06-deferred-steps.ts

I'm sorry I can't post the entire stack trace - I work on a closed system...

Running 06-deferred-steps.ts in VS Code:

   Error: the number of constructor arguments in in the derived class bodyClass must be >= than [sic] the number of constructor arguments of its base class.
   at /....../node_modules/inversify/lib/planning/planner.js:111:27
    :

Multiple instances of step running at the same time

I'm having an issue where multiple instances of the same step run "at the same time".

I was wondering what are the chances of multiple instances of my application (running with load-balancer) hitting the database and getting the same RunnableInstances?

I looked into the code base and saw the function below, which seems to not work well with multiple instances, right?

private async process(self: PollWorker): Promise<void> {                
    self.logger.info("pollRunnables " + " - now = " + Date.now());
    //TODO: lock
    try {        
        let runnables = await self.persistence.getRunnableInstances();
        for (let item of runnables) {                    
            self.queueProvider.queueForProcessing(item, QueueType.Workflow);
        }
    }
    catch (err) {
        self.logger.error("Error running poll: " + err);
    }

    try {
        let events = await self.persistence.getRunnableEvents();
        for (let item of events) {
            self.queueProvider.queueForProcessing(item, QueueType.Event);
        }
    }
    catch (err) {
        self.logger.error("Error running poll: " + err);
    }
}

Callbacks needed for post-start and post-complete workflow execution

We have an requirement that users should be notified on workflow start and completed.

@danielgerlag afterWorkflowIteration function on workflow-step-base is executed before determineNextExecutionTime. So, we don't get status of workflow in WorkflowInstance argument.

Any idea how we get status of the workflow on realtime instead of polling database with db-provider?