tibcosoftware / flogo-lib

Project Flogo Library

This repository has been replaced with Project Flogo 0.9.0 Core. Project Flogo 0.5.8 (this repository) will be placed in maintaince mode and only critical issues fixed.

Flogo Lib

Flogo Lib contains the api and core libraries that are used to create and extend Flogo Applications.

License

flogo-lib is licensed under a BSD-type license. See LICENSE for license text.

Serverless functions and edge microservices made painless

Getting Started | Repos | Contributing | License

Project Flogo is an open source framework to simplify building efficient & modern serverless functions and edge microservices and _this_ repository contains the api and core libraries that are used to create and extend those **Flogo Applications**.

Getting started

If you want to get started with Project Flogo, you might want to check out the getting started guide in our docs or check out the Labs section in our docs for in depth tutorials.

Repos

Project Flogo consists of the following sub-projects available as separate repos:

• flogo-cli: Command line tools for building Flogo apps & extensions
• flogo-lib: The core Flogo library (you're here now)
• flogo-services: Backing services required by Flogo
• flogo-contrib: Flogo contributions/extensions

Contributing

Want to contribute to Project Flogo? We've made it easy, all you need to do is fork the repository you intend to contribute to, make your changes and create a Pull Request! Once the pull request has been created, you'll be prompted to sign the CLA (Contributor License Agreement) online.

Not sure where to start? No problem, you can browse the Project Flogo repos and look for issues tagged kind/help-wanted or good first issue. To make this even easier, we've added the links right here too!

• Project Flogo: kind/help-wanted and good first issue
• flogo-cli: kind/help-wanted and good first issue
• flogo-lib: kind/help-wanted and good first issue
• flogo-services: kind/help-wanted and good first issue
• flogo-contrib: kind/help-wanted and good first issue

Another great way to contribute to Project Flogo is to check flogo-contrib. That repository contains all of the contributions, such as activities, triggers, etc. Perhaps there is something missing? Create a new activity or trigger or fix a bug in an existing activity or trigger.

If you have any questions, feel free to post an issue and tag it as a question, email [email protected] or chat with the team and community:

• The project-flogo/Lobby Gitter channel should be used for general discussions, start here for all things Flogo!
• The project-flogo/developers Gitter channel should be used for developer/contributor focused conversations.

For additional details, refer to the Contribution Guidelines.

License

Flogo source code in this repository is under a BSD-style license, refer to LICENSE

flogo-lib's Issues

Resolver for settings (triggers)

As a Flow developer it is likely that I'll need/want to leverage environment variables and properties within the trigger/handler settings. A resolver should be enabled to allow $env and $property resolution from trigger/handler settings.

Change log implementation

Change log from github.com/op/go-logging to flogo-lib logger wrapper

Iterators

Add support for iterator construct in flows

Support for activity configuration/settings

Like triggers, activities should be allowed to set static configuration in settings.

e.g.
{
"name": "tibco-wi-sendmail",
"version": "1.0.0",
"title": "Send Mail",
"description": "Simple Send Mail Activity",
"category": "Mail",
"settings": [
{
"name": "server",
"type": "string",
"required": true,
"value": "mail.example.com"
},
{
"name": "port",
"type": "string",
"required": true,
"value": "25"
},
{
"name": "username",
"type": "string",
"value": ""
},
{
"name": "password",
"type": "string",
"value": ""
}
],
"inputs": [
{
"name": "sender",
"type": "string",
"required": true,
"value": "[email protected]"
},
{
"name": "recipients",
"type": "string",
"required": true,
"value": "[email protected],[email protected]"
},
{
"name": "subject",
"type": "string",
"value": ""
},
{
"name": "message",
"type": "string",
"value": ""
}
]
}

Supporting customized branching condition or Link expr

Like mapper, we need provide a way to support pluggable branching condition or call link expression.

For WI case, we need provide our own branching condition code.

Subflow execution

Architect and define a contribution model for pluggable mapper support

User story:
As a framework developer the need to extend the basic mapper functionality and support more complex concepts such as inline functions, etc is required to support complex mapping between activities.

Update activities and triggers to use "ref" as an identifier

activity.json will have a new attribute "ref" that will fully qualify the activity, and flogo.json will reference the activity with activityRef attribute

Map path traversal not working in expression or mappings

When attempting to transform an activity or configure a branch condition that includes a property nested at least an object deep the transform/conditional logic does not evaluate correctly. For example: {A3.result}.current_observation.temp_f does not yield the expected result during mapping.

config.json decomposition

• Work: Engine should provide defaults for configuration items
• Work: Engine should look for env variables that override configuration items
• Work: Document configuration items
• Work: Boilerplate will not need configuration instantiation in config.go
• Benefit: No need to instantiate it in the main.go
• Benefit: Cloud ready configuration
• Projects affected: flogo-cli, flogo-lib

Make logfactory.go concurrent safe

Make the default logfactory.go concurrent safe for the logs maps

Move all configuration defaults to config.go

All configuration default values should be moved to config/config.go.

Activity and Trigger Loggers don't pick FLOGO_LOG_LEVEL=ERROR

FLOGO_LOG_LEVEL env variable should drive the logger.

Allow Flow service to be colocated with process engine

Shared objects across activities

Shared resources or objects, such as connections should be shared across activities, thus avoiding the need to reestablish connections for every activity (MQTT publish for example).

Basically the activities can belong to several flows, not all of which will have the same configuration for something like an MQTT client. The solution to this would be to have a "shared resource" in which activities that use the same configuration can reference

Reference #111

Concatenation in inputMappings

I would like to be able to concatenate multiple elements as input for a mapping.

From what I can see (judging from mapping.go), this should probably have to be a type:3 (MtExpression) mapping and then following a certain syntax in the value element, which seems to be not yet implemented. Would this be the way forward, or is there another way to achieve this (without writing a specific activity for this)?

Thanks,

Jan.

Add type to app.Config struct

We need to add the type attribute to the Struct to comply with the spec

Performance benchmarking

Need to have some performance benchmarking to have visibility over the engines performance.
Some options are:

1.- golangs benchmarking test functionality (https://golang.org/pkg/testing/#hdr-Benchmarks)
2.- External tools (needs investigation)

Sync change of environment variable name STOP_ENGINE_ON_ERROR

We are changing the key names for all flogo environment names as specified here:
TIBCOSoftware/flogo#47

We need to synchronize the change of env name STOP_ENGINE_ON_ERROR so we don't break you.

Also we need to define a better logic, some suggestions can be:

flogo_engine_onerror="STOP"
flogo_error_strategy="STOP"

Activity.Init()

In this line it gets asked if activities should have an activity.Init() method.

I would love to re-raise this question as I consider it necessary. The same way a trigger usually holds the client inside one of its attributes and its shared accross multiple trigger.Start() calls, activities that send messages such as for example the MQTT activity that is missing from flogo-contrib should be able to share a client thus not requiring to stablish a connection every time it runs. Client initialization and connection creation should be done in the activity.Init() method.

Support for Flow input schema

Remove the tight binding of triggers to flows and enable flows to define their own input and output schema. This will enable sub-flow execution and also simplify reuse of flow logic across various triggers.

Make Services pluggable

- Work: Add pluggable functionality for services
- Work: Engine should read services plugged and use contribution
- Work: Document existing services type and functionality
- Benefit: Consistency, service pluggable
- Benefit: Remove them from flows.go boilerplate
- Projects affected: flogo-cli, flogo-lib

Expression syntax (json path)

Define expression syntax in support of mapper and introduce new json complex type.

Multiple branches with conditional logic cause execution to fail

If I have multiple branches of conditional logic and one branch has invalid conditional syntax none of the branches evaluate and execution freezes.

For example, the following expression is invalid:

{
"id": 14,
"from": 2,
"to": 16,
"type": 1,
"value": "${T.pathParams}.status == "testagain""
}

app.json.log

Functions within mapper

Functions should be exposed within mappings. This will enable basic scenarios where things like concat, split, substring, etc may be required.

Cannot get output data when the output type is object type but actual value is integer type

When I doing a very simple Invoke REST Service activity which return simple number type and the output type is object type. This case: we cannot get the output data.

For example:

GET Request: http://10.97.170.106:8000/GPIO/23/value
Result: 1

I go over the code and see the issue caused by belowing code

// CoerceToObject coerce a value to an object
func CoerceToObject(val interface{}) (map[string]interface{}, error) {

    switch t := val.(type) {
    case map[string]interface{}:
        return t, nil
    default:
        return nil, fmt.Errorf("Unable to coerce %#v to map[string]interface{}", val)
    }
}

This method return only map or err which is incorrect since it is a object type.

Empty param type convert to string type

When we define a flow from Web UI which contain Send CoAP Messages activity and leave queryParams as empty or click and leave it empty.

Run the flow and it throw string cannot be convert to map[string]string error, but actually the param type should be convert either nil or empty map.

I had fix it, Frank please help me review and I will create a pull request later

Ability to get output complex_object attributes metadata from activity/trigger code

There is no way to get metadata field from output complex_object attribute inside activity/trigger code, we might need the schemas in activity/trigger code to do validation or show the structure of the fields.

Take example: I define a activity.trigger with complex_object output attribute call “response”. Today, no way to get this field from activity/trigger code.

Extend the interceptor construct

As a Flow developer I want to add an interceptor to an activity. This also implies that a contribution model needs to be defined to enable extension developers the ability to build their own interceptors.

The contrib model should support both pre/post eval implementations for the interceptor.

Triggers need to expose a reply signature

Triggers need a reply signature. The expected params that a flow can map into.

Ability to set customized log format

Ability to set customized log format or change log format to fit TCI format.

%H:%M:%S.000 logLevel messages

Sample:
21:10:01.920 INFO [mockAppThread:215] app.js - API Mock App Server started successfully on 0.0.0.0 with port 7002.

Application properties (property bag) support

We need a way to have app properties accessible from Trigger and Actions.

We need to identify the use cases and the scope of that access. Engine / Trigger / Actions

Once we identify the use cases we should design the property bag support. We should also include in this the externalization.

Cleanup of old code

Need to cleanup the following:

1.- Remove old runtime folder in activities/triggers projects
2.- Remove old cli executable
3.- Remove old moved code in action flow in flogo-lib
4.- Remove old apis in service project

Issue reading parameter from input

I have an issue reading a parameter from the input (mapping from trigger). I may not use the right expression, but this is what is generated by flogo-web and I can't find a proper example.

Attached is my app flogo.json
flogo.json.zip

And here are the DEBUG logs from execution:

`2017-11-21 11:57:50.352 DEBUG [engine] - Received work request
2017-11-21 11:57:50.353 DEBUG [engine] - Dispatching work request
2017-11-21 11:57:50.353 DEBUG [engine] - worker-2: Received Request

2017-11-21 11:57:50.353 DEBUG [engine] - No mapping specified, adding trigger outputs as inputs to action
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: proc_id, Type: string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: version, Type: integer, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: client, Type: string, Value: 127.0.0.1:49500
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: content, Type: string, Value: LEEF:1.0|Extreme Networks|Purview|1.0|Purview Flow|devTime=1467030101 proto=TCPsrc=192.168.171.17 dst=17.252.11.246 srcPort=50951 dstPort=443 Application=Apple ApplicationInt=12975 ClientOS=iOS iPhone ClientOSFamily=iOS DHCP_ClientIP=192.168.171.17 ServerIP=17.252.11.246 SwitchType=CoreFlow uuid=a0379a36
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: hostname, Type: string, Value: purview
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: message, Type: string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: facility, Type: integer, Value: 3
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: tag, Type: string, Value: appid
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: priority, Type: integer, Value: 27
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: timestamp, Type: string, Value: 2017-06-27T14:21:41Z
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: tls_peer, Type: string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: severity, Type: integer, Value: 3
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: app_name, Type: string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr: msg_id, Type: string, Value:
2017-11-21 11:57:50.354 INFO [engine] - In Flow Run uri: 'syslogs_flow'
2017-11-21 11:57:50.354 DEBUG [engine] - Getting Flow: syslogs_flow

2017-11-21 11:57:50.354 DEBUG [engine] - Creating Instance: 796117c4ede01a42f7bcd22ab5a8ee1e
2017-11-21 11:57:50.354 DEBUG [engine] - Input Attributes:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:proc_id, Type:string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.proc_id, Type:string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:version, Type:integer, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.version, Type:integer, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:client, Type:string, Value:127.0.0.1:49500
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.client, Type:string, Value:127.0.0.1:49500
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:content, Type:string, Value:LEEF:1.0|Extreme Networks|Purview|1.0|Purview Flow|devTime=1467030101 proto=TCPsrc=192.168.171.17 dst=17.252.11.246 srcPort=50951 dstPort=443 Application=Apple ApplicationInt=12975 ClientOS=iOS iPhone ClientOSFamily=iOS DHCP_ClientIP=192.168.171.17 ServerIP=17.252.11.246 SwitchType=CoreFlow uuid=a0379a36
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.content, Type:string, Value:LEEF:1.0|Extreme Networks|Purview|1.0|Purview Flow|devTime=1467030101 proto=TCPsrc=192.168.171.17 dst=17.252.11.246 srcPort=50951 dstPort=443 Application=Apple ApplicationInt=12975 ClientOS=iOS iPhone ClientOSFamily=iOS DHCP_ClientIP=192.168.171.17 ServerIP=17.252.11.246 SwitchType=CoreFlow uuid=a0379a36
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:hostname, Type:string, Value:purview
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.hostname, Type:string, Value:purview
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:message, Type:string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.message, Type:string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:facility, Type:integer, Value:3
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.facility, Type:integer, Value:3
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:tag, Type:string, Value:appid
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.tag, Type:string, Value:appid
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:priority, Type:integer, Value:27
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.priority, Type:integer, Value:27
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:timestamp, Type:string, Value:2017-06-27T14:21:41Z
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.timestamp, Type:string, Value:2017-06-27T14:21:41Z
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:tls_peer, Type:string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.tls_peer, Type:string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:severity, Type:integer, Value:3
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.severity, Type:integer, Value:3
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:app_name, Type:string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.app_name, Type:string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:msg_id, Type:string, Value:
2017-11-21 11:57:50.354 DEBUG [engine] - Attr:_T.msg_id, Type:string, Value:
2017-11-21 11:57:50.354 INFO [engine] - FlowInstance Flow: &{tibco-simple 0x160acf0 map[1:0x160acf0]}
2017-11-21 11:57:50.354 DEBUG [engine] - Scheduling EVAL on task:

2017-11-21 11:57:50.354 DEBUG [engine] - Executing instance: 796117c4ede01a42f7bcd22ab5a8ee1e

2017-11-21 11:57:50.354 DEBUG [engine] - *** Worker received result: &{map[id:0xc4201601b0 data:0xc420160210 code:0xc420160240] }

2017-11-21 11:57:50.355 DEBUG [engine] - Step: 1

2017-11-21 11:57:50.355 DEBUG [engine] - popped item off queue
2017-11-21 11:57:50.355 DEBUG [model-tibco-simple] - Task Eval: Task[%!d(string=1)]:''

2017-11-21 11:57:50.355 DEBUG [model-tibco-simple] - Has Children

2017-11-21 11:57:50.355 DEBUG [model-tibco-simple] - Task Enter: Log Message

2017-11-21 11:57:50.355 DEBUG [engine] - FromInstLinks: task=0x130c210

2017-11-21 11:57:50.355 DEBUG [model-tibco-simple] - Task Ready

2017-11-21 11:57:50.355 DEBUG [engine] - Scheduling EVAL on task: Log Message

2017-11-21 11:57:50.355 DEBUG [engine] - Step: 2

2017-11-21 11:57:50.355 DEBUG [engine] - popped item off queue
2017-11-21 11:57:50.355 DEBUG [engine] - Applying InputMapper
2017-11-21 11:57:50.355 DEBUG [engine] - AddAttr - name: {Error.type}, type: string, value:mapper

2017-11-21 11:57:50.355 DEBUG [engine] - AddAttr - name: {Error.message}, type: string, value:invalid resolution expression [${severity}]

2017-11-21 11:57:50.355 DEBUG [engine] - Done Executing A.instance [796117c4ede01a42f7bcd22ab5a8ee1e] - Status: 700

2017-11-21 11:57:50.355 DEBUG [engine] - worker-2: Completed Request`

Define flogo.json and flogodata.json

Work: define flogo.json and flogodata.json structure and names

Handler mappings

Introduce the concept of mappings within the handler to support the decoupling of of triggers + flows.

Engine should not be started when trigger reports error during start

When trigger instance reports error during start, the engine should not be started.
Currently, engine ignores the error and get started despite trigger failure.

activities.Factory

¿Why do the triggers follow the Factory pattern and the activities instead just use a constructor method?

Support for embedded application

We should support embedding of flogo.json into the executable.
This will make executables more portable as users only have to manage single executable file unlike today where they have to manage both executable and flogo.json files.

This is also good from security perspective as no one can see contents of application json file.

Application monitoring hooks

There should be a way to monitor the application. So there should be some hooks to help that monitoring.

We need to have design discussions about this.

Ability to plug-in logging implementation which is common across activities/triggers

The FLOGO engine should let plug-in third-party logging implementations that can help segregate engine logs vs application logs vs activity/trigger logs.

OOB support for circuit breaker pattern in Flogo engine

We should think about adding built-in support for circuit breaker pattern in Flogo engine.
https://martinfowler.com/bliki/CircuitBreaker.html
https://github.com/Netflix/Hystrix/wiki/How-it-Works

Flows need Output Params

A flow needs to define its output params, as well as input params.

Error must be reported for unsupported activity configuration type

Currently, no validation is performed for activity configuration type.
No error is reported for following invalid configuration.
e.g.
"attributes": [
{
"name": "MyCustomField",
"value": "MyValue",
"type": "foo"
}]

Unsupported types(foo in this case) should be reported to the activity developer.

Make Actions contribution

• Flows should be an implementation of actions
• Flows should be contributed, so figure out contribution model and metadata
• Reference by path/name

Ability to collect application execution metrics

There should be pluggable way to collect application execution data for flow instances/activities/transitions.
This will help build app monitoring solutions for Flogo as well as WI.

Error handling for link expr

From an end user, they need error handler while executing branching linker.

Any error happening inside link should trigger error handler(Erro flow).

Add error return for EvalLinkExpr and make sure the error handled by engine

type LinkExprManager interface {

	// EvalLinkExpr evaluate the link expression
	EvalLinkExpr(link *Link, scope data.Scope) bool
}

Update flowdef/schema.json with new structure

After app support there were a couple of minor changes in the schema.json that need to be updated.

Also review trigger and activity schemas

Error handling for mapper

So far there is no error return to the engine while doing the mapping. code is here:

We should provide error handling while mapper goes wrong and make it to error flow.

Common API to create errors in the trigger

For better consistency, there should be a common API to create trigger errors like activity.NewError().
It would be helpful once error handler support is added for Triggers.