debeando / lightflow Goto Github PK
View Code? Open in Web Editor NEWA flexible, light, easy to use, automation framework for typical data manipulation with terminal commands.
License: Apache License 2.0
A flexible, light, easy to use, automation framework for typical data manipulation with terminal commands.
License: Apache License 2.0
For example in the .yaml
file:
pipes:
- name: demo
execute: echo -n "demo
ignore: true
Only run this when specify pipe name in the cli args.
In both case, the output is this: ... I T2 Finished 00:00:00
lightflow$ make build && ./lightflow --config tests/duration.yaml --pipes=T1ST1P1,T1ST1P2
go generate ./...
go build -ldflags "-s -w -X github.com/debeando/lightflow/cli.BuildTime=`date +%Y%m%d%H%M`" -o lightflow main.go
2021-05-31 18:40:38 I T1/T1ST1/T1ST1P1 Starting...
2021-05-31 18:40:39 I T1/T1ST1/T1ST1P1 Finished 00:00:01
2021-05-31 18:40:39 I T1/T1ST1 Finished 00:00:01
2021-05-31 18:40:39 I T1/T1ST1/T1ST1P2 Starting...
2021-05-31 18:40:39 I T1/T1ST1/T1ST1P2 Finished 00:00:00
2021-05-31 18:40:39 I T1/T1ST1 Finished 00:00:00
2021-05-31 18:40:39 I T1 Finished 00:00:01
2021-05-31 18:40:39 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:40:39 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:40:39 I T2 Finished 00:00:00
2021-05-31 18:40:39 I Finished 00:00:01
lightflow$
lightflow$ make build && ./lightflow --config tests/duration.yaml --pipes=T1ST1P1,T1ST1P2 --ai-date="{\"start\":\"2021-05-27\",\"end\":\"2021-05-28\"}"
go generate ./...
go build -ldflags "-s -w -X github.com/debeando/lightflow/cli.BuildTime=`date +%Y%m%d%H%M`" -o lightflow main.go
2021-05-31 18:38:40 I T1/T1ST1 Increment 2021-05-27
2021-05-31 18:38:40 I T1/T1ST1/T1ST1P1 Starting...
2021-05-31 18:38:41 I T1/T1ST1/T1ST1P1 Finished 00:00:01
2021-05-31 18:38:41 I T1/T1ST1 Finished 00:00:01
2021-05-31 18:38:41 I T1/T1ST1/T1ST1P2 Starting...
2021-05-31 18:38:41 I T1/T1ST1/T1ST1P2 Finished 00:00:00
2021-05-31 18:38:41 I T1/T1ST1 Finished 00:00:00
2021-05-31 18:38:41 I T1/T1ST1 Increment 2021-05-28
2021-05-31 18:38:41 I T1/T1ST1/T1ST1P1 Starting...
2021-05-31 18:38:42 I T1/T1ST1/T1ST1P1 Finished 00:00:01
2021-05-31 18:38:42 I T1/T1ST1 Finished 00:00:01
2021-05-31 18:38:42 I T1/T1ST1/T1ST1P2 Starting...
2021-05-31 18:38:42 I T1/T1ST1/T1ST1P2 Finished 00:00:00
2021-05-31 18:38:42 I T1/T1ST1 Finished 00:00:00
2021-05-31 18:38:42 I T1 Finished 00:00:02
2021-05-31 18:38:42 I T2/T2ST1 Increment 2021-05-27
2021-05-31 18:38:42 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:38:42 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:38:42 I T2/T2ST1 Increment 2021-05-28
2021-05-31 18:38:42 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:38:42 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:38:42 I T2 Finished 00:00:00
2021-05-31 18:38:42 I Finished 00:00:02
lightflow$
$ ./lightflow --config cleanup.yaml --subtask bas --variables='{"start_date": "2020-12-03"}' --dry-run
WARN[2020-12-14 13:27:42] Running in safe mode, not execute any commands, only print commands.
panic: runtime error: index out of range [0] with length 0
goroutine 1 [running]:
github.com/debeando/lightflow/flow.(*Flow).GetPipeVariables(...)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/common.go:77
github.com/debeando/lightflow/flow.(*Flow).SetDefaults(0xc0000f1ee8)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/common.go:83 +0x4cb
github.com/debeando/lightflow/flow.(*Flow).AutoIncrement(0xc0000f1ee8, 0x0, 0xc0000942a0)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/autoincrement.go:12 +0x32
github.com/debeando/lightflow/flow.(*Flow).Subtask.func1()
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/subtask.go:20 +0x3e
github.com/debeando/lightflow/flow/iterator.(*Iterator).Loops(0xc0000c8320, 0xc0000f1b70)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:65 +0xcc
github.com/debeando/lightflow/flow/iterator.(*Iterator).Run.func1()
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:29 +0x33
github.com/debeando/lightflow/flow/duration.Start(0xc0000f1af0, 0x409bfd, 0x20)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/duration/duration.go:10 +0x50
github.com/debeando/lightflow/flow/iterator.(*Iterator).Run(0xc0000c8320, 0xc0000f1b70)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:28 +0x60
github.com/debeando/lightflow/flow.(*Flow).Subtask(0xc0000f1ee8)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/subtask.go:17 +0x1c3
github.com/debeando/lightflow/flow.(*Flow).Tasks.func1()
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/tasks.go:21 +0x3c
github.com/debeando/lightflow/flow/iterator.(*Iterator).Loops(0xc0000c82d0, 0xc0000f1d88)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:65 +0xcc
github.com/debeando/lightflow/flow/iterator.(*Iterator).Run.func1()
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:29 +0x33
github.com/debeando/lightflow/flow/duration.Start(0xc0000f1d00, 0x409bfd, 0x20)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/duration/duration.go:10 +0x50
github.com/debeando/lightflow/flow/iterator.(*Iterator).Run(0xc0000c82d0, 0xc0000f1d88)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:28 +0x60
github.com/debeando/lightflow/flow.(*Flow).Tasks(0xc0000f1ee8)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/tasks.go:17 +0x19f
github.com/debeando/lightflow/flow.(*Flow).Run(0xc0000f1ee8)
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/flow.go:30 +0x10e
github.com/debeando/lightflow/cli.Run()
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/cli/cli.go:91 +0x41a
main.main()
/Users/nsc/Sync/go/src/github.com/debeando/lightflow/main.go:8 +0x20
Like:
when: exit_code(1) == 0 ==> false
Like this:
--filters="count > 0"
Use any variables defined in pipes.
This option is used only to print when condition return true?
All variables name should by declared with lowercase and separated with - or _ and not space. Use this regular expression:
^[0-9A-Za-z\-\_]+$
curl:
method: POST
url: {{ .job_url }}
url-data:
- key1: val1
- key2: val2
header:
- Authorization: Bearer {{ .TB_TOKEN}}
Maybe in this case, the previous pipe generate empty file and decide not continue to the rest of pipes, and them skip the current pipes block and go to the next subtask.
- name: count
execute: >
mysql -h {{ .mysql_host }} -u {{ .mysql_user }} -p{{ .mysql_pwd }} thn -ABse "
SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
SELECT COUNT(*) AS count
FROM \`demo\`.\`{{ .mysql_table }}\`
WHERE {{ .find_field }} BETWEEN '{{ .date }} 00:00:00' AND '{{ .date }} 23:59:59';"
skip_on_fail: true
package main
import (
"text/template"
"log"
"bytes"
"fmt"
)
func main() {
data := make(map[string]string)
//data["AppVersion"] = "Octane_3.0"
text := "APP_VERSION={{.AppVersion}}"
tmpl, err := template.New("").Option("missingkey=zero").Parse(text)
if err != nil {
log.Fatal(err)
}
var b bytes.Buffer
err = tmpl.Execute(&b, data)
if err != nil {
fmt.Println("template.Execute failed", err)
}
fmt.Println("Template text:", text)
fmt.Println("Expanded:", b.String())
}
case:
./lightflow \
--config tinybird.yaml
--task deltatables \
--subtask emails_sent \
--variables '{"dup_cnt": "1"}' \
--ai-date='{"start": "2021-01-23", "end": "2021-02-15"}' \
--pipes export,rename,sign,replace
For example:
--variables='{"date": "2021-01-22"}' --variables '{"dup_cnt": "1"}' --dry-run
Have error because de date is empty, and maybe is incorrect to use twice --variables
.
Actually when use this argument, you see <no value>
.
For example:
{
"var": {
"foo": 1,
"bar": 2,
"level": {
"test": 3
}
}
}
New parse:
var.foo=1
var.bar=2
var.level.test=3
---
tasks:
- name: delta
subtask:
- name: d1
pipes:
- name: change foo
set:
variable: foo
expression: "..."
For example:
--ai-date='{"start": "2020-01-01", "end": "2019-07-31"}'
Retry struct {
Attempts int `yaml:"attempts"`
Wait int `yaml:"wait"`
}
Evaluate []struct {
Expression string //
Level string // None, Info, Warning, Error
Action string // Default: Terminate. Options: Terminate, Skip, When, Logging, Retry, Until
ExitCode int // Default: 1
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.