debeando / lightflow Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 0.0 3.8 MB

A flexible, light, easy to use, automation framework for typical data manipulation with terminal commands.

License: Apache License 2.0

Go 94.50% Makefile 1.20% Shell 4.31%

automation-framework csv data etl-pipeline golang mysql

lightflow's People

Contributors

Stargazers

Watchers

lightflow's Issues

When finish retry no verify error conditions

https://api.tinybird.co/v0/jobs/331317f4-5526-4f00-ac2e-2a3db4d2ce1f

When one pipe received exit code, skip complete pipes set.

Move execute class into retry class.

Skip parameter not found in ai

Allow specific pipe to ignore in overall process

For example in the .yaml file:

pipes:
  - name: demo
    execute: echo -n "demo
    ignore: true

Only run this when specify pipe name in the cli args.

No count ET

In both case, the output is this: ... I T2 Finished 00:00:00

lightflow$ make build && ./lightflow --config tests/duration.yaml --pipes=T1ST1P1,T1ST1P2
go generate ./...
go build -ldflags "-s -w -X github.com/debeando/lightflow/cli.BuildTime=`date +%Y%m%d%H%M`" -o lightflow main.go
2021-05-31 18:40:38 I T1/T1ST1/T1ST1P1 Starting...
2021-05-31 18:40:39 I T1/T1ST1/T1ST1P1 Finished 00:00:01
2021-05-31 18:40:39 I T1/T1ST1 Finished 00:00:01
2021-05-31 18:40:39 I T1/T1ST1/T1ST1P2 Starting...
2021-05-31 18:40:39 I T1/T1ST1/T1ST1P2 Finished 00:00:00
2021-05-31 18:40:39 I T1/T1ST1 Finished 00:00:00
2021-05-31 18:40:39 I T1 Finished 00:00:01
2021-05-31 18:40:39 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:40:39 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:40:39 I T2 Finished 00:00:00
2021-05-31 18:40:39 I Finished 00:00:01
lightflow$

lightflow$ make build && ./lightflow --config tests/duration.yaml --pipes=T1ST1P1,T1ST1P2 --ai-date="{\"start\":\"2021-05-27\",\"end\":\"2021-05-28\"}"
go generate ./...
go build -ldflags "-s -w -X github.com/debeando/lightflow/cli.BuildTime=`date +%Y%m%d%H%M`" -o lightflow main.go
2021-05-31 18:38:40 I T1/T1ST1 Increment 2021-05-27
2021-05-31 18:38:40 I T1/T1ST1/T1ST1P1 Starting...
2021-05-31 18:38:41 I T1/T1ST1/T1ST1P1 Finished 00:00:01
2021-05-31 18:38:41 I T1/T1ST1 Finished 00:00:01
2021-05-31 18:38:41 I T1/T1ST1/T1ST1P2 Starting...
2021-05-31 18:38:41 I T1/T1ST1/T1ST1P2 Finished 00:00:00
2021-05-31 18:38:41 I T1/T1ST1 Finished 00:00:00
2021-05-31 18:38:41 I T1/T1ST1 Increment 2021-05-28
2021-05-31 18:38:41 I T1/T1ST1/T1ST1P1 Starting...
2021-05-31 18:38:42 I T1/T1ST1/T1ST1P1 Finished 00:00:01
2021-05-31 18:38:42 I T1/T1ST1 Finished 00:00:01
2021-05-31 18:38:42 I T1/T1ST1/T1ST1P2 Starting...
2021-05-31 18:38:42 I T1/T1ST1/T1ST1P2 Finished 00:00:00
2021-05-31 18:38:42 I T1/T1ST1 Finished 00:00:00
2021-05-31 18:38:42 I T1 Finished 00:00:02
2021-05-31 18:38:42 I T2/T2ST1 Increment 2021-05-27
2021-05-31 18:38:42 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:38:42 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:38:42 I T2/T2ST1 Increment 2021-05-28
2021-05-31 18:38:42 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:38:42 I T2/T2ST1 Finished 00:00:00
2021-05-31 18:38:42 I T2 Finished 00:00:00
2021-05-31 18:38:42 I Finished 00:00:02
lightflow$

Define new variable pipe_eta

Maybe retry not fount, please verivy

fix ai-date

Bug on use template variable

$ ./lightflow --config cleanup.yaml --subtask bas --variables='{"start_date": "2020-12-03"}' --dry-run
WARN[2020-12-14 13:27:42] Running in safe mode, not execute any commands, only print commands.
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/debeando/lightflow/flow.(*Flow).GetPipeVariables(...)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/common.go:77
github.com/debeando/lightflow/flow.(*Flow).SetDefaults(0xc0000f1ee8)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/common.go:83 +0x4cb
github.com/debeando/lightflow/flow.(*Flow).AutoIncrement(0xc0000f1ee8, 0x0, 0xc0000942a0)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/autoincrement.go:12 +0x32
github.com/debeando/lightflow/flow.(*Flow).Subtask.func1()
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/subtask.go:20 +0x3e
github.com/debeando/lightflow/flow/iterator.(*Iterator).Loops(0xc0000c8320, 0xc0000f1b70)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:65 +0xcc
github.com/debeando/lightflow/flow/iterator.(*Iterator).Run.func1()
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:29 +0x33
github.com/debeando/lightflow/flow/duration.Start(0xc0000f1af0, 0x409bfd, 0x20)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/duration/duration.go:10 +0x50
github.com/debeando/lightflow/flow/iterator.(*Iterator).Run(0xc0000c8320, 0xc0000f1b70)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:28 +0x60
github.com/debeando/lightflow/flow.(*Flow).Subtask(0xc0000f1ee8)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/subtask.go:17 +0x1c3
github.com/debeando/lightflow/flow.(*Flow).Tasks.func1()
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/tasks.go:21 +0x3c
github.com/debeando/lightflow/flow/iterator.(*Iterator).Loops(0xc0000c82d0, 0xc0000f1d88)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:65 +0xcc
github.com/debeando/lightflow/flow/iterator.(*Iterator).Run.func1()
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:29 +0x33
github.com/debeando/lightflow/flow/duration.Start(0xc0000f1d00, 0x409bfd, 0x20)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/duration/duration.go:10 +0x50
github.com/debeando/lightflow/flow/iterator.(*Iterator).Run(0xc0000c82d0, 0xc0000f1d88)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/iterator/iterator.go:28 +0x60
github.com/debeando/lightflow/flow.(*Flow).Tasks(0xc0000f1ee8)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/tasks.go:17 +0x19f
github.com/debeando/lightflow/flow.(*Flow).Run(0xc0000f1ee8)
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/flow/flow.go:30 +0x10e
github.com/debeando/lightflow/cli.Run()
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/cli/cli.go:91 +0x41a
main.main()
        /Users/nsc/Sync/go/src/github.com/debeando/lightflow/main.go:8 +0x20

Calculate execution time by auto increment step.

Inform when condition is not satisfactory

Message about expression and when are different, and put variable names

Like:

when: exit_code(1) == 0 ==> false

Skip pipes block when template is malformed or have error

Remove subtask and leave task

Add parameter on cli to filter with conditions

Like this:

--filters="count > 0"

Use any variables defined in pipes.
This option is used only to print when condition return true?

Show warning message when override variable, show before and after value.

Validate all args on CLI

All variables name should by declared with lowercase and separated with - or _ and not space. Use this regular expression:

^[0-9A-Za-z\-\_]+$

Implement curl plugin

curl:
  method: POST
  url: {{ .job_url }}
  url-data:
  - key1: val1
  - key2: val2
  header:
    - Authorization: Bearer {{ .TB_TOKEN}}

Add skip pipes block on fail

Maybe in this case, the previous pipe generate empty file and decide not continue to the rest of pipes, and them skip the current pipes block and go to the next subtask.

- name: count
  execute: >
  mysql -h {{ .mysql_host }} -u {{ .mysql_user }} -p{{ .mysql_pwd }} thn -ABse "
  SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
  SELECT COUNT(*) AS count
  FROM \`demo\`.\`{{ .mysql_table }}\`
  WHERE {{ .find_field }} BETWEEN '{{ .date }} 00:00:00' AND '{{ .date }} 23:59:59';"
  skip_on_fail: true

Add slack support to send error message

Empty variable into template print "<no value>"

golang/go#24963

package main

import (
	"text/template"
	"log"
	"bytes"
	"fmt"
)

func main() {
	data := make(map[string]string)
	//data["AppVersion"] = "Octane_3.0"
	
	text := "APP_VERSION={{.AppVersion}}"
	tmpl, err := template.New("").Option("missingkey=zero").Parse(text)
	if err != nil {
		log.Fatal(err)
	}
	
	var b bytes.Buffer
	err = tmpl.Execute(&b, data)
	if err != nil {
		fmt.Println("template.Execute failed", err)
	}
	
	fmt.Println("Template text:", text)
	fmt.Println("Expanded:", b.String())
}

Allow many pipes via YAML inside LOOP

Print stdout or stderr when the pipe is exit by error

Get last date automatically in global variables

Imposible to combine --variables with --ai-date

case:

./lightflow \
--config tinybird.yaml 
--task deltatables \
--subtask emails_sent \
--variables '{"dup_cnt": "1"}' \
--ai-date='{"start": "2021-01-23", "end": "2021-02-15"}' \
--pipes export,rename,sign,replace

Implement functions in templates

Example:

http://masterminds.github.io/sprig/strings.html

Allow incluide other .yaml into the task and pipe

Prevent to use many same argument in one command.

For example:

--variables='{"date": "2021-01-22"}'  --variables '{"dup_cnt": "1"}' --dry-run

Have error because de date is empty, and maybe is incorrect to use twice --variables.

Render commands in --dry-run mode

Actually when use this argument, you see <no value>.

Add messages when finished retry loop

attempts exhausted
attempts completed successfully

Convert all levels on JSON into variables

For example:

{
	"var": {
		"foo": 1,
		"bar": 2,
		"level": {
			"test": 3
		}
	}
}

New parse:

var.foo=1
var.bar=2
var.level.test=3

Create cli "class" to have methods to receive and validate all args.

Allow set variable by condition on pipe

---
tasks:
  - name: delta
    subtask:
      - name: d1
    pipes:
      - name: change foo
        set:
          variable: foo
          expression: "..."

Add arg to list task, loops and pipes.

Add auto increment by hour and minutes

Validate greater/less than in --ai-date

For example:

--ai-date='{"start": "2020-01-01", "end": "2019-07-31"}'

Fix when to use set --ai-date to set year, month and day variable from date.

Refactoring and merge the many way to evaluate expression

Retry struct {
	Attempts   int    `yaml:"attempts"`
	Wait       int    `yaml:"wait"`
}

Evaluate []struct {
	Expression string // 
	Level      string // None, Info, Warning, Error
	Action     string // Default: Terminate. Options: Terminate, Skip, When, Logging, Retry, Until
	ExitCode   int    // Default: 1
}

debeando / lightflow Goto Github PK

lightflow's People

Contributors

Stargazers

Watchers

lightflow's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs