xiaomi / thain

Thain is a distributed flow schedule platform.

Thain

Other language versions

简体中文

Introduction

Thain is a distributed flow schedule platform, it was invented at XiaoMi Technology and the project has been proven easily-learning, developed-rapidly, stable in the XiaoMi internal departments. Thain has completed backend management system which supports depending, re-run, roll-back and monitoring of the flows, contains bautiful and easily-manipulating web UI for users to maintain your jobs.

Quick Start

1. Environmental Requirements

   • jdk >= 8
   • nodejs >= 8

2. Clone Code

   git clone https://github.com/XiaoMi/thain.git

3. Execute in order under the project root directory

   cd thain-fe
   npm install
   npm run build
   cd ..
   ./gradlew clean build

4. Run

   java -jar thain-server/build/libs/thain-server-1.3.0-beta.0.jar

5. Open localhost:9900 to see the effect

6. Initial account password: admin, admin

Contributors

• liangyongrui
• godisjohnny
• zuojianguang
• poplp
• SingleMeen

Help Us Do Better

Now Thain continues to be developed by xiaomi internal developers. If you have any questions, you can ask in the issues or send email to:

• [email protected]

License

Apache License Version 2.0 © Xiaomi

thain's Issues

Job time is negative

请问大佬们，支持延时Job吗

有些场景定时任务不能满足或者说不太优雅，所以需要延时任务的功能

Dashboard add waiting

Compatible with h2, sql return value needs to be case sensitive.

Administrator configuration module

The administrator configuration module provides user management and APP management.

Optimize flow list

flow editor Radio default is wrong

发现一行代码错误

FlowServiceImpl.java 67行
addFlowRq.toBuilder().slaKill(true).slaDuration(3L * 60 * 60).build();

Optimize the thread pool

Restore accidentally deleted flow

com.fasterxml.jackson.core jackson-databind security issue

        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.9.8</version>

version should be 2.9.10 or later

Troubleshoot memory leaks

use antvis/x6

Waiting for it to be stable

add docker image

flow execution carries additional information

1. 前端立即执行提供一个下拉箭头，填写执行变量
2. x5添加参数
3. flow execution 表增加执行变量字段
4. 执行的时候加到FlowExecutionStorage中
5. 配合shell
   • g_xxx (global, 表示全局一直有的变量） #107
6. 适配重试机制
7. flow execution 页上展示执行变量内容

Flow batch kill

Time zone setting for Thain

Add time zone settings on the page

Global variables

1. 全局变量视为一个 job name 为g的任务产生的结果
2. Built-in Global dynamic variables: like date, time ...
   • g_xxx
3. Administrators can customize global variables in the management page
4. flow 执行时候的临时变量，也存在全局中，并且覆盖之前指定的全局变量 #80

The SDK adds the ability to modify the job properties.

enhance http component

1. http 节点失败策略

   • http支持正则表达式
   • 不匹配视为失败

2. 配合 #15 进行重试

3. http 节点 x5发送

   • 新的 http x5 component
   • 使用客户端提供的x5配置进行加密

Json注入

https://github.com/XiaoMi/Thain/blob/b610792d35127704747c017ca41ba7d98d0f150c/thain-server/src/main/java/com/xiaomi/thain/server/controller/FlowController.java#L103-L106
应用程序所解析的JSON文档或请求来源于不可信赖的数据源，程序没有对这些不可信赖的数据进行验证、过滤，攻击者可以插入一些元素做一些可以预测的操作，比如解析JSON失败，导致抛出异常，甚至可以导致XSS和动态解析代码。

The component json definition contains whether the front-end display

machine monitoring

1. 探测失效的机器，用于恢复数据
   • 现在的恢复机制在每日任务超过30w后可能会有一些异常
   • 减少对quartz的依赖

mybatis configuration.variables cannot be read directly

mybatis 全局参数读不到

com.xiaomi.thain.core.config.DatabaseHandler

目前手动传递临时解决一下

Thain return error detail

The message in the callback interface of the scheduling task I execute should be the details of the error, not the job [sql_name] exception

Flow operation log

Flow operation log
flow 操作日志

Extend and unify "flow last_run_status", "flowExecution status", "flow callback status"

retry mechanism

I suggest developing a retry mechanism

Initialize the demo script

x5config bug flx

x5 config bug waht when access to /amdin/client data is not existed

Java SDK add the "flow execution id based on flow id" function.

Add queue status

1. add queue status
2. The flow in the queue can be executed with one click on the page.
3. Perform flows manually without queuing.

The processing strategy when running overlaps.

Let the user choose:

     1. Kill the previous one
     2. Give up the current
     3. Parallel

Generate flow from json

先做一个json版本的

Support multiple scheduling modes

1. Loop execution

   • cron

   • execution interval

2. Once execution

   • specific time and date

   • delay time

3. 多个调度计划组合

Flow Retry

1. 用户可以指定失败重试次数
2. 用户可以指定失败重试的时间间隔
3. 重试的触发条件只有执行失败
4. (等待)重试时
   • flow last run status 为 running
     • 所以不能重叠执行
   • flow execution status 为 失败等待下一次重试
5. 重试的flow execution trigger type为retry
6. flow http callback的“结束callback” 只有最后一次重试才发送
7. flow http callback的“开始callback” 只有第一次执行发送

发现一行代码错误

FlowServiceImpl.java 67行
addFlowRq.toBuilder().slaKill(true).slaDuration(3L * 60 * 60).build();

Optimize models/entities

Enhance shell component

Shell 是重要且特殊的组件

1. 用户可以直接在网页上输入shell脚本
2. 每个用户有一个地方管理自己上传的shell脚本，方便多个脚本复用（这个需求待确认）
3. shell 可以设置变量给下一个shell（不太好做）
4. shell 变量命名
   • 当前的命名方式可能有兼容性问题
   • jobName_xxx

Flow supports nested execution

Append the trigger plan to the specified flow

Optimize the admin page

• tips

升级antd 4

1. 重构前端login组件，和antd pro 保持一致

Display the flow query condition on the url

This allows the browser to "back" operation

After editing a flow, the scheduling status of the flow should not change.

job name at least two characters

Pause automatic trigger flow

In order to facilitate the service offline, thain needs to be able to pause the automatic trigger flow.

Sla timeout kill state

Job Retry

The log details page needs to be watermarked