GithubHelp home page GithubHelp logo

xxl-job / xxl-job-executor-go Goto Github PK

View Code? Open in Web Editor NEW
409.0 9.0 103.0 27 KB

xxl-job 执行器(golang 客户端)

License: MIT License

Go 100.00%
xxl-job xxljob xxl-job-client xxl-job-executor crontab cronjob go-cron

xxl-job-executor-go's Introduction

xxl-job-executor-go

很多公司java与go开发共存,java中有xxl-job做为任务调度引擎,为此也出现了go执行器(客户端),使用起来比较简单:

支持

1.执行器注册
2.耗时任务取消
3.任务注册,像写http.Handler一样方便
4.任务panic处理
5.阻塞策略处理
6.任务完成支持返回执行备注
7.任务超时取消 (单位:秒,0为不限制)
8.失败重试次数(在参数param中,目前由任务自行处理)
9.可自定义日志
10.自定义日志查看handler
11.支持外部路由(可与gin集成)
12.支持自定义中间件

Example

package main

import (
	"context"
	"fmt"
	xxl "github.com/xxl-job/xxl-job-executor-go"
	"github.com/xxl-job/xxl-job-executor-go/example/task"
	"log"
)

func main() {
	exec := xxl.NewExecutor(
		xxl.ServerAddr("http://127.0.0.1/xxl-job-admin"),
		xxl.AccessToken(""),            //请求令牌(默认为空)
		xxl.ExecutorIp("127.0.0.1"),    //可自动获取
		xxl.ExecutorPort("9999"),       //默认9999(非必填)
		xxl.RegistryKey("golang-jobs"), //执行器名称
		xxl.SetLogger(&logger{}),       //自定义日志
	)
	exec.Init()
	exec.Use(customMiddleware)
	//设置日志查看handler
	exec.LogHandler(customLogHandle)
	//注册任务handler
	exec.RegTask("task.test", task.Test)
	exec.RegTask("task.test2", task.Test2)
	exec.RegTask("task.panic", task.Panic)
	log.Fatal(exec.Run())
}

// 自定义日志处理器
func customLogHandle(req *xxl.LogReq) *xxl.LogRes {
	return &xxl.LogRes{Code: xxl.SuccessCode, Msg: "", Content: xxl.LogResContent{
		FromLineNum: req.FromLineNum,
		ToLineNum:   2,
		LogContent:  "这个是自定义日志handler",
		IsEnd:       true,
	}}
}

// xxl.Logger接口实现
type logger struct{}

func (l *logger) Info(format string, a ...interface{}) {
	fmt.Println(fmt.Sprintf("自定义日志 - "+format, a...))
}

func (l *logger) Error(format string, a ...interface{}) {
	log.Println(fmt.Sprintf("自定义日志 - "+format, a...))
}

// 自定义中间件
func customMiddleware(tf xxl.TaskFunc) xxl.TaskFunc {
	return func(cxt context.Context, param *xxl.RunReq) string {
		log.Println("I am a middleware start")
		res := tf(cxt, param)
		log.Println("I am a middleware end")
		return res
	}
}

示例项目

github.com/xxl-job/xxl-job-executor-go/example/

与gin框架集成

https://github.com/gin-middleware/xxl-job-executor

xxl-job-admin配置

添加执行器

执行器管理->新增执行器,执行器列表如下:

AppName		名称		注册方式	OnLine 		机器地址 		操作
golang-jobs	golang执行器	自动注册 		查看 ( 1 )   

查看->注册节点

http://127.0.0.1:9999

添加任务

任务管理->新增(注意,使用BEAN模式,JobHandler与RegTask名称一致)

1	测试panic	BEAN:task.panic	* 0 * * * ?	admin	STOP	
2	测试耗时任务	BEAN:task.test2	* * * * * ?	admin	STOP	
3	测试golang	BEAN:task.test		* * * * * ?	admin	STOP

xxl-job-executor-go's People

Contributors

rushuinet avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

xxl-job-executor-go's Issues

注册执行器不需要轮询调用,HTTP1.1默认支持keepalive

  • 原代码运行抓包

40ddf125757817761095bd53d57ac2a


>从上面可以看出本地端口都使用不同链接,通过ACK确认TCP报文可以看出(58421,58425,58441)
  • 去掉本地注册轮询之后抓包

6aeb8ef0af362744ded6bea7dc70ec4

很明显进行第一次尝试keepalive TCPack 之后 服务端会主动发送FIN SYN 包主动断开连接,

总结,虽然不确定为什么服务端会主动发送FIN 包 但是可以确定的是,轮询防止心跳过期代码并没有起到什么作用

代码如下:

func (e *executor) registry() {

	t := time.NewTimer(time.Second * 0) //初始立即执行
	defer t.Stop()
	req := &Registry{
		RegistryGroup: "EXECUTOR",
		RegistryKey:   e.opts.RegistryKey,
		RegistryValue: "http://" + e.address,
	}
	param, err := json.Marshal(req)
	if err != nil {
		log.Fatal("执行器注册信息解析失败:" + err.Error())
	}
	for {
		<-t.C
		t.Reset(time.Second * time.Duration(20)) //20秒心跳防止过期
		func() {
			result, err := e.post("/api/registry", string(param))
			if err != nil {
				e.log.Error("执行器注册失败1:" + err.Error())
				return
			}
			defer result.Body.Close()
			body, err := ioutil.ReadAll(result.Body)
			if err != nil {
				e.log.Error("执行器注册失败2:" + err.Error())
				return
			}
			res := &res{}
			_ = json.Unmarshal(body, &res)
			if res.Code != SuccessCode {
				e.log.Error("执行器注册失败3:" + string(body))
				return
			}
			e.log.Info("执行器注册成功:" + string(body))
		}()

	}
}

心跳检测方法

e.beate.idleBeat 方法可否像 e.runTask 对应的 RunTask 一样给开放出来

xxl-job 2.3.0 使用异常

服务端接收参数:
image

客户端发送参数:
image

结果:
xxl-job-admin 界面端无法查看到执行结果
image

建议:
image

执行器执行成功,但并未注册到调度器上

exec := xxl.NewExecutor(
		xxl.ServerAddr("http://localhost:18001/xxl-job-admin"),
		xxl.AccessToken("default_token"), //请求令牌(默认为空)
		xxl.ExecutorPort("9999"),         //默认9999(非必填)
		xxl.RegistryKey("golang-jobs"),   //执行器名称
	)
	exec.Init()
	//注册任务handler
	exec.RegTask("task.test", task.Test)
	exec.RegTask("task.test2", task.Test2)
	exec.RegTask("task.panic", task.Panic)
	fmt.Println(exec.Run())

terminal上输出:

Starting server at 169.254.51.207:9999
执行器注册成功:{"code":200,"msg":null,"content":null}
执行器注册成功:{"code":200,"msg":null,"content":null}
执行器注册成功:{"code":200,"msg":null,"content":null}

但是web ui上的执行器上并没有,所以这个执行器注册成功了也没用?还是和调度器版本有关系,那么请告诉哪个版本是有用的。
image

调用XXLJobExecutor.Stop()以后任务还是可以执行的,原因不明

如下面的代码所示,我想让它执行一次以后就结束,因为这本身就是在一个简单的测试函数里。

结果执行了以后呢,确实是能够看到日志 stopped,预期是在stopped以后就啥也不做,退出,但实际结果不符合这样的预期。

但是代码里的task还是会持续进行。

var wg sync.WaitGroup
	var once sync.Once
	wg.Add(1)
	XXLJobExecutor.RegTask("task.test.demo", func(cxt context.Context, param *xxl.RunReq) string {
		defer func() {
			once.Do(func() {
				wg.Done() //只能done一次
			})
		}()
		t.Log("OK")
		return "OK"
	})

	t.Log("step")

	go func() {
		require.NoError(t, XXLJobExecutor.Run())
		t.Log("can not print this log because the stop is wrong")
	}()

	wg.Wait()

	time.Sleep(time.Second)

	t.Log("stopping")
	XXLJobExecutor.Stop()
	t.Log("stopped")

	time.Sleep(time.Second)

log is:

2024/04/25 11:54:15 INFO 执行器注册成功:{"code":200,"msg":null,"content":null}
2024/04/25 11:54:32 INFO 任务参数:&{2 task.test.demo  SERIAL_EXECUTION 0 24 1714020872593 BEAN  1714016646000 0 1}
2024/04/25 11:54:32 INFO 任务[2]开始执行:task.test.demo
    xxl_job_test.go:34: OK
2024/04/25 11:54:32 INFO 任务回调成功:{"code":200,"msg":null,"content":null}
    xxl_job_test.go:48: stopping
2024/04/25 11:54:33 INFO 执行器摘除成功:{"code":200,"msg":null,"content":null}
    xxl_job_test.go:50: stopped
2024/04/25 11:54:35 INFO 执行器注册成功:{"code":200,"msg":null,"content":null}
2024/04/25 11:54:46 INFO 任务参数:&{2 task.test.demo  SERIAL_EXECUTION 0 25 1714020886165 BEAN  1714016646000 0 1}
2024/04/25 11:54:46 INFO 任务[2]开始执行:task.test.demo
    xxl_job_test.go:34: OK
2024/04/25 11:54:46 INFO 任务回调成功:{"code":200,"msg":null,"content":null}
2024/04/25 11:54:54 INFO 任务参数:&{2 task.test.demo  SERIAL_EXECUTION 0 26 1714020894733 BEAN  1714016646000 0 1}
2024/04/25 11:54:54 INFO 任务[2]开始执行:task.test.demo
    xxl_job_test.go:34: OK
2024/04/25 11:54:54 INFO 任务回调成功:{"code":200,"msg":null,"content":null}

v0.6.1 报panic 错误

2021/08/20 06:49:30 callback err : %!(EXTRA string=Post http://xxl-job/xxl-job-admin/api/callback: EOF)
goroutine 467204 [running]:
runtime/debug.Stack(0x1, 0x1, 0x83)
/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
/usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/xxl-job/xxl-job-executor-go.(*Task).Run.func1(0xc00058f800, 0xc00046d4e0, 0xc0003cc1c0)
/go/pkg/mod/github.com/xxl-job/[email protected]/task.go:31 +0x20f
panic(0xecec80, 0x1978140)
/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/xxl-job/xxl-job-executor-go.(*executor).callback(0xc0002552b0, 0xc00058f800, 0xc8, 0x0, 0x0)
/go/pkg/mod/github.com/xxl-job/[email protected]/executor.go:312 +0xda
github.com/xxl-job/xxl-job-executor-go.(*executor).runTask.func1(0xc8, 0x0, 0x0)
/go/pkg/mod/github.com/xxl-job/[email protected]/executor.go:166 +0x51
github.com/xxl-job/xxl-job-executor-go.(*Task).Run(0xc00058f800, 0xc00046d4e0)
/go/pkg/mod/github.com/xxl-job/[email protected]/task.go:37 +0xbe
created by github.com/xxl-job/xxl-job-executor-go.(*executor).runTask
/go/pkg/mod/github.com/xxl-job/[email protected]/executor.go:165 +0x699

不应该使用注册列表中的task作为运行列表的task

问题描述:
当前实现机制,有一个新的任务请求过来,根据ExecutorHandler从注册列表获取task的指针,然后填充task的运行时参数,接着把task放入运行列表,并其一个协程运行task。存在的问题是,如果有两个或者多个相同的task,它们都是相同的ExecutorHandler(可以理解为task类型),在运行列表存的都是同一个task对象,那么task记录的是最后一个请求的参数。这样导致多任务运行结果错误。比如先运行同ExecutorHandler的A任务,然后运行同ExecutorHandler的B任务,现在要取消A任务,结果把B任务结束了。

解决办法:
从注册task中获取task后,复制一份task,作为运行的task对象。

yoyogo框架已集成xxl-job-executor-go项目,讨论日志功能能否集成到xxl-job-executor-go项目中?

案例:

https://github.com/yoyofx/yoyogo/tree/dev/examples/xxl-job-console
已支持xxl-job日志(实时获取)

func main() {
	// -f ./conf/test_conf.yml 指定配置文件 , 默认读取 config_{profile}.yml , -profile [dev,test,prod]
	config := configuration.YAML("config")
	scheduler.NewXxlJobBuilder(config).
		ConfigureServices(func(collection *dependencyinjection.ServiceCollection) {
			scheduler.AddJobs(collection, NewDemoJob)
		}).
		Build().Run()
}

type DemoJob struct {}

func NewDemoJob() *DemoJob {
	return &DemoJob{}
}

func (*DemoJob) Execute(cxt *scheduler.JobContext) (msg string) {
	cxt.Report("Job %d is beginning...", cxt.LogID)

	for i := 1; i <= 100; i++ {
		cxt.Report("Job Progress: %d Percent.", i)
		time.Sleep(time.Second)
	}

	return cxt.Done("666")
}

//GetJobName 自定义任务的名字
func (*DemoJob) GetJobName() string {
	return "job1"
}

在任务中使用dao

JobHandler 里面无法使用外面的service或者dao,会出现import cycle not allowed的问题

可以 admin 推送任务到 executor 吗

现在的执行器设计成是一个 server,admin 通过 curl executor 的接口,可以修改成 executor 仅仅作为一个客户端,由 admin 作为 server 推送任务到 executor 吗,websocket 或者 grpc 都可以

关于日志查看handler

想请教下,自定义日志查看handler,这个任务日志需要自己存储,然后返回吗。
想问下有具体的task例子吗,不走默认的LogHandler

The access token is wrong.

目前最新版使用的token header 为 XXL-JOB-ACCESS-TOKEN
但在本项目中,使用的是:
request.Header.Set("XXL-RPC-ACCESS-TOKEN", e.opts.AccessToken)

执行器完毕后回调时候发生网络中断,这个任务可能一直会卡住

res, err := e.post("/api/callback", string(returnCall(task.Param, code, msg)))

如上,如果在执行完毕回调给注册器通知的时候,发生了网络之类的中断,这次请求没有成功,则后续再执行会报

msg:xxl-rpc remoting (url=http://10.10.57.249:9999/run) response content invalid([{"logId":7515,"logDateTim":1632794520006,"executeResult":{"code":500,"msg":"There are tasks running"},"handleCode":500,"handleMsg":"There are tasks running"}]).

请问是否这里针对执行器可以有重试的方案或者其他优化吗?

Executor 部署在Docker中时可能不能正常工作

问题: 当Executor 使用docker部署的时候, Executor的访问需要通过宿主机进行映射,当前配置中ExecutorIP:ExecutorPort 同时用于run服务以及向xxl-job-admin注册,会导致可以注册成功,但是任务回调会失败。
如如下错误:
image

解决办法:将run服务以及注册任务回调的ip:port 区分配置。

com.xxl.rpc.util.XxlRpcException: xxl-rpc, request timeout at

执行器部署完了,浏览器中访问http://****:9977/run,返回
[{"logId":0,"logDateTim":0,"executeResult":{"code":500,"msg":"params err"},"handleCode":500,"handleMsg":"params err"}]
说明执行器部署没问题,网络也没问题。
然后admin中执行任务,报错:

任务触发类型:手动触发
调度机器:admin IP地址
执行器-注册方式:手动录入
执行器-地址列表:[****:9977]
路由策略:第一个
阻塞处理策略:单机串行
任务超时时间:0
失败重试次数:0

触发调度<<<<<<<<<<<
触发调度:
address:****:9977
code:500
msg:com.xxl.rpc.util.XxlRpcException: xxl-rpc, request timeout at:1662349128294, request:XxlRpcRequest{requestId='19136019-dee8-4611-bd7b-d8d501c67e1a', createMillisTime=1662349125222, accessToken='', className='com.xxl.job.core.biz.ExecutorBiz', methodName='run', parameterTypes=[class com.xxl.job.core.biz.model.TriggerParam], parameters=[TriggerParam{jobId=1, executorHandler='testJob', executorParams='', executorBlockStrategy='SERIAL_EXECUTION', executorTimeout=0, logId=35657, logDateTime=1662349125077, glueType='BEAN', glueSource='', glueUpdatetime=1541254891000, broadcastIndex=0, broadcastTotal=1}], version='null'} at com.xxl.rpc.remoting.net.params.XxlRpcFutureResponse.get(XxlRpcFutureResponse.java:117) at com.xxl.rpc.remoting.invoker.reference.XxlRpcReferenceBean$1.invoke(XxlRpcReferenceBean.java:242) at com.sun.proxy.$Proxy93.run(Unknown Source) at com.xxl.job.admin.core.trigger.XxlJobTrigger.runExecutor(XxlJobTrigger.java:196) at com.xxl.job.admin.core.trigger.XxlJobTrigger.processTrigger(XxlJobTrigger.java:149) at com.xxl.job.admin.core.trigger.XxlJobTrigger.trigger(XxlJobTrigger.java:74) at com.xxl.job.admin.core.thread.JobTriggerPoolHelper$3.run(JobTriggerPoolHelper.java:90) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

有个类似的问题:#29
不知道那个具体怎么配置,我的执行器也是部署在docker中。

回调时传参错误

目前传参为 [{"logId":38,"logDateTim":1619504225663,"executeResult":{"code":200,"msg":"done"}}]

而xxl-job中接受参数的结构体是
private long logId;
private long logDateTim;
private int handleCode;
private String handleMsg;

[{"logId":38,"logDateTim":1619504225663,"handleCode":200,"handleMsg":"done"}]

Executor注册时不能使用公网IP

大佬,Executor 端部署时注册IP只能填写本机内网IP,如果填入公网IP无法被admin调用,端口也没有起来。 admin 和 executor 不在一台服务器上。也不在同一个内网网段

注册XXL-JOBS 2.0失败

执行器注册失败3:

<link rel="icon" href="/xxl-job-admin/static/favicon.ico" />

<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<!-- Tell the browser to be responsive to screen width -->
<meta content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no" name="viewport">
<!-- Bootstrap -->
<link rel="stylesheet" href="/xxl-job-admin/static/adminlte/bower_components/bootstrap/css/bootstrap.min.css">
<!-- Font Awesome -->
<link rel="stylesheet" href="/xxl-job-admin/static/adminlte/bower_components/font-awesome/css/font-awesome.min.css">
<!-- Ionicons -->
<link rel="stylesheet" href="/xxl-job-admin/static/adminlte/bower_components/Ionicons/css/ionicons.min.css">
<!-- Theme style -->
<link rel="stylesheet" href="/xxl-job-admin/static/adminlte/dist/css/AdminLTE.min.css">
<!-- AdminLTE Skins. Choose a skin from the css/skins folder instead of downloading all of them to reduce the load. -->
<link rel="stylesheet" href="/xxl-job-admin/static/adminlte/dist/css/skins/_all-skins.min.css">

<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->

<!-- pace -->
<link rel="stylesheet" href="/xxl-job-admin/static/adminlte/bower_components/PACE/themes/blue/pace-theme-flash.css">

进程退出时,Task感知不到

是否可以将Run()方法增加一个context参数,后续的Task执行时都继承这个context呢?
或者其他方案,让Task执行体能够感知到整体进程要退出了

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.