GithubHelp home page GithubHelp logo

dotxxljob's Introduction

DotXxlJob

xxl-job的dotnet core 最新执行器实现,支持XXL-JOB 2.2+

注意XXL-JOB 2.0.1版本请使用 1.0.8的执行器实现 ,xxl-job 从 2.0.2 到2.2版本又使用了xxl-rpc的新协议,本执行器不做支持,确实需要的朋友请自行fork..

1 XXL-JOB概述

XXL-JOB是一个轻量级分布式任务调度平台,其核心设计目标是开发迅速、学习简单、轻量级、易扩展。现已开放源代码并接入多家公司线上产品线,开箱即用。以下是它的架构图 架构图

2. 关于DotXxlJob产生

在工作中调研过多个任务调度平台,如Hangfire、基于Quatz.NET的第三方扩展,都与实际的需求有一点差距。 之前一直使用Hangfire,Hangfire的执行器在同步调用业务服务时,如果即时业务服务正在重新部署或者重启,有一定概率会出现死锁,导致CPU100%,后来全部调整为异步,但是这样就无法获得执行结果,这样的设计有蛮大问题,XxlJob的回调机制很好的解决了这个问题。本身如果通过http的方式调用,只要部署springbootd的一个执行器就可以解决问题,但是扩展性较差。所以萌生了实现DotNet版本的执行器的想法,为避免重复造轮子,开始之前也进行过调研,以下仓库https://github.com/yuniansheng/xxl-job-dotnet给了较大的启发,但是该库只支持1.9版本的xxljob,还有一些其他小问题,所以还是自力更生。

3. 如何使用

目前只实现了BEAN的方式,即直接实现IJobHandler调用的方式,Glue源码的方式实际上实现起来也并不复杂(有需求再说把),或者各位有需求Fork 实现一下

可参考sample

安装:

dotnet add package DotXxlJob.Core

3.1 在AspNetCore中使用

  1. 声明一个AspNet的Middleware中间件,并扩展ApplicationBuilder,本质是拦截Post请求,解析Body中的流信息
 public class XxlJobExecutorMiddleware
    {
        private readonly IServiceProvider _provider;
        private readonly RequestDelegate _next;

        private readonly XxlRestfulServiceHandler _rpcService;
        public XxlJobExecutorMiddleware(IServiceProvider provider, RequestDelegate next)
        {
            this._provider = provider;
            this._next = next;
            this._rpcService = _provider.GetRequiredService<XxlRestfulServiceHandler>();
        }


        public async Task Invoke(HttpContext context)
        {
            string contentType = context.Request.ContentType;

            if ("POST".Equals(context.Request.Method, StringComparison.OrdinalIgnoreCase)
                && !string.IsNullOrEmpty(contentType)
                && contentType.ToLower().StartsWith("application/json"))
            {
            
                await _rpcService.HandlerAsync(context.Request,context.Response);              
            
                return;
            }
            
            await _next.Invoke(context);
        }
    }

扩展ApplicationBuilderExtensions,可根据实际情况绑定在特殊的Url Path上

public static class ApplicationBuilderExtensions
{
    public static IApplicationBuilder UseXxlJobExecutor(this IApplicationBuilder @this)
    {
       return @this.UseMiddleware<XxlJobExecutorMiddleware>();
    }
}

在Startup中添加必要的引用,其中自动注册。

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    private IConfiguration Configuration { get; }
    
    public void ConfigureServices(IServiceCollection services)
    {
      
        services.AddXxlJobExecutor(Configuration);
        services.AddSingleton<IJobHandler, DemoJobHandler>(); // 添加自定义的jobHandler
        services.AddAutoRegistry(); // 自动注册
    }


    public void Configure(IApplicationBuilder app,IHostingEnvironment env)
    {
        //启用XxlExecutor
        app.UseXxlJobExecutor();
    }
}

编写JobHandler,继承AbstractJobHandler或者直接实现接口IJobHandler,通过context.JobLogger 记录执行过程和结果,在AdminWeb上可查看的哦

[JobHandler("demoJobHandler")]
public class DemoJobHandler:AbstractJobHandler
{
    public override Task<ReturnT> Execute(JobExecuteContext context)
    {
        context.JobLogger.Log("receive demo job handler,parameter:{0}",context.JobParameter);

        return Task.FromResult(ReturnT.SUCCESS);
    }
}

3.2 配置信息

管理端地址和端口是必填信息,其他根据实际情况,选择配置,配置项说明见下代码中的注释

 public class XxlJobExecutorOptions
{
   
    /// <summary>
    /// 管理端地址,多个以;分隔
    /// </summary>
    public string AdminAddresses { get; set; }
    /// <summary>
    /// appName自动注册时要去管理端配置一致
    /// </summary>
    public string AppName { get; set; } = "xxl-job-executor-dotnet";
    /// <summary>
    /// 自动注册时提交的地址,为空会自动获取内网地址
    /// </summary>
    public string SpecialBindAddress { get; set; }
    /// <summary>
    /// 绑定端口
    /// </summary>
    public int Port { get; set; }
    /// <summary>
    /// 是否自动注册
    /// </summary>
    public bool AutoRegistry { get; set; }
    /// <summary>
    /// 认证票据
    /// </summary>
    public string AccessToken { get; set; }
    /// <summary>
    /// 日志目录,默认为执行目录的logs子目录下,请配置绝对路径
    /// </summary>
    public string LogPath { get; set; } = Path.Combine(AppContext.BaseDirectory, "./logs");
    /// <summary>
    /// 日志保留天数
    /// </summary>
    public int LogRetentionDays { get; set; } = 30;
}

其他说明

注意XXL-JOB 2.0.1版本请使用 1.0.8的执行器实现

有任何问题,可Issue反馈 ,最后感谢 xxl-job

dotxxljob's People

Contributors

fanhousanbu avatar fatuitycookie avatar guochen2 avatar sniperking1234 avatar xuanye avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

dotxxljob's Issues

xxljob admin 2.2.0 执行器中文日志乱码问题

XxlRestfulServiceHandler
HandlerAsync

await response.WriteAsync(JsonConvert.SerializeObject(ret));
应为 await response.WriteAsync(JsonConvert.SerializeObject(ret, new JsonSerializerSettings() { StringEscapeHandling = StringEscapeHandling.EscapeNonAscii }));

地址检测问题

我的内网地址 192.122.xxx.xxx
您设置的校验规则
ipABegin = ConvertToNumber("10.0.0.0");
ipAEnd = ConvertToNumber("10.255.255.255");

ipBBegin = ConvertToNumber("172.16.0.0");
ipBEnd = ConvertToNumber("172.31.255.255");

ipCBegin = ConvertToNumber("192.168.0.0");
ipCEnd = ConvertToNumber("192.168.255.255");

_options.SpecialBindAddress = IPUtility.GetLocalIntranetIP().MapToIPv4().ToString();
导致这里抛异常

执行器执行任务完成后,调度器无法修改执行结果

xxl-job2.3.0版本中,callback api接口的数据格式有变化,所以2边的数据格式不一样,导致无法修改执行结果

xxl-job中
public HandleCallbackParam(long logId, long logDateTim, int handleCode, String handleMsg) { this.logId = logId; this.logDateTim = logDateTim; this.handleCode = handleCode; this.handleMsg = handleMsg; }

DotXxlJob中
public HandleCallbackParam(TriggerParam triggerParam, ReturnT result) { this.LogId = triggerParam.LogId; this.LogDateTime = triggerParam.LogDateTime; this.ExecuteResult = result; }

阻塞处理策略设置为“丢弃后续调度“,只有第一次调度正常执行,后面的所有调度都被丢弃了。

场景:调度频率1分钟/次,job执行所需时长超过3分钟,阻塞处理策略设置为“丢弃后续调度“
期望效果:第4/5次调度时job可以正常执行
实际效果:第一次调度正常执行,后续所有调度都被丢弃

我修改了这段代码后可以达到预期效果 [HandleCallbackParam添加了属性JobId]
private void TriggerCallback(object sender, HandleCallbackParam callbackParam)
{
if(RUNNING_QUEUE.TryRemove(callbackParam.JobId, out var _))
this._callbackTaskQueue.Push(callbackParam);
}
我第一次用xxl-job,不知道这个修改是否有其他潜在的问题。

任务取消,未通知到实际执行器,导致真实任务并没有停止

现象:如题,当调用停止任务,对CancellationToken进行取消时,实际执行任务里并不知道已经停止了,线程还是会继续执行
期望:将CancellationToken作为参数传入到Executor,实际执行任务里能对CancellationToken进行IsCancellationRequested判断,终止任务。

image

LogRequest的LogId类型int是否可以改成long

this._jobLogger.ReadLog读取的logid是long,但是LogRequest定义的是int,有些情况会导致强制转换
IJobLogger源代码中 LogResult ReadLog(long logTime, long logId, int fromLine)读取的也是long

XxlJobAdmin有改版会传入雪花ID

CollectBody解析出错

解析Body时报错,能不能帮看下什么问题
.netCore我改成了.net5.0 xxl-job是2.2版本
因为.net版本升级不支持同步读取,所以添加了支持同步读取的配置
image
image

QQ截图20210108160615

单机串行无效

xxljob version 2.3.1

[JobHandler("demoJobHandler")]
public class DemoJobHandler : AbstractJobHandler
{
    public override async Task<ReturnT> Execute(JobExecuteContext context)
    {
        context.JobLogger.Log("receive demo job handler,parameter:{0}", context.JobParameter);
        context.JobLogger.Log("开始休眠10秒");
        await Task.Delay(10 * 1000);
        context.JobLogger.Log("休眠10秒结束");
        return ReturnT.SUCCESS;
    }
}

理论这个任务要跑10s才算结束
xxl-job面板里按5s一次启动这个任务 单机串行 , 按理说时机效果是10s执行一次, 实际看日志根本不管上个任务有没有执行完毕,就是5s投递一次任务 , 这样是不是错了?

执行管理器可以注册上去,执行任务xxljob-admin出现500 Connection refused

java xxl-job-admin 错误信息

c.x.job.core.util.XxlJobRemotingUtil - Connection refused: connect
java.net.ConnectException: Connection refused: connect
	at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
	at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at sun.net.www.http.HttpClient.New(HttpClient.java:339)
	at sun.net.www.http.HttpClient.New(HttpClient.java:357)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
	at com.xxl.job.core.util.XxlJobRemotingUtil.postBody(XxlJobRemotingUtil.java:95)
	at com.xxl.job.core.biz.client.ExecutorBizClient.run(ExecutorBizClient.java:43)
	at com.xxl.job.admin.core.trigger.XxlJobTrigger.runExecutor(XxlJobTrigger.java:211)
	at com.xxl.job.admin.core.trigger.XxlJobTrigger.processTrigger(XxlJobTrigger.java:164)
	at com.xxl.job.admin.core.trigger.XxlJobTrigger.trigger(XxlJobTrigger.java:89)
	at com.xxl.job.admin.core.thread.JobTriggerPoolHelper$3.run(JobTriggerPoolHelper.java:95)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
10:29:54.179 logback [xxl-job, admin JobTriggerPoolHelper-fastTriggerPool-329495173] ERROR c.x.job.core.util.XxlJobRemotingUtil - Connection refused: connect

执行日志

任务触发类型:手动触发
调度机器:192.168.0.221
执行器-注册方式:自动注册
执行器-地址列表:[http://192.168.0.221:5000/]
路由策略:第一个
阻塞处理策略:单机串行
任务超时时间:0
失败重试次数:0

>>>>>>>>>>>触发调度<<<<<<<<<<<
触发调度:
address:http://192.168.0.221:5000/
code:500
msg:xxl-rpc remoting error(Connection refused: connect), for url : http://192.168.0.221:5000/run

是.netcore端配置有什么问题么?

你写的那个中间件有没有考虑过一个问题?

public async Task Invoke(HttpContext context)
{
string contentType = context.Request.ContentType;

        if ("POST".Equals(context.Request.Method, StringComparison.OrdinalIgnoreCase)
            && !string.IsNullOrEmpty(contentType)
            && contentType.ToLower().StartsWith("application/json"))
        {

            await _rpcService.HandlerAsync(context.Request,context.Response);              

            return;
        }
        
        await _next.Invoke(context);
    }

我觉得你这么写会导致一些正常的一些请求也会去执行await _rpcService.HandlerAsync(context.Request,context.Response); 了。这肯定不行的吧。

关于执行任务的通信协议

请问下注册任务后,xxl-job是如何调用业务逻辑的。是需要部署为API,然后通过http协议进行调用吗。

XxlRestfulServiceHandler兼容不是xxl-job的请求

  1. 现况
  • XxlRestfulServiceHandler现在处理请求,无论是否xxl-job的请求都会短路请求,直接响应。
    image
  1. 期望
  • XxlRestfulServiceHandler根据XxlJobExecutorOptions的服务器地址判断是否是xxl-job的请求再作处理。
  • 如果不是xxl-job的请求也不要做短路处理,交由下一中间件处理请求。
  1. 原因
  • xxl-job当前虽然请求较为固定,但是没有一个强制约束的请求头标记或者路由约束。(token校验不是强制的。其他的请求容易和其他路由冲突)
  • XxlRestfulServiceHandler已经对路由做出判断,应用程序不应该在做额外一次判断。
  • XxlRestfulServiceHandler对请求的处理应该兼容不是xxl-job的请求,而不是无论结果与否,直接短路请求

注册进去无法触发

image
在服务器上ping程序所在的机子是通的,xia下面是job配置
image
程序运行图如下:
image
job类代码如下:
image

image

image

麻烦帮忙看下是什么问题?

xxljob admin 2.2.0 执行器中文日志乱码问题

XxlRestfulServiceHandler
HandlerAsync

await response.WriteAsync(JsonConvert.SerializeObject(ret));
应为 await response.WriteAsync(JsonConvert.SerializeObject(ret, new JsonSerializerSettings() { StringEscapeHandling = StringEscapeHandling.EscapeNonAscii }));

不支持https

public async Task RegistryAsync(CancellationToken cancellationToken) { var registryParam = new RegistryParam { RegistryGroup = "EXECUTOR", RegistryKey = _options.AppName, RegistryValue = string.IsNullOrEmpty(_options.SpecialBindUrl)? $"http://{_options.SpecialBindAddress}:{_options.Port}/" : _options.SpecialBindUrl };
这里固定写死了http

解析stream失败

TriggerParam类中的LogId在xxl-job中未long类型,而在dotXxlJob中是int类所以在反序列化时会出现问题

对新版的xxl-job,注册不上去,报错

fail: DotXxlJob.Core.AdminClient[0]
request admin error.Response status code does not indicate success: 404 ().
System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 ().
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at DotXxlJob.Core.AdminClient.DoPost(HttpClient client, AddressEntry address, Byte[] postBuf) in D:\work\code\cshrap\github\DotXxlJob-master\src\DotXxlJob.Core\AdminClient.cs:line 172
at DotXxlJob.Core.AdminClient.InvokeRpcService(String methodName, List`1 parameterTypes, Object parameters, Boolean polling) in D:\work\code\cshrap\github\DotXxlJob-master\src\DotXxlJob.Core\AdminClient.cs:line 107

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.