GithubHelp home page GithubHelp logo

blqw.dataflow's Introduction

dataflow

★ 简介

提供一种方便操作批量数据的框架
实现需要自己写

★ 主要操作

// 创建工作参数
WorkSettings<JSONObject, JSONObject> settings = new WorkSettings<>();
// 设置参数
settings.setReaders(...);    // 设置读取数据的组件
settings.setProcesses(...);  // 设置处理数据的组件
settings.setKeyFetcher(...); // ID提取组件 (可选, 用于任务异常后重启的幂等判断)
settings.setCache(...);      // 缓存组件 (可选, 用于任务异常后重启的幂等判断)
settings.setSnapshoter(...); // 快照组件 (可选, 用于任务异常后重启的游标设置)
settings.setProperty(name, value);   // 设置拓展属性 (可选)

// 创建工作中心
SingleBlockingWorkCenter<JSONObject> workCenter = new SingleBlockingWorkCenter<>(settings);
// 开始工作(读取线程数, 处理线程数)
workCenter.startAsync(4, 10);
// 等待工作完成
workCenter.await();

★ 组件介绍

1. IDataReader<T>(读取数据组件)

>源码<

从任意位置读取指定长度的数据和游标，并可根据游标继续读取下一批数据

@FunctionalInterface
public interface IDataReader<T> extends IDataOutput<T> {
}

@FunctionalInterface
public interface IDataOutput<T> {
    IBatchData<T> get(String cursor, Integer size);
}

public interface IBatchData<T> {
    boolean isEnd();
    String cursor();
    List<T> data();
}

2. IDataProcess<T>(数据处理组件)

>源码<

用于处理单个数据, 如果处理失败直接抛出异常

@FunctionalInterface
public interface IDataProcess<T> extends IDataInput<T> {

}
@FunctionalInterface
public interface IDataInput<T> {
    void set(T data);
}

3. IDataKeyFetcher<T>(主键提取器)

>源码<

用于返回数据中的唯一标识

@FunctionalInterface
public interface IDataKeyFetcher<T> {
    String getKey(T data);
}

4. IStringCache(缓存组件)

>源码<

用于暂时缓存已经处理的数据唯一标识

public interface IStringCache {
    String get(String key);
    void set(String key, String value, int expireSecond);
    boolean has(String key);
    void remove(String key);
    void release();
    boolean isReleased();
}

5. IWorkSnapshoter(快照组件)

>源码<

用于保存读取数据的游标

public interface IWorkSnapshoter {
    void save(String[] cursors);
    String[] load();
}

☆ 一段测试demo

>源码<
输出

2019-11-18 16:49:04.568  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> 状态变更:[0]准备中 -> [1]就绪
2019-11-18 16:49:04.571  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> 状态变更:[1]就绪 -> [2]运行中
2019-11-18 16:49:04.576  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[0] run.
2019-11-18 16:49:04.576  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[2] run.
2019-11-18 16:49:04.576  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[1] run.
2019-11-18 16:49:04.576  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[3] run.
2019-11-18 16:49:04.576  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[0] run.
2019-11-18 16:49:04.577  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[1] run.
2019-11-18 16:49:04.577  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[2] run.
2019-11-18 16:49:04.577  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[3] run.
2019-11-18 16:49:04.577  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[4] run.
2019-11-18 16:49:04.577  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[5] run.
2019-11-18 16:49:04.577  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[6] run.
2019-11-18 16:49:04.577  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[7] run.
2019-11-18 16:49:04.577  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[8] run.
2019-11-18 16:49:04.577  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[9] run.
2019-11-18 16:49:04.585  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[4] run.
2019-11-18 16:49:04.585  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[5] run.
2019-11-18 16:49:04.587  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[6] run.
2019-11-18 16:49:04.603  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[0] exit.
2019-11-18 16:49:04.604  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[1] exit.
2019-11-18 16:49:04.605  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[2] exit.
2019-11-18 16:49:04.606  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[4] exit.
2019-11-18 16:49:04.607  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[3] exit.
2019-11-18 16:49:04.607  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[6] exit.
2019-11-18 16:49:04.611  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> reader[5] exit.
2019-11-18 16:49:04.611  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[2] exit.
2019-11-18 16:49:04.611  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[4] exit.
2019-11-18 16:49:04.611  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[6] exit.
2019-11-18 16:49:09.612  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[0] exit.
2019-11-18 16:49:09.612  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[3] exit.
2019-11-18 16:49:09.612  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[9] exit.
2019-11-18 16:49:09.612  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[7] exit.
2019-11-18 16:49:09.612  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[1] exit.
2019-11-18 16:49:09.612  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[5] exit.
2019-11-18 16:49:09.612  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> 状态变更:[2]运行中 -> [3]完成
2019-11-18 16:49:09.612  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> process[8] exit.
2019-11-18 16:49:09.614  com.blqw.work.impl.BlockingWorkCenter[2E23D1E3A9964510B69909A1EDDA2240] >> 退出...

☆ 异常中断后的恢复

// 创建工作中心
SingleBlockingWorkCenter<JSONObject> workCenter = new SingleBlockingWorkCenter<>(settings);
// 记录本次工作的凭证
this.ticket = workCenter.ticket;
// 开始工作(读取线程数, 处理线程数)
workCenter.startAsync(4, 10);
// 等待工作完成
workCenter.await();

// 意外断开后的重启, 传入上次的工作凭证
SingleBlockingWorkCenter<JSONObject> workCenter = new SingleBlockingWorkCenter<>(this.ticket, settings);
// 如果存在IWorkSnapshoter快照组件, 则会自动恢复上次工作中断时的游标
// 如果存在IStringCache和IDataKeyFetcher, 则会跳过已经被处理的数据
workCenter.startAsync(4, 10);

workCenter.await();

更新说明

[1.0.0.0] 2019.11.19

  • 初始版本

blqw.dataflow's People

Contributors

dependabot[bot] avatar zijian666 avatar

Watchers

avatar avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.