tianlangstudio / dataxserver

为DataX(https://github.com/alibaba/DataX) 提供远程多语言调用（ThriftServer，HttpServer） 分布式运行（DataX on YARN） 功能

Hi there 👋

Hello, Glad to meet you. I'm Fusion Zhu, a Full-stack development with experience in Java, JavaScript, Bigdata, and Rust.
Yep, I just like to learn new techniques.

Projects I did or participated in on GitHub:

Big Data

• DataXServer Distributed ETL Hadoop Kafka postgresql Java Scala
• Databend Distributed Warehouse Rust
• Spark JobServer Scala
  
• BigQuery Proxy bigdata gcp async actix_web Rust
• Email Analysis Clickhouse ElasticSearch D3.js Java JavaScript
  
• Heatmap Clickstream Spark Kafka Postgres Java Scala Html5
  

Data Visualization

• Charts D3.js echarts JavaScript
• ModelShow Clickhouse Spark React Java JavaScript
• excel-column-letter2num NodeJs JavaScript

Application

• Content Manager System CMS Blog actix-web diesel mysql Rust JavaScript
• Win On GitHub Desktop Tauri React NextJs Rust JavaScript
• TlPortal Web Spring ReactJS Java JavaScript
• Make Image As Map React Rust JavaScript

Game

• PengPengQiu NodeJs Html5 websocket JavaScript
• Pingpang Piston Rust
• Arithmetic Bevy Rust
  

Other

• Sokoban Game Book Translation Rust

👯 I look for cooperation opportunities on Upwork.

👨‍💼 I currently work in Shenzhen,China.

🏴󠁧󠁢󠁥󠁮󠁧󠁿 I currently learn English.

📺 I currently continue to make some videos about learning programming.

dataxserver's Issues

每个 datax 的Excutor 进程 都是接收 threft消息 还是 HTTP请求？

excutor和master之间是使用的的akka

是为了实现分布式多任务吗？

怎么实现的？没看明白

dataX on yarn

@TianLangStudio
DataXServer/src/main/java/com/tianlangstudio/data/datax/yarn/YarnApplicationMaster.java AM的实现不完整?

集群里每台机器都要要安装datax？

不需要每台机器都安装 可以打到zip包里放hdfs里 这就是excutor的zip包里

httpserver 编译出错

[INFO] ------------------------------------------------------------------------
[INFO] Building Hamal HTTP Server 2.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- build-helper-maven-plugin:1.9.1:add-source (add-source) @ hamal-http-server ---
[INFO] Source directory: D:\code\my\workspace\DataXServer-master\httpserver\src\main\thrift\gen-java added.
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hamal-http-server ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ hamal-http-server ---
[WARNING] Expected all dependencies to require Scala version: 2.13.0
[WARNING] com.typesafe.akka:akka-actor_2.13:2.5.25 requires scala version: 2.12.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] Using incremental compilation
[INFO] 'compiler-interface' not yet compiled for Scala 2.13.0. Compiling...
C:\Users\002696\AppData\Local\Temp\sbt_3fa94ed9\xsbt\Compat.scala:127: error: postfix operator headOption needs to be enabled
by making the implicit value scala.language.postfixOps visible.
This can be achieved by adding the import clause 'import scala.language.postfixOps'
or by setting the compiler option -language:postfixOps.
See the Scaladoc for value scala.language.postfixOps for a discussion
why the feature needs to be explicitly enabled.
} headOption
^
one error found

是否支持单个job里 传输的数据分片，在yarn上运行多个Container并行执行？

DataX3.0 的怎么用你的这个呀 谢谢

你好，因为现在我们也想DataX On Yarn，看到您开源的 但是你的使用手册 能多写一些吗 请问现在还维护嘛

如果 任务 逐渐增多，datax Excutor的会增多，任务少了 datax 的Excutor也会减少？

会逐渐增多也会逐渐减少 可以通过配置文件配置

单机多线程模式，并发提交两个reader不同类型任务报错

同时提交两个不同reader类型的任务，
后者能成功，前者在加载reader的时候直接报错。

可以像spark一样，以client模式提交到yarn，这样master的地址就固定了？

excutor进程就是运行任务的datax进程？

是的 每个excutor就是一个运行datax的进程