Comments (8)
测试了一下,导入一部分数据后出现错误,日志如下:
2021-02-24 15:34:23.754 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [26] tasks.
2021-02-24 15:34:23.768 [job-0] INFO JobContainer - DataX Writer.Job [doriswriter] splits to [26] tasks.
2021-02-24 15:34:23.812 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2021-02-24 15:34:23.842 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2021-02-24 15:34:23.851 [job-0] INFO JobContainer - Running by standalone Mode.
2021-02-24 15:34:23.884 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [5] channels for [26] tasks.
2021-02-24 15:34:23.901 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2021-02-24 15:34:23.906 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2021-02-24 15:34:23.920 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2021-02-24 15:34:23.929 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from detail where (1 <= order_id AND order_id < 366603)
] jdbcUrl:[jdbc:mysql://192.168.2.100:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-02-24 15:34:23.933 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] attemptCount[1] is started
2021-02-24 15:34:23.970 [0-0-1-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from detail where (366603 <= order_id AND order_id < 733205)
] jdbcUrl:[jdbc:mysql://192.168.2.100:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-02-24 15:34:23.977 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] attemptCount[1] is started
2021-02-24 15:34:23.988 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] attemptCount[1] is started
2021-02-24 15:34:23.990 [0-0-2-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from detail where (733205 <= order_id AND order_id < 1099807)
] jdbcUrl:[jdbc:mysql://192.168.2.100:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-02-24 15:34:24.026 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] attemptCount[1] is started
2021-02-24 15:34:24.023 [0-0-3-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from detail where (1099807 <= order_id AND order_id < 1466409)
] jdbcUrl:[jdbc:mysql://192.168.2.100:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-02-24 15:34:24.038 [0-0-4-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from detail where (1466409 <= order_id AND order_id < 1833011)
] jdbcUrl:[jdbc:mysql://192.168.2.100:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-02-24 15:34:24.098 [0-0-1-writer] INFO DorisWriterTask - connect DorisDB with http://192.168.2.100:8030/api/ssb/detail/_stream_load
2021-02-24 15:34:24.099 [0-0-0-writer] INFO DorisWriterTask - connect DorisDB with http://192.168.2.100:8030/api/ssb/detail/_stream_load
2021-02-24 15:34:24.098 [0-0-4-writer] INFO DorisWriterTask - connect DorisDB with http://192.168.2.100:8030/api/ssb/detail/_stream_load
2021-02-24 15:34:24.098 [0-0-3-writer] INFO DorisWriterTask - connect DorisDB with http://192.168.2.100:8030/api/ssb/detail/_stream_load
2021-02-24 15:34:24.099 [0-0-2-writer] INFO DorisWriterTask - connect DorisDB with http://192.168.2.100:8030/api/ssb/detail/_stream_load
2021-02-24 15:34:34.142 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2021-02-24 15:34:44.299 [job-0] INFO StandAloneJobContainerCommunicator - Total 152928 records, 2835540 bytes | Speed 276.91KB/s, 15292 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.123s | All Task WaitReaderTime 28.330s | Percentage 0.00%
2021-02-24 15:34:53.881 [job-0] INFO StandAloneJobContainerCommunicator - Total 217632 records, 4143828 bytes | Speed 127.76KB/s, 6470 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.129s | All Task WaitReaderTime 86.984s | Percentage 0.00%
2021-02-24 15:35:03.892 [job-0] INFO StandAloneJobContainerCommunicator - Total 325856 records, 6469076 bytes | Speed 227.07KB/s, 10822 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.133s | All Task WaitReaderTime 141.762s | Percentage 0.00%
2021-02-24 15:35:09.717 [0-0-2-writer] ERROR WriterRunner - Writer Runner Received Exceptions:
org.apache.http.client.ClientProtocolException: null
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.4.jar:4.4]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.sendData(DorisWriterTask.java:169) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.startWrite(DorisWriterTask.java:127) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.startWrite(DorisWriter.java:100) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_172]
Caused by: org.apache.http.ProtocolException: Content-Length header already present
at org.apache.http.protocol.RequestContent.process(RequestContent.java:96) ~[httpcore-4.4.jar:4.4]
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) ~[httpcore-4.4.jar:4.4]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.4.jar:4.4]
... 7 common frames omitted
from addax.
@zhbdesign 这是我的版本吗?
另外,这是因为httpclient 组件的问题,请升级到 4.5及以上版本,这个错误,在官方样例代码有特别注释说明:
* 1 wrong dependency version(such as 4.4) of httpclient may cause shaded.org.apache.http.ProtocolException
* Caused by: shaded.org.apache.http.ProtocolException: Content-Length header already present
* at shaded.org.apache.http.protocol.RequestContent.process(RequestContent.java:96)
* at shaded.org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
* at shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182)
* at shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
* at shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
* at shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
from addax.
@zhbdesign 这是我的版本吗?
另外,这是因为httpclient 组件的问题,请升级到 4.5及以上版本,这个错误,在官方样例代码有特别注释说明:* 1 wrong dependency version(such as 4.4) of httpclient may cause shaded.org.apache.http.ProtocolException * Caused by: shaded.org.apache.http.ProtocolException: Content-Length header already present * at shaded.org.apache.http.protocol.RequestContent.process(RequestContent.java:96) * at shaded.org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) * at shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182) * at shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) * at shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) * at shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
我移动到我以前的代码了,明天看下版本,另外,速度确实上去了,100000/s
from addax.
从你提供的信息来看,你刚好用的是 httpclient 4.4版本,而且报错信息和官方注释一样,所以很有可能就是这个问题。
from addax.
从你提供的信息来看,你刚好用的是 httpclient 4.4版本,而且报错信息和官方注释一样,所以很有可能就是这个问题。
我检查了版本,是4.5的版本,打包后对比的图片如下:
这是后来改成的和官方一样的4.5.3也不行,4.5.13也不行,是不是我把代码移到阿里的代码上去出现的问题
from addax.
移动后的源码如下
打包后的包如下:
doriswriter.zip
from addax.
注意看你报错信息这一段
2021-02-24 15:35:09.717 [0-0-2-writer] ERROR WriterRunner - Writer Runner Received Exceptions:
org.apache.http.client.ClientProtocolException: null
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.4.jar:4.4]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.sendData(DorisWriterTask.java:169) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.startWrite(DorisWriterTask.java:127) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.startWrite(DorisWriter.java:100) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_172]
Caused by: org.apache.http.ProtocolException: Content-Length header already present
at org.apache.http.protocol.RequestContent.process(RequestContent.java:96) ~[httpcore-4.4.jar:4.4]
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) ~[httpcore-4.4.jar:4.4]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.4.jar:4.4]
注意其中的 httpclient
, httpcore
版本,你可以用新版本直接替换 plugin/writer/doriswriter/libs
目录下的这几个jar包,然后再测试看看。
from addax.
注意看你报错信息这一段
2021-02-24 15:35:09.717 [0-0-2-writer] ERROR WriterRunner - Writer Runner Received Exceptions: org.apache.http.client.ClientProtocolException: null at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) ~[httpclient-4.4.jar:4.4] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.4.jar:4.4] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.4.jar:4.4] at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.sendData(DorisWriterTask.java:169) ~[doriswriter-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.startWrite(DorisWriterTask.java:127) ~[doriswriter-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.startWrite(DorisWriter.java:100) ~[doriswriter-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) ~[datax-core-0.0.1-SNAPSHOT.jar:na] at java.lang.Thread.run(Unknown Source) [na:1.8.0_172] Caused by: org.apache.http.ProtocolException: Content-Length header already present at org.apache.http.protocol.RequestContent.process(RequestContent.java:96) ~[httpcore-4.4.jar:4.4] at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) ~[httpcore-4.4.jar:4.4] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182) ~[httpclient-4.4.jar:4.4] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.4.jar:4.4] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.4.jar:4.4] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.4.jar:4.4]
注意其中的
httpclient
,httpcore
版本,你可以用新版本直接替换plugin/writer/doriswriter/libs
目录下的这几个jar包,然后再测试看看。
多谢指导,终于找到原因了,阿里的版本里,core和common两个包里也用到了httpclient,都换成4.5.13才行
from addax.
Related Issues (20)
- Release notes for 4.1.2 HOT 1
- [Bug]: hdfsReader to clickhouse 出现 jackson/databind NoClassDefFoundError HOT 2
- [Bug]: 执行bin/addax.sh job/test.json 报错:没有那个文件或目录 HOT 2
- [Bug]: 你好 是否必须要先在输出端先建表才能使用 HOT 7
- [Bug]: Doris writer写入bug
- 在线文档能提供星环inceptor的配置示例吗 HOT 1
- oracle在使用"writeMode": "update(id)"模式下写入,表有clob大字段时报错[Bug]: HOT 5
- [Bug]: 部署在墨西哥当地机器的时候,数据源连接自动生成的时区参数错误,异常退出 HOT 5
- [Bug]: channel描述是不是有点问题 HOT 1
- Release notes for 4.1.3 HOT 2
- Release notes for 4.1.4 HOT 1
- Sap Hana组件会支持吗 HOT 4
- [Bug]: dateFormat格式化日期时间无效
- 自行编译项目后,执行 addax.sh 时无法正常加载 shared和plugin/*/*/libs的依赖 HOT 6
- [Bug]: java.security.InvalidAlgorithmParameterException: Cannot reuse iv for GCM encryption
- 新需求:ftpreader 支持Excel格式的文件读取 HOT 2
- [Bug]: 执行脚本时出现 java.lang.NoClassDefFoundError HOT 3
- [Bug]: 数据转换插件 提示找不到Class文件 NoClassDefFoundError HOT 22
- 【feat】支持一下kyuubi,要比hive性能更好 HOT 2
- [Bug]: mysql-分库分表情况下channel无效 HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from addax.