doris连接问题 about addax

[zhbdesign]

测试了一下，导入一部分数据后出现错误，日志如下：
2021-02-24 15:34:23.754 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [26] tasks.
2021-02-24 15:34:23.768 [job-0] INFO JobContainer - DataX Writer.Job [doriswriter] splits to [26] tasks.
2021-02-24 15:34:23.812 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2021-02-24 15:34:23.842 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2021-02-24 15:34:23.851 [job-0] INFO JobContainer - Running by standalone Mode.
2021-02-24 15:34:23.884 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [5] channels for [26] tasks.
2021-02-24 15:34:23.901 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2021-02-24 15:34:23.906 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2021-02-24 15:34:23.920 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2021-02-24 15:34:23.929 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from detail where (1 <= order_id AND order_id < 366603)
] jdbcUrl:[jdbc:mysql://192.168.2.100:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-02-24 15:34:23.933 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] attemptCount[1] is started
2021-02-24 15:34:23.970 [0-0-1-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from detail where (366603 <= order_id AND order_id < 733205)
] jdbcUrl:[jdbc:mysql://192.168.2.100:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-02-24 15:34:23.977 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] attemptCount[1] is started
2021-02-24 15:34:23.988 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] attemptCount[1] is started
2021-02-24 15:34:23.990 [0-0-2-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from detail where (733205 <= order_id AND order_id < 1099807)
] jdbcUrl:[jdbc:mysql://192.168.2.100:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-02-24 15:34:24.026 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] attemptCount[1] is started
2021-02-24 15:34:24.023 [0-0-3-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from detail where (1099807 <= order_id AND order_id < 1466409)
] jdbcUrl:[jdbc:mysql://192.168.2.100:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-02-24 15:34:24.038 [0-0-4-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from detail where (1466409 <= order_id AND order_id < 1833011)
] jdbcUrl:[jdbc:mysql://192.168.2.100:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-02-24 15:34:24.098 [0-0-1-writer] INFO DorisWriterTask - connect DorisDB with http://192.168.2.100:8030/api/ssb/detail/_stream_load
2021-02-24 15:34:24.099 [0-0-0-writer] INFO DorisWriterTask - connect DorisDB with http://192.168.2.100:8030/api/ssb/detail/_stream_load
2021-02-24 15:34:24.098 [0-0-4-writer] INFO DorisWriterTask - connect DorisDB with http://192.168.2.100:8030/api/ssb/detail/_stream_load
2021-02-24 15:34:24.098 [0-0-3-writer] INFO DorisWriterTask - connect DorisDB with http://192.168.2.100:8030/api/ssb/detail/_stream_load
2021-02-24 15:34:24.099 [0-0-2-writer] INFO DorisWriterTask - connect DorisDB with http://192.168.2.100:8030/api/ssb/detail/_stream_load
2021-02-24 15:34:34.142 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2021-02-24 15:34:44.299 [job-0] INFO StandAloneJobContainerCommunicator - Total 152928 records, 2835540 bytes | Speed 276.91KB/s, 15292 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.123s | All Task WaitReaderTime 28.330s | Percentage 0.00%
2021-02-24 15:34:53.881 [job-0] INFO StandAloneJobContainerCommunicator - Total 217632 records, 4143828 bytes | Speed 127.76KB/s, 6470 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.129s | All Task WaitReaderTime 86.984s | Percentage 0.00%
2021-02-24 15:35:03.892 [job-0] INFO StandAloneJobContainerCommunicator - Total 325856 records, 6469076 bytes | Speed 227.07KB/s, 10822 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.133s | All Task WaitReaderTime 141.762s | Percentage 0.00%
2021-02-24 15:35:09.717 [0-0-2-writer] ERROR WriterRunner - Writer Runner Received Exceptions:
org.apache.http.client.ClientProtocolException: null
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.4.jar:4.4]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.sendData(DorisWriterTask.java:169) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.startWrite(DorisWriterTask.java:127) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.startWrite(DorisWriter.java:100) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_172]
Caused by: org.apache.http.ProtocolException: Content-Length header already present
at org.apache.http.protocol.RequestContent.process(RequestContent.java:96) ~[httpcore-4.4.jar:4.4]
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) ~[httpcore-4.4.jar:4.4]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.4.jar:4.4]
... 7 common frames omitted

[wgzhao]

@zhbdesign 这是我的版本吗？
另外，这是因为httpclient 组件的问题，请升级到 4.5及以上版本，这个错误，在官方样例代码有特别注释说明：

 * 1 wrong dependency version(such as 4.4) of httpclient may cause shaded.org.apache.http.ProtocolException
 *   Caused by: shaded.org.apache.http.ProtocolException: Content-Length header already present
 *     at shaded.org.apache.http.protocol.RequestContent.process(RequestContent.java:96)
 *     at shaded.org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
 *     at shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182)
 *     at shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
 *     at shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
 *     at shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)

[zhbdesign]

@zhbdesign 这是我的版本吗？
另外，这是因为httpclient 组件的问题，请升级到 4.5及以上版本，这个错误，在官方样例代码有特别注释说明：

 * 1 wrong dependency version(such as 4.4) of httpclient may cause shaded.org.apache.http.ProtocolException
 *   Caused by: shaded.org.apache.http.ProtocolException: Content-Length header already present
 *     at shaded.org.apache.http.protocol.RequestContent.process(RequestContent.java:96)
 *     at shaded.org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
 *     at shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182)
 *     at shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
 *     at shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
 *     at shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)

我移动到我以前的代码了，明天看下版本，另外，速度确实上去了，100000/s

[wgzhao]

从你提供的信息来看，你刚好用的是 httpclient 4.4版本，而且报错信息和官方注释一样，所以很有可能就是这个问题。

[zhbdesign]

从你提供的信息来看，你刚好用的是 httpclient 4.4版本，而且报错信息和官方注释一样，所以很有可能就是这个问题。

我检查了版本，是4.5的版本，打包后对比的图片如下：

这是后来改成的和官方一样的4.5.3也不行，4.5.13也不行，是不是我把代码移到阿里的代码上去出现的问题

[zhbdesign]

移动后的源码如下

doriswriter.zip

打包后的包如下：
doriswriter.zip

[wgzhao]

注意看你报错信息这一段

2021-02-24 15:35:09.717 [0-0-2-writer] ERROR WriterRunner - Writer Runner Received Exceptions:
org.apache.http.client.ClientProtocolException: null
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.4.jar:4.4]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.sendData(DorisWriterTask.java:169) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.startWrite(DorisWriterTask.java:127) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.startWrite(DorisWriter.java:100) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_172]
Caused by: org.apache.http.ProtocolException: Content-Length header already present
at org.apache.http.protocol.RequestContent.process(RequestContent.java:96) ~[httpcore-4.4.jar:4.4]
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) ~[httpcore-4.4.jar:4.4]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.4.jar:4.4]

注意其中的 httpclient, httpcore 版本，你可以用新版本直接替换 plugin/writer/doriswriter/libs 目录下的这几个jar包，然后再测试看看。

[zhbdesign]

注意看你报错信息这一段

2021-02-24 15:35:09.717 [0-0-2-writer] ERROR WriterRunner - Writer Runner Received Exceptions:
org.apache.http.client.ClientProtocolException: null
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.4.jar:4.4]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.sendData(DorisWriterTask.java:169) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterTask.startWrite(DorisWriterTask.java:127) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.startWrite(DorisWriter.java:100) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_172]
Caused by: org.apache.http.ProtocolException: Content-Length header already present
at org.apache.http.protocol.RequestContent.process(RequestContent.java:96) ~[httpcore-4.4.jar:4.4]
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) ~[httpcore-4.4.jar:4.4]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.4.jar:4.4]

注意其中的 httpclient, httpcore 版本，你可以用新版本直接替换 plugin/writer/doriswriter/libs 目录下的这几个jar包，然后再测试看看。

多谢指导，终于找到原因了，阿里的版本里，core和common两个包里也用到了httpclient，都换成4.5.13才行