zeromicro / cds Goto Github PK
View Code? Open in Web Editor NEWData syncing in golang for ClickHouse.
License: MIT License
Data syncing in golang for ClickHouse.
License: MIT License
ERROR: for canal-server Container "4252636c3eef" is unhealthy.
ERROR: Encountered errors while bringing up the project.
Traceback (most recent call last):
File "/tmp/init_db.py", line 74, in
init_mongo()
File "/tmp/init_db.py", line 69, in init_mongo
collection.insert_many(result)
File "/usr/local/lib/python3.9/site-packages/pymongo/collection.py", line 761, in insert_many
blk.execute(write_concern, session=session)
File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 528, in execute
return self.execute_command(generator, write_concern, session)
File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 359, in execute_command
client._retry_with_session(
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1384, in _retry_with_session
return self._retry_internal(retryable, func, session, bulk)
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1414, in _retry_internal
raise last_error
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1416, in _retry_internal
return func(session, sock_info, retryable)
File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 353, in retryable_bulk
self._execute_command(
File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 309, in _execute_command
result, to_send = bwc.execute(ops, client)
File "/usr/local/lib/python3.9/site-packages/pymongo/message.py", line 907, in execute
result = self.write_command(request_id, msg, to_send)
File "/usr/local/lib/python3.9/site-packages/pymongo/message.py", line 999, in write_command
reply = self.sock_info.write_command(request_id, msg)
File "/usr/local/lib/python3.9/site-packages/pymongo/pool.py", line 771, in write_command
helpers._check_command_response(result, self.max_wire_version)
File "/usr/local/lib/python3.9/site-packages/pymongo/helpers.py", line 151, in _check_command_response
raise NotMasterError(errmsg, response)
pymongo.errors.NotMasterError: operation was interrupted, full error: {'errorLabels': ['RetryableWriteError'], 'topologyVersion': {'processId': ObjectId('6071d9eecc09c5721c8f196f'), 'counter': 6}, 'operationTime': Timestamp(1618074848, 500), 'ok': 0.0, 'errmsg': 'operation was interrupted', 'code': 11602, 'codeName': 'InterruptedDueToReplStateChange', '$clusterTime': {'clusterTime': Timestamp(1618074848, 500), 'signature': {'hash': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'keyId': 0}}}
make: *** [init] Error 1
环境: centos 7 | clickhouse ClickHouse server version 20.12.5 | cds 最新版(建议添加一个版本号, commitid: 28c51c5)
现像: 在配置好全量同步,执行同步过程出现如下错误:
{"@timestamp":"2021-01-04T17:03:44.268+08","level":"error","content":"mysqltypeconv.go:59 sql: Scan error on column index 4, name \"Default\":
converting NULL to string is unsupported"}
可能原因: 建表语句指定了字段类型, 导致NULL转换失败
问题:
* 建表语句是否可支持 Nullable
* 全库同步需要选中所有表 ?
是使用这种方式吗 https://mp.weixin.qq.com/s/a8OfsBn9VFnj7oxp0IIVGg
ReplacingMergeTree + 视图(argMax)?
RT
mongodb test_mongo
.example
inserted 10000 lines
Traceback (most recent call last):
File "/tmp/init_db.py", line 74, in
init_mongo()
File "/tmp/init_db.py", line 69, in init_mongo
collection.insert_many(result)
File "/usr/local/lib/python3.10/site-packages/pymongo/collection.py", line 615, in insert_many
blk.execute(write_concern, session=session)
File "/usr/local/lib/python3.10/site-packages/pymongo/bulk.py", line 459, in execute
return self.execute_command(generator, write_concern, session)
File "/usr/local/lib/python3.10/site-packages/pymongo/bulk.py", line 351, in execute_command
with client._tmp_session(session) as s:
File "/usr/local/lib/python3.10/contextlib.py", line 135, in enter
return next(self.gen)
File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1656, in _tmp_session
s = self._ensure_session(session)
File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1643, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1594, in __start_session
server_session = self._get_server_session()
File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1629, in _get_server_session
return self._topology.get_server_session()
File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 534, in get_server_session
session_timeout = self._check_session_support()
File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 520, in _check_session_support
self._select_servers_loop(
File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 223, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: mongo1:30001: [Errno -2] Name or service not known,mongo2:30002: [Errno -2] Name or service not known,mongo3:30003: [Errno -2] Name or service not known, Timeout: 30s, Topology Description: <TopologyDescription id: 624a83e9fd437d56bbaaf7a3, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongo1', 30001) server_type: Unknown, rtt: None, error=AutoReconnect('mongo1:30001: [Errno -2] Name or service not known')>, <ServerDescription ('mongo2', 30002) server_type: Unknown, rtt: None, error=AutoReconnect('mongo2:30002: [Errno -2] Name or service not known')>, <ServerDescription ('mongo3', 30003) server_type: Unknown, rtt: None, error=AutoReconnect('mongo3:30003: [Errno -2] Name or service not known')>]>
make: *** [init] Error 1
我这边clickhouse是一个3分片集群, 配置同步任务时clickhouse这边创建的是分布式表还是普通单表啊?
如题!
正在研究Go大数据相关生态,想加入社群,了解更多。
希望能及时更新微信二维码,谢谢!
开发组成员们,你们好。
在研究本项目的时候产生了一个疑问,希望能得到解答。
背景:在Galaxy生成的Clichouse DDL中,为每个表建立了”ck_is_delete“列,复制表的引擎选用:ReplicatedMergeTree。
问题:这里为什么没有选用ReplicatedCollapsingMergeTree作为复制表的引擎,使用”sign“代替”ck_is_delete“?
望解答
这个可以作为MySQL异地主从备份吗?当另一MySQL故障时,切到当前MySQL
mysql 的 binlog 实时解析,没有看到,你们这个同步工具,有没有做mysql binlog实时同步??
我全文搜索了 binlog ,没有发现binlog这个插件,请问你们实现了binlog解析吗?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.