hellobike / amazonriver Goto Github PK
View Code? Open in Web Editor NEWamazonriver 是一个将postgresql的实时数据同步到es或kafka的服务
License: Apache License 2.0
amazonriver 是一个将postgresql的实时数据同步到es或kafka的服务
License: Apache License 2.0
您好,我用amazonriver从PG同步到kafka,数据读取正常,但是未写入对应kafka的topic中,只是把数据打印到了终端,请问是不是我的配置或者操作存在问题:
kafka版本:0.10.2.0
postgresql版本:9.6.12
1.下载文件:amazonriver-linux-amd64.zip
2.解压后执行:./amazonriver -config kafka.conf
kafka.conf 文件配置信息如下:
{
"pg_dump_path": "",
"subscribes": [{
"dump": false,
"slotName": "slot_for_kafka",
"pgConnConf": {
"host": "192.168.216.87",
"port": 6432,
"database": "grabdb",
"user": "replica",
"password": "*****“
},
"rules": [
{
"table": "db40.dbm_monitor_timeliness",
"pks": ["table_name"],
"topic": "dbm_monitor_timeliness"
}
],
"kafkaConf": {
"addrs": ["192.168.200.94:9092"]
},
"retry": 0
}],
"prometheus_address": ":8080"
}
3.kafka的topic:dbm_monitor_timeliness信息如下
Topic:dbm_monitor_timeliness PartitionCount:1 ReplicationFactor:3 Configs:
Topic: dbm_monitor_timeliness Partition: 0 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1
PGSQL 中执行TRUNCATE table test
报错
time="2020-01-17T18:07:42+08:00" level=info msg="handle wal data: &{BEGIN map[] 1579255662349 4169506000 <nil>}"
time="2020-01-17T18:07:42+08:00" level=error msg="handle replication msg err: invalid pgoutput msg: invalid parser end state: 4"
time="2020-01-17T18:07:42+08:00" level=info msg="handle wal data: &{COMMIT map[] 1579255662349 4169508368 <nil>}"
PGSQL:PostgreSQL 12.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit
amazonriver:1.0.3
ES:7.3
请问amazonriver能否实现高可用呢?如何实现?
INFO[0366] producer/leader/student_name_logs/0 state change to [normal]
ERRO[0366] write kafka msg err: kafka: Failed to deliver 2102 messages.
运行环境:win10
数据库环境:pg 10.9.0
es版本:7.4
已经将pg_dump 配置到环境变量了
配置信息:
{
"pg_dump_path": "",
"subscribes": [{
"dump": true,
"slotName": "tms_for_es",
"pgConnConf": {
"host": "127.0.0.1",
"port": 5432,
"database": "pg2",
"user": "postgres",
"password": "post"
},
"rules": [
{
"table": "product",
"pks": ["tid"],
"esid": ["id"],
"index": "product_name",
"type": "string"
},
{
"table": "version_object",
"pks": ["tid"],
"esid": ["id"],
"index": "version",
"type": "string"
}
],
"esConf": {
"addrs": "http://localhost:9200",
"user": "",
"password": ""
},
"retry": -1
}]
}
运行amazonriver -config config.json报错如题,使用curl命令在es上面已经创建了对应的索引,es版本为5.5
Please check if it works. Thank you.
cannot work with new version pgx
这个问题必现,处理数据到32000多条的时候,后面的改动就监听不到了,逻辑复制WaitForReplicationMessage那里直接就接收不到表改动的waldata
一段时间不更新的表(10分钟以上),第一次更新的数据没有被写入到到kafka中,第二次更新的才会写入kafka,出现问题频率较高
pg_dump_path 指向的pg_dump 备份文件 有什么要求?是 sql 还是二进制?文件后缀有没有要求?
我都试了,没效果,能提供pg_dump 的命令示例吗?
我是这样创建的备份文件:
pg_dump -h 10.168.4.91 -p 5321 -U postgres -W -F p -b -v -f "/opt/amazonriver/xzdssituation_p.sql" xzdssituation
这样配置的
"pg_dump_path": "/opt/amazonriver/xzdssituation_p.sql",
"subscribes": [{
"dump": true,
能否支持的PG的wal物理文件方式的复制,不用逻辑方式,因为PG在用逻辑方式复制后,会出现主从延迟高的问题。
表必须设置主键,不然delete时无法同步,此时从pg订阅到的消息是空的,导致amazonriver无法删除。
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.