dotnetcore / canalsharp Goto Github PK
View Code? Open in Web Editor NEWAlibaba mysql database binlog subscription & consumer components Canal's .NET client.
License: Apache License 2.0
Alibaba mysql database binlog subscription & consumer components Canal's .NET client.
License: Apache License 2.0
大家好,我们想收集一下使用了CanalSharp的用户,如果可以,请您留下你的公司名字,最好能带Logo,我们将收集并展示在Readme中。
就很简单用的Demo的东西,然后连着Canal平均一个多小时就和Canal断开连接了,跑的环境是在一个线程上跑,得到结果后,会在另一条线程通知出去,能够知道是什么原因吗?
我看sample代码中是while 轮训读取订阅的消息,是否可以考虑支持事件订阅的方式?我总觉得轮训的方式实现不够优美。
//获取数据但是不需要发送Ack来表示消费成功
connector.Get(batchSize);
//获取数据并且需要发送Ack表示消费成功
connector.GetWithoutAck(batchSize);
Please answer these questions before submitting your issue.
Which version of Canal-Server, OS ,Mysql?
MySQL版本:8.0.35
CanalSharp:1.2.1
What happen?
If possible, provide a way for reproducing the error. e.g. demo application.
var loggerFactory = LoggerFactory.Create(builder =>
{
builder
.AddFilter("Microsoft", LogLevel.Debug)
.AddFilter("System", LogLevel.Information)
.AddConsole();
});
var _logger = loggerFactory.CreateLogger();
string id = Console.ReadLine();
var conn = new SimpleCanalConnection(new SimpleCanalOptions("127.0.0.1", 11111, id), _logger);
//连接到 Canal Server
await conn.ConnectAsync();
//订阅
await conn.SubscribeAsync("canal_test.test_user");
while (true)
{
//var entries = await conn.GetAsync(1024);
var entries = await conn.GetWithoutAckAsync(1024);
foreach (var entry in entries.Entries)
{
//不处理事务标记
if (entry.EntryType == EntryType.Transactionbegin || entry.EntryType == EntryType.Transactionend)
{
continue;
}
RowChange rowChange = null;
try
{
rowChange = RowChange.Parser.ParseFrom(entry.StoreValue);
var eventType = rowChange.EventType;
foreach (var rowData in rowChange.RowDatas)
{
//删除的数据
if (eventType == CanalSharp.Protocol.EventType.Delete)
{
PrintColumn(rowData.BeforeColumns.ToList());
}
//插入的数据
else if (eventType == CanalSharp.Protocol.EventType.Insert)
{
PrintColumn(rowData.AfterColumns.ToList());
}
//更新的数据
else
{
_logger.LogInformation("-------> before");
PrintColumn(rowData.BeforeColumns.ToList());
_logger.LogInformation("-------> after");
PrintColumn(rowData.AfterColumns.ToList());
}
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
}
await conn.RollbackAsync(entries.Id);
}
Console.ReadLine();
比如我线上有三台mysql,一个canal server三个instance分别连接到这三台mysql;
现在一个canalsharp客户端能获取到三个instance的消息吗?
使用CanalSharp的公司可以记录下 公司+网址,谢谢
When I use CanalSharp, I found it will not work when my account has no privilege of SUPER (but it has SELECT, REPLICATION SLAVE and REPLICATION CLIENT), that is to say, it can't get message from binlog.
On the contary, it works when has SUPER permission.
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO canal @'%';
There is one issue about this on Alibaba-Canal project, but I see others said no need the SUPER permission, so I'd like to know is really no need it from your side because it doesn't works on my side.
Reference: alibaba/canal#1323
Canal 服务端用kakfa 模式,不会启用11111端口,客户端却从11111端口取数据。
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.
sample/Docker/docker-compose.yml
mysql 5.7
docs/package.json
vuepress ^1.7.1
sample/CanalSharp.SimpleApp/CanalSharp.SimpleApp.csproj
Microsoft.Extensions.Logging.Console 7.0.0
src/CanalSharp/CanalSharp.csproj
NZookeeper 1.0.0
Newtonsoft.Json 12.0.3
Microsoft.Extensions.Options 8.0.2
Microsoft.Extensions.Logging.Abstractions 8.0.1
Google.Protobuf 3.22.0
BeetleX 1.6.8
Please answer these questions before submitting your issue.
Which version of Canal-Server, OS ,Mysql?
What happen?
If possible, provide a way for reproducing the error. e.g. demo application.
sh run.sh -e canal.auto.scan=false -e canal.destinations=test -e canal.instance.master.address=192.168.13.86:3306 -e canal.instance.dbUsername=root -e canal.instance.dbPassword=123456789 -e canal.instance.connectionCharset=UTF-8 -e canal.instance.tsdb.enable=true -e canal.instance.gtidon=false
docker run -d -it -h 192.168.13.45 -e canal.auto.scan=false -e canal.destinations=test -e canal.instance.master.address=192.168.13.86:3306 -e canal.instance.dbUsername=root -e canal.instance.dbPassword=123456789 -e canal.instance.connectionCharset=UTF-8 -e canal.instance.tsdb.enable=true -e canal.instance.gtidon=false --name=canal-server --net=host -p 8000:8000 -p 2222:2222 -p 11111:11111 -p 11112:11112 -m 4096m canal/canal-server
DOCKER_DEPLOY_TYPE=VM
==> INIT /alidata/init/02init-sshd.sh
==> EXIT CODE: 0
==> INIT /alidata/init/fix-hosts.py
==> EXIT CODE: 0
==> INIT DEFAULT
Generating SSH1 RSA host key: [ OK ]
Starting sshd: [ OK ]
Starting crond: [ OK ]
==> INIT DONE
==> RUN /home/admin/app.sh
==> START ...
start canal ...
start canal successful
==> START SUCCESSFUL ...
使用的canalsharp的demo代码
static void Main(string[] args)
{
LogManager.Configuration = new XmlLoggingConfiguration("NLog.Config");
CanalSharpLogManager.LoggerFactory.AddNLog();
_logger = CanalSharpLogManager.LoggerFactory.CreateLogger("Program");
var destination = "test";
var connector = CanalConnectors.NewSingleConnector("192.168.13.45", 11111, destination, "", "");
//连接 Canal
connector.Connect();
connector.Subscribe(".*\\..*");
while (true)
{
var message = connector.Get(1024);
var batchId = message.Id;
if (batchId == -1 || message.Entries.Count <= 0)
{
Thread.Sleep(300);
continue;
}
PrintEntry(message.Entries);
}
}
[14:20:24]:Canal connect success. IP: 192.168.13.45, Port: 11111
[14:20:24]:Subscribe success. Filter: .*\..*
看起来每个步骤都正常,但是操作语句update canal set id = 2,无法在客户端正常订阅。
在使用CanalSharp监听的时候,只能获取到增量数据,能否支持全量更新?先同步历史数据,然后再增量更新?
Please answer these questions before submitting your issue.
CanalSharp.Protocol.Exception.CanalClientException: failed to subscribe with reason: something goes wrong with channel:[id: 0x4d67f2df, /172.16.10.16:63064 => /172.16.10.16:11111], exception=com.alibaba.otter.canal.server.exception.CanalServerException: destination:database_dn1 should start first 在 CanalSharp.Client.Impl.SimpleCanalConnector.Subscribe(String filter) 在 GdkySyncService.canal_sync_job.DoSync(CancellationToken token, Bas_Datasource example) 位置 D:\projects\GDKY_Data_MS\GDKY_Data_MS\GdkySyncService\Jobs\Sync\canal_sync_job.cs:行号 154
经常出现该问题,需要删除meta.dat和h2.mv.db文件后,重启才能好,这是为啥呢???急急急
canal1.1.3
--mysql 5.7
--windows server2012
What happen?
运行一段时间
connector.Subscribe("db.table");
这句话不会覆盖掉服务端设置的默认filter,无法做到监听指定库指定表的binlog变化,必须在服务端的instance.properties里设置才有效
获取变更列的中文值是是乱码,canal(canal.deployer-1.1.4.tar.gz)服务端在centos 中运行,已经设置了canal.instance.connectionCharset = UTF-8。数据库也是utf8mb4。客户端在linux 和windows中获取的中文值均为乱码
Canal supports protobuf3.0, is there sample code? I'm a newbie, thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.