dotnetcore / canalsharp Goto Github PK

大家好，我们想收集一下使用了CanalSharp的用户，如果可以，请您留下你的公司名字，最好能带Logo，我们将收集并展示在Readme中。

就很简单用的Demo的东西，然后连着Canal平均一个多小时就和Canal断开连接了，跑的环境是在一个线程上跑，得到结果后，会在另一条线程通知出去，能够知道是什么原因吗？

我看sample代码中是while 轮训读取订阅的消息，是否可以考虑支持事件订阅的方式？我总觉得轮训的方式实现不够优美。

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• [√] Question or discussion
• Bug
• Requirement
• Feature or performance improvement

Question

• What do you want to know?
  我在测试回滚，回滚后再获取数据时报错“Received an error returned by the server: something goes wrong with channel:[id: 0x146b0a4c, /127.0.0.1:64186 => /127.0.0.1:11111], exception=com.alibaba.otter.canal.server.exception.CanalServerException: rollback error, clientId:1 batchId:-1 is not exist , please check
  ”

Bug

• Which version of Canal-Server, OS ,Mysql?
  MySQL版本：8.0.35
  CanalSharp：1.2.1

• What happen?
  If possible, provide a way for reproducing the error. e.g. demo application.
  var loggerFactory = LoggerFactory.Create(builder =>
  {
  builder
  .AddFilter("Microsoft", LogLevel.Debug)
  .AddFilter("System", LogLevel.Information)
  .AddConsole();
  });
  var _logger = loggerFactory.CreateLogger();
  string id = Console.ReadLine();
  var conn = new SimpleCanalConnection(new SimpleCanalOptions("127.0.0.1", 11111, id), _logger);
  //连接到 Canal Server
  await conn.ConnectAsync();
  //订阅
  await conn.SubscribeAsync("canal_test.test_user");

while (true)
{
//var entries = await conn.GetAsync(1024);
var entries = await conn.GetWithoutAckAsync(1024);
foreach (var entry in entries.Entries)
{
//不处理事务标记
if (entry.EntryType == EntryType.Transactionbegin || entry.EntryType == EntryType.Transactionend)
{
continue;
}

    RowChange rowChange = null;
    try
    {
        rowChange = RowChange.Parser.ParseFrom(entry.StoreValue);
        var eventType = rowChange.EventType;
        foreach (var rowData in rowChange.RowDatas)
        {
            //删除的数据
            if (eventType == CanalSharp.Protocol.EventType.Delete)
            {
                PrintColumn(rowData.BeforeColumns.ToList());
            }
            //插入的数据
            else if (eventType == CanalSharp.Protocol.EventType.Insert)
            {
                PrintColumn(rowData.AfterColumns.ToList());
            }
            //更新的数据
            else
            {
                _logger.LogInformation("-------> before");
                PrintColumn(rowData.BeforeColumns.ToList());
                _logger.LogInformation("-------> after");
                PrintColumn(rowData.AfterColumns.ToList());
            }
        }

    }
    catch (Exception e)
    {
        Console.WriteLine(e);
    }
}
await conn.RollbackAsync(entries.Id);

}

Console.ReadLine();

Requirement or improvement

• Please describe about your requirements or improvement suggestions.

比如我线上有三台mysql，一个canal server三个instance分别连接到这三台mysql；
现在一个canalsharp客户端能获取到三个instance的消息吗？

使用CanalSharp的公司可以记录下 公司+网址，谢谢

When I use CanalSharp, I found it will not work when my account has no privilege of SUPER (but it has SELECT, REPLICATION SLAVE and REPLICATION CLIENT), that is to say, it can't get message from binlog.
On the contary, it works when has SUPER permission.
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO canal @'%';
There is one issue about this on Alibaba-Canal project, but I see others said no need the SUPER permission, so I'd like to know is really no need it from your side because it doesn't works on my side.

Reference: alibaba/canal#1323

Canal 服务端用kakfa 模式，不会启用11111端口，客户端却从11111端口取数据。

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency beetlex to v1.8.24.435
• chore(deps): update dependency google.protobuf to v3.27.3
• chore(deps): update dependency microsoft.extensions.logging.console to v8
• chore(deps): update mysql docker tag to v9
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(deps): update dependency vuepress to v1.9.10
• chore(deps): update dependency newtonsoft.json to v13

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• [x ] Question or discussion
• Bug
• Requirement
• Feature or performance improvement

Question

• What do you want to know?

Bug

• Which version of Canal-Server, OS ,Mysql?

• What happen?
  If possible, provide a way for reproducing the error. e.g. demo application.

Requirement or improvement

• Please describe about your requirements or improvement suggestions.

environment

• canal version docker lastest
• mysql version 8.0

Issue Description

在docker启动容器

sh run.sh -e canal.auto.scan=false   -e canal.destinations=test  -e canal.instance.master.address=192.168.13.86:3306   -e canal.instance.dbUsername=root   -e canal.instance.dbPassword=123456789    -e canal.instance.connectionCharset=UTF-8   -e canal.instance.tsdb.enable=true   -e canal.instance.gtidon=false  
docker run -d -it -h 192.168.13.45 -e canal.auto.scan=false -e canal.destinations=test -e canal.instance.master.address=192.168.13.86:3306 -e canal.instance.dbUsername=root -e canal.instance.dbPassword=123456789 -e canal.instance.connectionCharset=UTF-8 -e canal.instance.tsdb.enable=true -e canal.instance.gtidon=false --name=canal-server --net=host -p 8000:8000 -p 2222:2222 -p 11111:11111 -p 11112:11112 -m 4096m canal/canal-server

canal.server的启动日志

DOCKER_DEPLOY_TYPE=VM
==> INIT /alidata/init/02init-sshd.sh
==> EXIT CODE: 0
==> INIT /alidata/init/fix-hosts.py
==> EXIT CODE: 0
==> INIT DEFAULT
Generating SSH1 RSA host key:                              [  OK  ]
Starting sshd:                                             [  OK  ]
Starting crond:                                            [  OK  ]
==> INIT DONE
==> RUN /home/admin/app.sh
==> START ...
start canal ...
start canal successful
==> START SUCCESSFUL ...

C#客户端代码

使用的canalsharp的demo代码

static void Main(string[] args)
        {
            LogManager.Configuration = new XmlLoggingConfiguration("NLog.Config");
            CanalSharpLogManager.LoggerFactory.AddNLog();
            _logger = CanalSharpLogManager.LoggerFactory.CreateLogger("Program");
            var destination = "test";
            var connector = CanalConnectors.NewSingleConnector("192.168.13.45", 11111, destination, "", "");
            //连接 Canal
            connector.Connect();
            connector.Subscribe(".*\\..*");
            while (true)
            {
                var message = connector.Get(1024);
                var batchId = message.Id;
                if (batchId == -1 || message.Entries.Count <= 0)
                {
                    Thread.Sleep(300);
                    continue;
                }

                PrintEntry(message.Entries);
            }
        }

控制台结果

[14:20:24]:Canal connect success. IP: 192.168.13.45, Port: 11111
[14:20:24]:Subscribe success. Filter: .*\..*

Actual behaviour

看起来每个步骤都正常，但是操作语句update canal set id = 2，无法在客户端正常订阅。

在使用CanalSharp监听的时候,只能获取到增量数据,能否支持全量更新?先同步历史数据,然后再增量更新?

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• Question or discussion

Question

CanalSharp.Protocol.Exception.CanalClientException: failed to subscribe with reason: something goes wrong with channel:[id: 0x4d67f2df, /172.16.10.16:63064 => /172.16.10.16:11111], exception=com.alibaba.otter.canal.server.exception.CanalServerException: destination:database_dn1 should start first 在 CanalSharp.Client.Impl.SimpleCanalConnector.Subscribe(String filter) 在 GdkySyncService.canal_sync_job.DoSync(CancellationToken token, Bas_Datasource example) 位置 D:\projects\GDKY_Data_MS\GDKY_Data_MS\GdkySyncService\Jobs\Sync\canal_sync_job.cs:行号 154

经常出现该问题，需要删除meta.dat和h2.mv.db文件后，重启才能好，这是为啥呢？？？急急急

Bug

• canal1.1.3
  --mysql 5.7
  --windows server2012

• What happen?
  运行一段时间

Requirement or improvement

• Please describe about your requirements or improvement suggestions.

connector.Subscribe("db.table");
这句话不会覆盖掉服务端设置的默认filter，无法做到监听指定库指定表的binlog变化，必须在服务端的instance.properties里设置才有效

获取变更列的中文值是是乱码，canal（canal.deployer-1.1.4.tar.gz）服务端在centos 中运行，已经设置了canal.instance.connectionCharset = UTF-8。数据库也是utf8mb4。客户端在linux 和windows中获取的中文值均为乱码

Canal supports protobuf3.0, is there sample code? I'm a newbie, thanks!