whitesock / open-replicator

Open Replicator is a high performance MySQL binlog parser written in Java. It unfolds the possibilities that you can parse, filter and broadcast the binlog events in a real time manner.

open-replicator

Open Replicator is a high performance MySQL binlog parser written in Java. It unfolds the possibilities that you can parse, filter and broadcast the binlog events in a real time manner.

note

For MySQL 5.6.6 users, binlog_checksum system variable is NOT supported by open-replicator at the moment, please set it to NONE.

releases

1.0.7

release date: 2014-05-12
support signed tinyint, smallint, mediumint, int, bigint

1.0.6

release date: 2014-05-08
remove dependency commons-lang, log4j
support MYSQL_TYPE_TIMESTAMP2, MYSQL_TYPE_DATETIME2, MYSQL_TYPE_TIME2 

1.0.0

release date: 2011-12-29

maven

<dependency>
        <groupId>open-replicator</groupId>
        <artifactId>open-replicator</artifactId>
        <version>1.0.7</version>
</dependency>

parsers

BinlogEventParser is plugable. All available implementations are registered by default, but you can register only the parsers you are interested in.

usage

final OpenReplicator or = new OpenReplicator();
or.setUser("root");
or.setPassword("123456");
or.setHost("localhost");
or.setPort(3306);
or.setServerId(6789);
or.setBinlogPosition(4);
or.setBinlogFileName("mysql_bin.000001");
or.setBinlogEventListener(new BinlogEventListener() {
    public void onEvents(BinlogEventV4 event) {
        // your code goes here
    }
});
or.start();

System.out.println("press 'q' to stop");
final BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
for(String line = br.readLine(); line != null; line = br.readLine()) {
    if(line.equals("q")) {
        or.stop();
        break;
    }
}

open-replicator's Issues

in slave query event,loss datanbaseName

in master queryEvent：

client print：

QueryEvent[header=BinlogEventV4HeaderImpl[timestamp=1421203248000,eventType=2,serverId=100011084,eventLength=59,nextPosition=365191,flags=8,timestampOfReceipt=1421215901077],threadId=25786576,elapsedTime=28751,databaseNameLength=0,errorCode=0,statusVariablesLength=21,statusVariables=[QFlags2Code[flags=0], QSQLModeCode[sqlMode=524288], QCharsetCode[characterSetClient=33,collationConnection=33,collationServer=33]],databaseName=test,sql=BEGIN]

atabaseName is test

but in slave queryEvent：
client print：

QueryEvent[header=BinlogEventV4HeaderImpl[timestamp=1421203248000,eventType=2,serverId=100011084,eventLength=59,nextPosition=365191,flags=8,timestampOfReceipt=1421215901077],threadId=25786576,elapsedTime=28751,databaseNameLength=0,errorCode=0,statusVariablesLength=21,statusVariables=[QFlags2Code[flags=0], QSQLModeCode[sqlMode=524288], QCharsetCode[characterSetClient=33,collationConnection=33,collationServer=33]],databaseName=,sql=BEGIN]

databaseName is ''

why？
they binglog is same：

at 336

#150113 17:10:35 server id 1 end_log_pos 405 Query thread_id=5 exec_time=0 error_code=0

TimeZone when parsing dates and times

I noticed that dates are always parsed in TimeZone.getDefault(). I was going to add a configuration option that allows a user of OpenReplicator to set the TimeZone when parsing the binlog. Have you thought much about this? Do you think it's a good idea? I think it's needed but wanted to double check before submitting a pull request.

Also, I think it's necessary to always parse timestamps in UTC since MySQL converts from the local timezone to UTC an save and update.

Please let me know if you agree with my thoughts and I'll do the work.

How to deal with the type of longtext and date in the database

Longtext:BlobColumn
Date:GMT
Is there any better?

I can not think of it，could you help me?

About binlogFileName

In rotate event, i can obtain a correct value with "((RotateEvent)event).getBinlogFileName".
But after that, in the other event(include rotate event), i get a incorrect value with "event.getHeader().getBinlogFileName", i think when rotate event, only the value of RotateEvent instance has been changed, not the others?

MySQL5.6.25 parse update sql error

1. parse update sql of mysql5.6.21:
   update==[Pair[before=Row[columns=[102103020, 100000003, 2015-03-14, m.city, 102103, 130, 72, Sat Mar 14 19:36:46 CST 2015]],after=Row[columns=[102103020, 100000003, 2015-03-14, m.city, 102103, 131, 73, Sat Mar 14 19:38:01 CST 2015]]]]
2. but parse update sql of mysql5.6.25:
   update=[Pair[before=Row[columns=[-2216200267346]],after=Row[columns=[null, null]]]]
3. can not parse mysql5.6.25?,How to solve?

open-replicator-1.0.7.jar is missed in maven repositories

Adding the following text to my pom.xml causes "artifact missing" error:

<dependency>
        <groupId>open-replicator</groupId>
        <artifactId>open-replicator</artifactId>
        <version>1.0.7</version>
</dependency>

Search for "open-replicator" in http://search.maven.org/ returns nothing.

assertion failed, available: 4, event type: 19

.11:13:48.020 [binlog-parser-1] ERROR c.g.c.o.b.impl.AbstractBinlogParser - failed to parse binlog
java.lang.RuntimeException: assertion failed, available: 4, event type: 19
at com.google.code.or.binlog.impl.FileBasedBinlogParser.doParse(FileBasedBinlogParser.java:140) ~[open-replicator-1.0.7.jar:na]
at com.google.code.or.binlog.impl.AbstractBinlogParser$Task.run(AbstractBinlogParser.java:244) ~[open-replicator-1.0.7.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

Can you tell me how to fix this?

Runs into errors on MySQL 5.6.13

Perhaps unknown event types? Downgrading to 5.5.33 resolved this issue, which is fine for this application, but I thought I'd file an issue for future users.

assertion failed, available: 8, event type: 19

ava.lang.RuntimeException: assertion failed, available: 8, event type: 19
at com.google.code.or.binlog.impl.ReplicationBasedBinlogParser.doParse(ReplicationBasedBinlogParser.java:130)
at com.google.code.or.binlog.impl.AbstractBinlogParser$Task.run(AbstractBinlogParser.java:240)
at java.lang.Thread.run(Thread.java:745)
Can you tell me how to fix this?

I am using mysql8.0.21

Any plans to support MySQL 5.7?

We are currently evaluating this for MySQL 5.6, but most probably we will need to migrate to 5.7 version of MySQL. So, can anyone suggest if there is a plan to support the next version and what are the challenges in adding that support.

Is GTID based replication supported?

a error con't be solved

java.lang.RuntimeException: ErrorPacket[packetMarker=255,errorCode=1236,slash=#,sqlState=HY000,errorMessage=Slave can not handle replication events with the checksum that master is configured to log; the first event 'mysql-bin-log.000001' at 4, the last event read from 'D:\MySql\mysql-5.6.6-m9-winx64\log\mysql-bin-log.000001' at 120, the last byte read from 'D:\MySql\mysql-5.6.6-m9-winx64\log\mysql-bin-log.000001' at 120.]
at com.google.code.or.binlog.impl.ReplicationBasedBinlogParser.doParse(ReplicationBasedBinlogParser.java:100)
at com.google.code.or.binlog.impl.AbstractBinlogParser$Task.run(AbstractBinlogParser.java:244)
at java.lang.Thread.run(Thread.java:748)

this is my pom.xml


open-replicator
open-replicator
1.0.7


ch.qos.logback
logback-classic


ch.qos.logback
logback-core


ch.qos.logback
logback-core


org.slf4j
slf4j-api




org.codehaus.groovy
groovy-all
2.3.11


ch.qos.logback
logback-classic
1.2.3


com.google.code.gson
gson
2.8.1


mysql
mysql-connector-java
5.1.44

and coding just like your usage；
mysql version is 5.6.6
and my.ini config is
[mysql]

设置mysql客户端默认字符集

default-character-set=utf8
[mysqld]
#skip-grant-tables
#设置3306端口
port = 3306

设置mysql的安装目录

basedir=D:\MySql\mysql-5.6.6-m9-winx64

设置mysql数据库的数据的存放目录

datadir=D:\MySql\mysql-5.6.6-m9-winx64\data

log_bin=D:\MySql\mysql-5.6.6-m9-winx64\log\mysql-bin-log
server-id=1
binlog-format = row

允许最大连接数

max_connections=200

服务端使用的字符集默认为8比特编码的latin1字符集

character-set-server=utf8

创建新表时将使用的默认存储引擎

default-storage-engine=INNODB
collation-server=utf8_general_ci

TransportException: Proxy ERROR:do not support such type sql

What would normally happen to make such a mistake

datetime timestamp column does not work

mysql 5.6.28
ubuntu15.10 64bit
see zendesk/maxwell#229 for more information.
what i want to is to keep the orignal string "'2015-08-26 17:44:55'" and add a config property to specify the timezone UTC or CST

should i edit "MySQLConstants.java" and "Timestamp2Column.java TimestampColumn.java Datetime2Column.java DatetimeColumn.java" or edit "AbstractRowEventParser.java just like the way in https://github.com/zendesk/open-replicator/blob/master/src/main/java/com/google/code/or/common/util/MySQLUtils.java

could you explain a little

mysql connection never closed

Hi, when i run it on dev, i kill the process, the program had finished, but on mysql server, the connection has been connected.
show processlist:
'1', 'wuysh', '127.0.0.1:61244', 'xxxx', 'Query', '0', 'init', 'show processlist'
'2', 'wuysh', '127.0.0.1:61245', 'xxxx', 'Sleep', '3', '', NULL
'5', 'wuysh', '127.0.0.1:61383', NULL, 'Binlog Dump', '633', 'Master has sent all binlog to slave; waiting for binlog to be updated', NULL
'6', 'wuysh', '127.0.0.1:61542', NULL, 'Binlog Dump', '510', 'Master has sent all binlog to slave; waiting for binlog to be updated', NULL
'7', 'wuysh', '127.0.0.1:61839', NULL, 'Binlog Dump', '326', 'Master has sent all binlog to slave; waiting for binlog to be updated', NULL
'8', 'wuysh', '127.0.0.1:62190', NULL, 'Binlog Dump', '88', 'Master has sent all binlog to slave; waiting for binlog to be updated', NULL

i had to kill the process manually. is there any other way to solve this program ?

Why returning nagetive position when I set binlogPosition greater than 4?

code: return first event by the specific position

OpenReplicator or = new OpenReplicator();
or.setHost(host);
or.setPort(port);
or.setUser(user);
or.setPassword(password);
or.setBinlogPosition(binlogStartPos); // greater than 4
or.setBinlogFileName(binlogFileName);
or.setBinlogEventListener(new BinlogEventListener() {
@OverRide
public void onEvents(BinlogEventV4 event) {
list.add(event);
// notify external thread to stop openReplicator
}
});
or.stop(xxx);
return list.get(0);

Then, I get binlog event like this:
event pos=-116 <--- first event
event timestamp=1517369647000
event pos=8808036 <--- second one is correct
event timestamp=1522235789000