copydb Apply event Is not Autocommit ? about ghostferry

[shuhaowu]

Hey. Can you post a log of the run and what replication event you found that was not replicated to the target?

[ymakedaq]

@shuhaowu

Here Is My Copydb Test Config, Source Mysql version perconae 5.6.23, Target 5.7.19 (gtid is ON)

`{
"Source": {
"Host": "ip2",
"Port": 3309,
"User": "root",
"Pass": "123456",
"Collation": "utf8_general_ci",
"Params": {
"charset": "utf8"
}
},

"Target": {
"Host": "ip1",
"Port": 3306,
"User": "root",
"Pass": "123456",
"Collation": "utf8_general_ci",
"Params": {
"charset": "utf8"
}
},
"Databases": {
"Whitelist": ["make"],
"Rewrites": {"make":"tmake"}
},
"VerifierType": "ChecksumTable"
}`

My Source Table before copy :

select count(*) from  t1;
+----------+
| count(*) |
+----------+
|    22238 |
+----------+

At Copying.... I delete some row at source table ...

delete from  make.t1  limit 100;
Query OK, 100 rows affected (0.01 sec)

But The Target t1 Table count Is:

root@tmake 09:02> select count(*) from  t1;
+----------+
| count(*) |
+----------+
|    22238 |
+----------+

I Consider Target Table count should be 22138 ~

Run Log & Not Cut over

INFO[0000] connecting to the source database             dsn="root:<masked>@tcp(ip1:3309)/?multiStatements=true&charset=utf8&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27" tag=ferry
INFO[0000] connected to root:<masked>@tcp(ip1:3309)/?multiStatements=true&charset=utf8&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27  dsn="root:<masked>@tcp(ip1:3309)/?multiStatements=true&charset=utf8&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27" hasSSL=false ssl_cipher= tag=ferry
INFO[0000] connecting to the target database             dsn="root:<masked>@tcp(ip2:3306)/?multiStatements=true&charset=utf8&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27" tag=ferry
INFO[0000] connected to root:<masked>@tcp(ip2:3306)/?multiStatements=true&charset=utf8&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27  dsn="root:<masked>@tcp(ip2:3306)/?multiStatements=true&charset=utf8&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27" hasSSL=false ssl_cipher= tag=ferry
INFO[0000] ferry initialized                             tag=ferry
INFO[0000] initializing                                  tag=control_server
INFO[0000] create BinlogSyncer with config {99399  ip1 3309 root    false false <nil> false true 0 0s 0s} 
INFO[0000] reading current binlog position               tag=binlog_streamer
INFO[0000] found binlog position, starting synchronization  file=mysql-bin.005942 pos=83047109 tag=binlog_streamer
INFO[0000] begin to sync binlog from position (mysql-bin.005942, 83047109) 
INFO[0000] register slave for master server ip1:3309 
INFO[0000] rotate to (mysql-bin.005942, 83047109)       
INFO[0000] table schemas cached                          tables="[make.t1]" tag=table_schema_cache
INFO[0000] creating databases and tables on target      
INFO[0000] running on 0.0.0.0:8000                       tag=control_server
INFO[0000] starting ferry run                            tag=ferry
INFO[0000] starting data iterator run                    tablesCount=1 tag=data_iterator
INFO[0000] starting binlog streamer                      tag=binlog_streamer
INFO[0000] rotated binlog file                           file=mysql-bin.005942 pos=83047109 tag=binlog_streamer
INFO[0000] done queueing tables to be iterated, closing table channel  tag=data_iterator
INFO[0017] finished iterations                           tag=ferry```

[ymakedaq]

And I Print apply sql At binlog_writer.go, But It's generated sql And print ok after b.DB.Exec(query)

[shuhaowu]

What's your schema like? Do you have some example data for this rows? This can possibly help us build a test case.

[ymakedaq]

I found the problem，
Source Table have Cloumn sa_back varchar(150)
The Vaule is

root@make 18:44> select sa_back  from  make.t1 limit 1;
+---------+
| sa_back |
+---------+
|        |
+---------+
1 row in set (0.00 sec)

root@make 18:44> select hex(sa_back)  from  make.t1 limit 1;
+--------------+
| hex(sa_back) |
+--------------+
| 1800         |
+--------------+
1 row in set (0.01 sec)

It's not a null value,

Lead to can't find that line when you delete it.

[ymakedaq]

@shuhaowu How to Deal it?

[shuhaowu]

I can't reproduce this issue using the docker compose servers that we spawn in local development (see docker-compose.yml for details). Are you sure your charset/collation settings are correct on both servers?

Specifically, this is what I did:

1. On source (localhost:29291):

mysql> create database abc;
Query OK, 1 row affected (0.01 sec)

mysql> use abc;
Database changed
mysql> create table test1 (id bigint(20), s varchar(150), primary key (id));
Query OK, 0 rows affected (0.04 sec)

mysql> show create table test1;
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                                                           |
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| test1 | CREATE TABLE `test1` (
  `id` bigint(20) NOT NULL,
  `s` varchar(150) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci |
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> insert into test1 values (1, UNHEX(1800));
Query OK, 1 row affected (0.00 sec)

mysql> select * from test1;
+----+------+
| id | s    |
+----+------+
|  1 | �    |
+----+------+
1 row in set (0.00 sec)

mysql> select HEX(s) from test1;
+--------+
| HEX(s) |
+--------+
| 1800   |
+--------+
1 row in set (0.00 sec)

2. Ran ghostferry-copydb with a patch that prints the binlog writer query to console. This simply copies the table abc.test1:

INFO[0000] hello world from 1.1.0+20180523150145+a8ba59c+dirty  tag=ferry
INFO[0000] connecting to the source database             dsn="root:<masked>@tcp(127.0.0.1:29291)/?collation=utf8mb4_unicode_ci&multiStatements=true&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27&charset=utf8mb4" tag=ferry
INFO[0000] connected to root:<masked>@tcp(127.0.0.1:29291)/?collation=utf8mb4_unicode_ci&multiStatements=true&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27&charset=utf8mb4  dsn="root:<masked>@tcp(127.0.0.1:29291)/?collation=utf8mb4_unicode_ci&multiStatements=true&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27&charset=utf8mb4" hasSSL=false ssl_cipher= tag=ferry
INFO[0000] connecting to the target database             dsn="root:<masked>@tcp(127.0.0.1:29292)/?collation=utf8mb4_unicode_ci&multiStatements=true&charset=utf8mb4&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27" tag=ferry
INFO[0000] connected to root:<masked>@tcp(127.0.0.1:29292)/?collation=utf8mb4_unicode_ci&multiStatements=true&charset=utf8mb4&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27  dsn="root:<masked>@tcp(127.0.0.1:29292)/?collation=utf8mb4_unicode_ci&multiStatements=true&charset=utf8mb4&time_zone=%27%2B00%3A00%27&sql_mode=%27STRICT_ALL_TABLES%2CNO_BACKSLASH_ESCAPES%27" hasSSL=false ssl_cipher= tag=ferry
INFO[0000] ferry initialized                             tag=ferry
INFO[0000] initializing                                  tag=control_server
INFO[0000] create BinlogSyncer with config {99399  127.0.0.1 29291 root    false false <nil> false true 0 0s 0s} 
INFO[0000] reading current binlog position               tag=binlog_streamer
INFO[0000] found binlog position, starting synchronization  file=mysql-bin.000003 pos=1350 tag=binlog_streamer
INFO[0000] begin to sync binlog from position (mysql-bin.000003, 1350) 
INFO[0000] register slave for master server 127.0.0.1:29291 
INFO[0000] rotate to (mysql-bin.000003, 1350)           
DEBU[0000] loading tables from database                  database=abc tag=table_schema_cache
DEBU[0000] fetching table schema                         database=abc table=test1 tag=table_schema_cache
DEBU[0000] caching table schema                          database=abc table=test1 tag=table_schema_cache
INFO[0000] table schemas cached                          tables="[abc.test1]" tag=table_schema_cache
INFO[0000] creating databases and tables on target      
INFO[0000] running on 0.0.0.0:8000                       tag=control_server
INFO[0000] starting ferry run                            tag=ferry
INFO[0000] starting data iterator run                    tablesCount=1 tag=data_iterator
INFO[0000] starting binlog streamer                      tag=binlog_streamer
INFO[0000] rotated binlog file                           file=mysql-bin.000003 pos=1350 tag=binlog_streamer
INFO[0000] done queueing tables to be iterated, closing table channel  tag=data_iterator
DEBU[0000] found 1 rows                                  args="[0]" sql="SELECT * FROM `abc`.`test1` WHERE `id` > ? ORDER BY `id` LIMIT 200 FOR UPDATE" table=abc.test1 tag=cursor
DEBU[0000] updated last successful PK to 1               table=abc.test1 tag=data_iterator
DEBU[0000] table iteration completed                     table=abc.test1 tag=data_iterator
INFO[0000] finished iterations                           tag=ferry

3. Deleted the row from the source database:

mysql> delete from abc.test1;
Query OK, 1 row affected (0.01 sec)

4. Observe it was executed by Ghostferry via the patch

DEBU[0024] received event *ghostferry.BinlogDeleteEvent at 2018-05-23 15:02:28 +0000 UTC  database=abc table=test1 tag=binlog_streamer
DEBU[0024] BEGIN;
DELETE FROM `abc`.`test1` WHERE `id`=1 AND `s`='�';
COMMIT 

5. Verified it was deleted from the target server via both the checksum table verifier and a query (localhost:29292)

INFO[0040] checksum table verification started           tag=checksum_verifier
INFO[0040] served http request                           method=POST path=/api/actions/verify tag=control_server time="68.008µs"
INFO[0040] checking table                                sourceTable="`abc`.`test1`" tag=checksum_verifier targetTable="`abc`.`test1`"
INFO[0040] tables on source and target verified to match  sourceChecksum=0 sourceTable="`abc`.`test1`" tag=checksum_verifier targetChecksum=0 targetTable="`abc`.`test1`"


mysql> select * from test1;
Empty set (0.00 sec)

[ymakedaq]

I'm print the values in code:

table stuct:

      Create Table: CREATE TABLE `tbo` (
        `sa_id` int(11) unsigned NOT NULL AUTO_INCREMENT COMMENT '主键',
         `sa_back` char(150) NOT NULL DEFAULT '' ,
         `sa_back_img` char(150) NOT NULL DEFAULT '' ,
          `sa_isinformation` tinyint(1) NOT NULL DEFAULT '0' ,
          `sa_is_add` char(255) NOT NULL DEFAULT '' ,
          `sa_is_advance` tinyint(1) unsigned NOT NULL DEFAULT '1' ,
          PRIMARY KEY (`sa_id`)
         ) ENGINE=InnoDB ;

---

pirnt code:

    func buildStringMapForWhere(columns []string, values []interface{}) string {
       var buffer []byte
       var tvalue []byte
      fmt.Println("make debug ++++++++++++++++++++++++++++++++++++++")
       for idx, v := range values {
	  fmt.Println(idx,v)
      	}	.....

---

runing ferry-copydb to migrate data..

on the source db execute:
delete from tbo limit 1;

ferry-copydb stdout:

 make` debug ++++++++++++++++++++++++++++++++++++++
 0 2018195
 1 
 2 
 3 0
 4 5 32
 //This is Apply sql  build  from  binlog event
`DELETE` FROM `tmake`.`tbo` WHERE `sa_id`=2018195 AND `sa_back`='' AND `sa_back_img`='' AND `sa_isinformation`=0 AND `sa_is_add`='AND`sa_is_advance`=32;`

---

and you can get sa_is_add='AND is error;

[ymakedaq]

	fmt.Println("make debug ++++++++++++++++++++++++++++++++++++++")
	for idx, v := range values {
		fmt.Println(idx,v)
		if  reflect.TypeOf(v).Name() ==  "string" {
			fmt.Println([]byte(v.(string)))
		}
	}	

delete from tbo limit 1;

Print Log:

---

make debug ++++++++++++++++++++++++++++++++++++++
0 2018253
1 
[24 0]
2 
[]
3 0
4  [128 128 48]
5 32
make debug ===================================================
BEGIN;
DELETE FROM `tmake`.`tbo` WHERE `sa_id`=2018253 AND `sa_back`='' AND `sa_back_img`='' AND `sa_isinformation`=0 AND `sa_is_add`=' AND `sa_is_advance`=32;
COMMIT

[shuhaowu]

Is your data actually just [128 128 48]? The code point 128 doesn't seem to be a printable character.

Please post a complete example with reproducing steps (a sql file to initialize the database would help) because I'm not sure how to try for this case.

[ymakedaq]

@shuhaowu Data generated in the test environment，It's No Problem when I Test another table the data from insert into tbt select * from tbo ...

I Can't Generate Problem data at mysql shell,

CREATE TABLE `tb2` (
  `sa_id` int(11) unsigned NOT NULL AUTO_INCREMENT COMMENT '主键',
  `sa_is_add` char(255) NOT NULL DEFAULT ''，
  `sa_is_advance` tinyint(1) unsigned NOT NULL DEFAULT '1'，
  PRIMARY KEY (`sa_id`)
) ENGINE=InnoDB AUTO_INCREMENT=2018740 DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC 

mysqldump values:

LOCK TABLES `tb2` WRITE;
/*!40000 ALTER TABLE `tb2` DISABLE KEYS */;
INSERT INTO `tb2` VALUES (2018255,',32),(2018256,',32),(2018310,'32),(2018311,'32),(2018348,'32),(2018400,',32),(2018639,',32),(2018642,'32),(2018671,'32),(2018739,'32);
/*!40000 ALTER TABLE `tb2` ENABLE KEYS */;
UNLOCK TABLES;
/*!50112 SET @disable_bulk_load = IF (@is_rocksdb_supported, 'SET SESSION rocksdb_bulk_load = @old_rocksdb_bulk_load', 'SET @dummy_rocksdb_bulk_load = 0') */;

And you load it,just ok

[shuhaowu]

I assume this is no longer a problem.