Data corruption in binary and varbinary columns about ghostferry

[shuhaowu]

Can you post an example dataset and query so we can add an integration test?

[shuhaowu]

Does the InlineVerifier catch this corruption? It should, right?

[kolbitsch-lastline]

Can you post an example dataset and query so we can add an integration test?

should be reproducable with literally any update of a BINARY column type that does uses the full length of the column but happens to have 0-bytes at the end. I can work on writing an integration test in the project - I have a patch that should fix it and can commit it together with the change

Unfortunately it looks like the replication library cannot know of the truncation (unless I'm misreading the docs, the mysql replication protocol doesn't transmit the max-size of a column update), so we'll need to extend the schema extraction and handle the padding in ghostferry (but it seems really simple).

Does the InlineVerifier catch this corruption? It should, right?

I am not overly familiar with this part yet, but from what I have learned so far, I would assume that it does

[kolbitsch-lastline]

ok, sent you 2 pull requests - the first one is to illustrate the problem, the second one contains a potential fix - but the second one should probably not be merged as-is, as I'm actually changing the vendor'ed module to show what I'd propose changing

let's discuss the details there

[kolbitsch-lastline]

FYI: tracking the upstream vendor module change here:

go-mysql-org/go-mysql#480

let's see what's the feedback we get there

NOTE: To increase chances to get it into upstream, I've made the change a bit more generic (added not only the FixedSize but also the MaxSize property). If that gets merged in the current PR version, we'll need to trivially update the ghostferry code when pulling in latest master (but we'll have to change other things to work with upstream master anyways)