Trouble with virtual generated columns about ghostferry

[shuhaowu]

You might be able to ignore the column in this function by removing the virtual columns from TableSchema.Table.Columns:

ghostferry/table_schema_cache.go

Line 205 in b3aaacf

for _, tableSchema := range tableSchemas {

[SpencerMalone]

That's a good thought, lemme pull on that thread locally and check back in!

[SpencerMalone]

Sorry, I'm still struggling with this a bit, but hopefully will have an update in a few weeks!

[SpencerMalone]

Ignoring in the table schema ended up being a bit of a dead end, it made the DML stuff pretty unhappy, in the end we settled with maintaining a ColumnsToSkip list in type TableSchema struct, and using that to rebuild ColumnsToSelect in the cursor, and simply skipping the rows in the DML events. Would y'all be interested in a PR?

[milanatshopify]

Would y'all be interested in a PR?

Always interested in PRs - did you have some work in progress?

[SpencerMalone]

I can try to untangle our change to get it into the upstream, we've had it in production for ~a year now, but our fork is so divergent it may not happen at this point D: