Race condition in tests causing intermittent failures about ghostferry

[shuhaowu]

In data_iterator_test.go, SetupTest adds a batch listener on the DataIterator that appends items in a map of lists (this.receivedRows). This can cause panics due to concurrent map write. See this example: https://travis-ci.com/Shopify/ghostferry/builds/116171401

To fix this, either add a mutex or use a SyncMap.

[shuhaowu]

In binlog_streamer.go, the logic maintaining the binlog streamer main loop states: s.lastStreamedBinlogPosition.Compare(s.targetBinlogPosition) < 0. In tests, the binlog is not advanced very often. There's a chance that this comparison will stay at 0, causing the BinlogStreamer to never quit in the tests.

We should be able to change this condition to <= 0. Before we do so, we need to check if this won't cause corruption.

[shuhaowu]

TestThrottlerMultipleServerIDs could be racy as it checks if the lag throttler throttles by setting a lag in the database and sleeping for 10ms.

[shuhaowu]

From #6:

In ghostferry.tests.TestIgnoresTables:

--- FAIL: TestIgnoresTables (1.70s)
	Error Trace:	iterative_verifier_integration_test.go:139
			integration_test_case.go:158
			integration_test_case.go:105
			integration_test_case.go:42
			integration_test_case.go:33
			iterative_verifier_integration_test.go:153
	Error:		Should be true

Some deadlocking issues:

ERRO[0006] failed to write events to target, 1 of 5 max retries  error="exec query (445 bytes): Error 1213: Deadlock found when trying to get lock; try restarting transaction" tag=binlog_writer
ERRO[0006] failed to write events to target, 2 of 5 max retries  error="exec query (445 bytes): Error 1213: Deadlock found when trying to get lock; try restarting transaction" tag=binlog_writer
ERRO[0006] failed to write events to target, 3 of 5 max retries  error="exec query (445 bytes): Error 1213: Deadlock found when trying to get lock; try restarting transaction" tag=binlog_writer
ERRO[0006] failed to write events to target, 4 of 5 max retries  error="exec query (445 bytes): Error 1213: Deadlock found when trying to get lock; try restarting transaction" tag=binlog_writer
ERRO[0006] failed to write events to target after 5 attempts, retry limit exceeded  error="exec query (445 bytes): Error 1213: Deadlock found when trying to get lock; try restarting transaction" tag=binlog_writer
ERRO[0006] fatal error detected, state dump coming in stdout  errfrom=binlog_writer error="exec query (445 bytes): Error 1213: Deadlock found when trying to get lock; try restarting transaction" tag=error_handler
{
  "CompletedTables": {},
  "LastSuccessfulBinlogPos": {
    "Name": "mysql-bin.000003",
    "Pos": 852531
  },
  "LastSuccessfulPrimaryKeys": {
    "gftest.table1": 459
  }
}
panic: fatal error detected, see logs for details

goroutine 1624 [running]:
github.com/Shopify/ghostferry.(*PanicErrorHandler).Fatal(0xc422171090, 0x8d27a3, 0xd, 0xb502a0, 0xc422fa6140)
	/home/ubuntu/.go_project/src/github.com/Shopify/ghostferry/error_handler.go:44 +0x70a
github.com/Shopify/ghostferry.(*BinlogWriter).Run(0xc420220af0)
	/home/ubuntu/.go_project/src/github.com/Shopify/ghostferry/binlog_writer.go:60 +0x37d
github.com/Shopify/ghostferry.(*Ferry).Run.func4(0xc422e91ba0, 0xc420492b40)
	/home/ubuntu/.go_project/src/github.com/Shopify/ghostferry/ferry.go:288 +0x55
created by github.com/Shopify/ghostferry.(*Ferry).Run
	/home/ubuntu/.go_project/src/github.com/Shopify/ghostferry/ferry.go:286 +0x249
exit status 2
FAIL	github.com/Shopify/ghostferry/test	6.349s

[shuhaowu]

=== FAIL: sharding/test TestSelectiveCopyDataWithInsertLoadOnAllTenants (1.21s)

    TestSelectiveCopyDataWithInsertLoadOnAllTenants: trivial_integration_test.go:109: 

        	Error Trace:	trivial_integration_test.go:109

        	Error:      	Should be true

        	Test:       	TestSelectiveCopyDataWithInsertLoadOnAllTenants

[shuhaowu]

  test_interrupt_resume_inline_verifier_with_datawriter           FAIL (1.06s)
        Expected nil to not be nil.
        /home/runner/work/ghostferry/ghostferry/test/integration/interrupt_resume_test.rb:149:in `test_interrupt_resume_inline_verifier_with_datawriter'
        test/main.rb:34:in `<main>'

[shuhaowu]

  test_resume_from_failure_with_state_callback                   ERROR (1.08s)
Minitest::UnexpectedError:         RuntimeError: Ghostferry did not fail
            /home/runner/work/ghostferry/ghostferry/test/helpers/ghostferry_helper.rb:132:in `run_expecting_failure'
            /home/runner/work/ghostferry/ghostferry/test/integration/interrupt_resume_test.rb:530:in `test_resume_from_failure_with_state_callback'