pkg/ccl/backupccl/backuprand/backuprand_test: TestBackupRestoreRandomDataRoundtrips failed about cockroach

[msbutler]

@RaduBerinde We're still observing empty copy spans on 24.1. Is this a new bug?

[RaduBerinde]

Yeah, all known bugs have been fixed, so we must have missed something.. We are sure that the test never causes external ingestions where there are no keys in the span?

[RaduBerinde]

@msbutler I have found that the test attempts an empty ingestion. I ran under stress against cockroachdb/pebble#3506 and I got:

* panic: external ingestion is empty: data/959416941262372865.sst ("\xf6w\x8a\x00" "\xf6w\x8b\x00" inclusive=false)

This means that there are no keys in that sst within the bound.

[RaduBerinde]

Here is an LSM that triggers the error. Tables 40 and 7 have no keys inside.

[msbutler]

aha! thanks for digging into this. I'll look into why this could happen.

[msbutler]

I stressed this test on top of a patch that panics in backup if it creates a backup file without any point keys, and no panic occurred during the repro, suggesting the source of the empty download span is not due to backup.

[RaduBerinde]

The physical file can have keys, the question is whether the portion that is being ingested (StartKey to EndKey) has any keys.

[msbutler]

Ah, to clarify: in DR land, a physical backup sst is partitioned into several backup files (i.e. roughly the export request responses). In Online restore, we create virtual sst's for each backup file. The terminology is really confusing and we should probably come up with a new term for backup file.

My patch above was verifying that no backup file was empty during the test failure.

[RaduBerinde]

The panic above shows that there is an external ingestion with no keys between the bounds. OR is the only one that does external ingestions. Maybe the bounds get screwed up somehow?

[RaduBerinde]

I can extend the patch to show all the keys in the physical file, that should give us some clue.

[msbutler]

Ah, that LSM visualizer is really helpful. I think understand how this could happen. Suppose table 120 has a secondary index, restore will partition the table's key space into two units of work (a restoreSpanEntry): [Table/120/1-Table/120/2),[Table/120/2-Table/120/3), but what if that second index is empty?

On the backup side, we very well could create a non empty backup file that spans the two indexes: [Table/120/1-Table/120/3) -- we construct backup units of work on the table level-- but restore we create units of work on the table-index level and will break up the table wide span for virtual sst ingestion here. Given all of this, we'll create a virtual sst that has no keys: [Table/120/2-Table/120/3).

To reproduce, I'll try to backup and online restore a table with an empty secondary index.

[msbutler]

here's the schema of the backed up table during a failing test case. i wonder if we can induce this via a partial index:

I240412 14:25:02.267796 906 5@util/log/event_log.go:32 â<8b>® [T1,Vsystem,n1,client=127.0.0.1:41058,hostssl,user=root] 127 ={"Timestamp":1712931902263077505,"EventType":"create_table","Statement":"CREATE TABLE â<80>¹randâ<80>º.public.â<80>¹t    able_1â<80>º (â<80>¹col1_0â<80>º BOX2D NULL, â<80>¹\"col1|_1\"â<80>º BOOL NOT NULL, â<80>¹col1_2â<80>º REGCLASS NULL, â<80>¹col1_3â<80>º JSONB NOT NULL, â<80>¹col1_4â<80>º VARCHAR NOT NULL, PRIMARY KEY (â<80>¹col1_4â<80>º, â<80>¹\"col1|_1\"â    <80>º ASC), INDEX (â<80>¹col1_0â<80>º ASC), INDEX (â<80>¹crdb_internal_idx_exprâ<80>º DESC), INDEX (â<80>¹col1_3â<80>º DESC) STORING (â<80>¹col1_0â<80>º), UNIQUE (â<80>¹\"col1|_1\"â<80>º, â<80>¹col1_4â<80>º DESC) STORING (â<80>¹col1_0â<80>º,     â<80>¹col1_3â<80>º) WHERE â<80>¹table_1â<80>º.â<80>¹\"col1|_1\"â<80>º, INDEX (â<80>¹\"col1|_1\"â<80>º) WHERE â<80>¹table_1â<80>º.â<80>¹col1_4â<80>º >= â<80>¹'\"'â<80>º:::STRING)","Tag":"CREATE TABLE","User":"root","DescriptorID":106,"TableN    ame":"â<80>¹rand.public.table_1â<80>º"}

[msbutler]

Alrighty, I think my above theory holds: if I patch restore to create units of work on the table level, I cannot reproduce the empty download span bug. Even better, if i patch backup to create work on the table-index level, I cannot hit the bug either.

I still haven't figured out how to easily repro this bug. For example, I cant get this unit test to fail under stess, even though it contains an empty index. Some how in mvcc export request, we generally avoid returning an export response that spans the empty index span.

I also am unsure how we should fix this. Some candidates:

• change backup work units to the table-index level. we currently merge index spans as a perf hack, but backup is O(rows) not O(# indexes), as I've done with my backup patch linked above.
• relax the pebble invariant that we can't download empty file spans
• I don't think we can change restore's unit of work to the table level, as i think that violates some of our logic around introduced and dropped index spans blah blah blah.

[cockroach-teamcity]

pkg/ccl/backupccl/backuprand/backuprand_test.TestBackupRestoreRandomDataRoundtrips failed on release-24.1 @ 53fcac10e573f8b3f0f6729fb7840f5ffba400b3:

=== RUN   TestBackupRestoreRandomDataRoundtrips
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestBackupRestoreRandomDataRoundtrips2167932939
    test_log_scope.go:81: use -show-logs to present logs inline
    jobs_verification.go:98: job failed: failed to generate and send download spans: failed to download spans on 1 nodes; n1 returned error requesting download spans from node 1 (NODE_STATUS_LIVE): rpc error: code = Unknown desc = CopySpan cannot copy empty span \x91\x00#10035,DELSIZED \x92\x00#72057594037927935,RANGEDEL
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestBackupRestoreRandomDataRoundtrips2167932939
--- FAIL: TestBackupRestoreRandomDataRoundtrips (29.79s)

Parameters:

• attempt=1
• run=3
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• #122341 pkg/ccl/backupccl/backuprand/backuprand_test: TestBackupRestoreRandomDataRoundtrips failed [C-test-failure O-robot P-0 T-disaster-recovery branch-master release-blocker]

This test on roachdash | Improve this report!

[msbutler]

closed via #122679