Verify explicitly synced data is recoverable about rocksdb

[jacobxli-xx]

Hey, Andrew.
The answer for the both questions are yes.
For both explicit flush and auto flush, we are going to limit the scope of verification to recover all writes till the maximum sequence number of write recorded before onFlushCompleted() (test cases limited to only 1 column family).

[jacobxli-xx]

hi, ajkr@ and hx235@.
Glad to take this task : )

[ajkr]

because when we do explicit flush on specific cf, it's not guaranteed that all cfs will be flushed, so the maximum of all seqno is not guaranteed to be equal to the maximum seqno of the cf we flushed.

It is guaranteed in the case where it is needed (atomic_flush=true) -

rocksdb/db_stress_tool/db_stress_test_base.cc

Lines 2266 to 2268 in 3ef9092

	if (FLAGS_atomic_flush) {
	return db_->Flush(flush_opts, column_families_);
	}

It is not guaranteed when WAL is enabled and atomic_flush is disabled, but there it is not needed because WAL ensures consistent recovery across column families.

WAL-disabled, atomic_flush-disabled may be worth testing at some future point but that is still an open topic to discuss. My thoughts are here - #11841.

Another case that may be worth testing at some point is WAL-disabled, atomic_flush-enabled, and manual flush triggered on a subset of column families (what you suggested we are already doing). Still, it's not something we decided to do yet, so adding extra complexity assuming we will do it feels premature

[hx235]

In case it's not already obvious, for WAL disabled case, we will only consider with atomic_flush=true. For a flush, we do not need to verify recoverability of writes concurrent to that flush as there is no guarantee from RocksDB on whether that concurrent writes are included in that flush or not.

[ajkr]

@jacobxli-xx Thank you, it is assigned now.

[ajkr]

@jacobxli-xx Here are some questions I had about what you are planning to do:

1. For explicit sync mechanisms, will the scope be limited to verifying writes that completed before the explicit sync are recovered?
2. For implicit sync mechanisms (i.e., automatic flush), will the scope be limited to verifying writes that are persisted according to OnFlushCompleted()?

If any answers are no, we'll probably want to know the reason and eventually the plan to make a guess of whether we'd be on board with it.

[hx235]

@jacobxli-xx If you are considering storing and m-mapping map<std::string, uint64_t> cf_flushed_seqno, I wonder why can't we just store and mmap the maximum of all the seqnos in the map?

[jacobxli-xx]

@hx235 The m-mapping map<std::string, uint64_t> cf_flushed_seqno stores the cf name and the corresponding maximum flushed seqno of the specific cf. Why we do not only store the maximum of all the seqnos is because when we do explicit flush on specific cf, it's not guaranteed that all cfs will be flushed, so the maximum of all seqno is not guaranteed to be equal to the maximum seqno of the cf we flushed.

Although our test cases are limited to 1 cf, we use this map for forward compatibility as it can be used in multiple cfs cases.