Detect missing records in WAL recovery about rocksdb

[cbi42]

The second approach sounds cleaner since it does not require consecutive sequence numbers in consecutive WALs. We may still need to put some special value in the marker record to tell it apart from an old marker record in a reused WAL.

[ajkr]

I guess we can use the recyclable record format for this marker record - https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log-File-Format#the-recyclable-record-format. It has a log number that we can match against the log number in the filename.

[cbi42]

It feels like we would need a new record type at the start anyways to indicate that.

Makes sense.

For the dummy record, if it contains the size of the previous WAL file, it seems this feature provides a better protection compared to track_and_verify_wals_in_manifest for non-active WALs.

[ajkr]

Another way could be to append a marker record to the active WAL before switching it. Recovery would only continue to recover a newer WAL after seeing such a marker record. Unlike the existing code for writing dummy entries, which is constrained to writing to the new WAL as it is part of DB open, the new code should be able to mutate a WAL right before switching over to a new one.

[ajkr]

The second approach sounds cleaner since it does not require consecutive sequence numbers in consecutive WALs.

For the first approach, a dummy record with a new type at the start of the WAL could require recovery to have reached a specific sequence number before continuing. The feature would be opt-in to not break downgrade compatibility.

For the second approach, how would we know that an existing WAL should end with an ending marker, as the WAL format is unversioned? It feels like we would need a new record type at the start anyways to indicate that. It may be worthwhile if we make this new record type carry a version number.

[nkg-]

@ajkr : Another case, where some write is not fsynced to file, and missing completely from the file. That should cause missing records from WAL. Would that be caught today. Atleast I have seen (based on corruption logs), we check for crc checksum per record, and truncated records (which are a result of partial writes).