kv/kvserver: TestLeaseQueueProactiveEnqueueOnPreferences failed about cockroach

[kvoli]

The lease hits this unexpected error when attempting to switch the lease type in pre-processing by the lease queue:

E240525 08:51:03.323593 6990355 kv/kvserver/queue.go:1202 ⋮ [T1,Vsystem,n3,lease,s3,r69/3:‹/{Table/Max-Max}›] 415  expiration of liveness record liveness(nid:3 epo:1 exp:1716627069.258107621,0) is not greater than expiration of the previous lease repl=(n3,s3):3 seq=2 start=1716627063.286141249,0 exp=1716627069.285657100,0 pro=1716627063.285657100,0 after liveness heartbeat

The lease does actually transfer later, right before the test shuts down the cluster:

I240525 08:51:48.622312 8480724 13@kv/kvserver/lease_queue.go:143 ⋮ [T1,Vsystem,n2,lease,s2,r69/2:‹/{Table/Max-Max}›] 1152  transferring lease to s1 usage=[batches/s=0.0 request_cpu/s=0µs raft_cpu/s=0µs write(keys)/s=0.0 write(bytes)/s=0 B read(keys)/s=0.0 read(bytes)/s=0 B]

This is potentially related to #124223, which merged two weeks prior.

[kvoli]

Actually, the error is from:

cockroach/pkg/kv/kvserver/replica_range_lease.go

Lines 553 to 554 in a7aad51

	return errors.AssertionFailedf("expiration of liveness record %s is not greater than "+
	"expiration of the previous lease %s after liveness heartbeat", l, status.Lease)

Which was was added in #123442. Not having looked much deeper, this appears like a case we may have permitted previously but do not now. cc @nvanbenschoten.

[cockroach-teamcity]

kv/kvserver.TestLeaseQueueProactiveEnqueueOnPreferences failed on master @ bd4c5fd7c65c0a9afa8ba05675e49747a4f68881:

=== RUN   TestLeaseQueueProactiveEnqueueOnPreferences
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLeaseQueueProactiveEnqueueOnPreferences2520501900
    test_log_scope.go:81: use -show-logs to present logs inline
    lease_queue_test.go:410: -- test log scope end --
test logs left over in: outputs.zip/logTestLeaseQueueProactiveEnqueueOnPreferences2520501900
--- FAIL: TestLeaseQueueProactiveEnqueueOnPreferences (56.39s)
=== RUN   TestLeaseQueueProactiveEnqueueOnPreferences/less-preferred
    lease_queue_test.go:398: condition failed to evaluate within 45s: from lease_queue_test.go:404: lease not on preferred node 1, on n2,s2
    --- FAIL: TestLeaseQueueProactiveEnqueueOnPreferences/less-preferred (45.20s)

Parameters:

• attempt=1
• deadlock=true
• run=1
• shard=22

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• #124691 kv/kvserver: TestLeaseQueueProactiveEnqueueOnPreferences failed [C-test-failure O-robot P-1 T-kv X-duplicate branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!