kv/kvserver/reports: TestCriticalLocalitiesReportIntegration failed about cockroach

[pav-kv]

The error formatted in consumable way:

error executing query="ALTER TABLE t EXPERIMENTAL_RELOCATE VALUES (ARRAY[1,4,5], 1)" args=[]:
pq: while carrying out changes [{ADD_VOTER n5,s5} {REMOVE_VOTER n2,s2}]:
change replicas of r69 failed:
(n2,s2):2VOTER_DEMOTING_LEARNER received invalid ChangeReplicasTrigger LEAVE_JOINT:
after=[(n1,s1):1 (n2,s2):2LEARNER (n3,s3):3 (n5,s5):4] next=5 to remove self (leaseholder); lhRemovalAllowed: true;
current desc: r69:/{Table/104/1/0-Max} [(n1,s1):1, (n2,s2):2VOTER_DEMOTING_LEARNER, (n3,s3):3, (n5,s5):4VOTER_INCOMING, next=5, gen=7, sticky=9223372036.854775807,2147483647];
proposed desc: r69:/{Table/104/1/0-Max} [(n1,s1):1, (n2,s2):2LEARNER, (n3,s3):3, (n5,s5):4, next=5, gen=8, sticky=9223372036.854775807,2147483647]:
lease target replica cannot hold lease

[arulajmani]

From the logs, we see:

W240605 10:55:07.442566 13586 kv/kvclient/kvcoord/dist_sender.go:2786 ⋮ [T1,Vsystem,n1,client=127.0.0.1:52210,hostssl,user=root] 875  slow replica RPC: have been waiting 10.61s (0 attempts) for RPC AdminRelocateRange [/Table/104/1/‹1›/‹0›] to replica (n1,s1):1; resp: ‹(err: while carrying out changes [{ADD_VOTER n5,s5} {REMOVE_VOTER n2,s2}]: change replicas of r69 failed: (n2,s2):2VOTER_DEMOTING_LEARNER received invalid ChangeReplicasTrigger LEAVE_JOINT: after=[(n1,s1):1 (n2,s2):2LEARNER (n3,s3):3 (n5,s5):4] next=5 to remove self (leaseholder); lhRemovalAllowed: true; current desc: r69:/{Table/104/1/0-Max} [(n1,s1):1, (n2,s2):2VOTER_DEMOTING_LEARNER, (n3,s3):3, (n5,s5):4VOTER_INCOMING, next=5, gen=7, sticky=9223372036.854775807,2147483647]; proposed desc: r69:/{Table/104/1/0-Max} [(n1,s1):1, (n2,s2):2LEARNER, (n3,s3):3, (n5,s5):4, next=5, gen=8, sticky=9223372036.854775807,2147483647]: lease target replica cannot hold lease)›

We're unable to exit the joint configuration because the leaseholder (n5) is still a VOTER_DEMOTING_LEARNER and is being demoted to the LEARNER. Given how long the slow range RPC is waiting, my guess is that it transferred the lease away, which expired, and then it was able to reacquire the lease.

We've previously closed such issues out (when they were running on engflow under race) as infra-flakes. I'll do the same here. If this comes back, we can look into retrying after these errors.