ccl/kvccl/kvfollowerreadsccl: TestBoundedStalenessDataDriven failed about cockroach

[miraradeva]

Haven't been able to repro on a gce worker, under deadlock, on the same sha as the failure:

./dev test pkg/ccl/kvccl/kvfollowerreadsccl/ --filter=TestBoundedStalenessDataDriven --stress --deadlock --count=5000

The main thing I see in the logs right before the timeout are repeated (over 200 of them) lines like this:

I240525 08:16:07.851733 2280561 sql/catalog/lease/descriptor_state.go:256 ⋮ [T1,Vsystem,n3,client=127.0.0.1:43292,hostssl,user=root] 2083  release: 104(‹"t"›,0101807ad272fe77444245aeb087b0700b9108) ver=9:1716625276.970286272,0, refcount=0

I think it's possible that one of the schema change jobs is not completing on time while waiting for that sql lease expire:

I240525 08:16:06.960110 2381445 sql/catalog/lease/lease.go:264 ⋮ [T1,Vsystem,n1,job=‹NEW SCHEMA CHANGE id=971749981398958081›] 2077  waiting for 1 leases to expire: desc=[{‹t› 104 8}]

[rickystewart]

FYI, this test is given only one core even under deadlock. You may find it easier to reproduce with --test_env=GOMAXPROCS=1 or alternatively we can grant it additional resources under deadlock to see if that helps.

[miraradeva]

Also no luck reproducing with ./dev test pkg/ccl/kvccl/kvfollowerreadsccl/ --filter=TestBoundedStalenessDataDriven --stress --deadlock --count=5000 -- --test_env=GOMAXPROCS=1.

[arulajmani]

I'm going to close this out as unactionable. We've seen this fail only once, and there's nothing in the logs that's useful and Mira couldn't repro this.