sql/rowexec: TestDrainingProcessorSwallowsUncertaintyError failed about cockroach

[yuzefovich]

The test is blocked on cluster shutdown due to this goroutine being stuck

goroutine 21448 [sync.Cond.Wait, 40 minutes]:
sync.runtime_notifyListWait(0xc004db7610, 0x4)
	GOROOT/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc004db7600)
	GOROOT/src/sync/cond.go:70 +0x75
github.com/cockroachdb/cockroach/pkg/sql/rowexec.TestDrainingProcessorSwallowsUncertaintyError.func1({0xc0054607e0?, 0xb2a19d8?}, 0xc0054607e0)
	github.com/cockroachdb/cockroach/pkg/sql/rowexec/processors_test.go:486 +0x1c6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc002d02d88, {0xb2a19d8, 0xc0012a43c0}, 0xc0054607e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:170 +0x507
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc00468e408, {0xb2a19d8, 0xc00159fec0}, 0xc0054607e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0xd12
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc00285f050, {0xb2a19d8, 0xc00159fec0}, 0xc0054607e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203 +0xdf
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc004a08808, {0xb2a19d8, 0xc00159fe30}, {0xc006644840?}, 0xc0054607e0)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1414 +0x8d3
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc004a08808, {0xb2a19d8, 0xc00159fdd0}, 0xc0054607e0)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1548 +0x465
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0xb2a19d8, 0xc00159fdd0}, {0x82e9e60, 0xc0054607e0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x75
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0xb2a19d8, 0xc00159fdd0}, {0x82e9e60, 0xc0054607e0}, 0xc00521a9c0, 0xc0052d5488)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x72c
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xb2a19d8, 0xc00159fdd0}, {0x82e9e60, 0xc0054607e0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0xb2a19d8, 0xc00159fdd0}, {0x82e9e60, 0xc0054607e0}, 0xc00521a9c0, 0xc00521a9e0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0xf1
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xb2a19d8, 0xc00159fdd0}, {0x82e9e60, 0xc0054607e0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc0026ae000?, {{0xc004d40848?}, {0xb2de290?, 0xc0050cd110?}}}, {0xb2a19d8, 0xc00159fdd0}, {0x82e9e60, 0xc0054607e0}, 0xc00521a9c0, 0xc00521aa00)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x2b7
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xb2a19d8, 0xc00159fdd0}, {0x82e9e60, 0xc0054607e0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0xb2a19d8, 0xc00159fdd0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc004d40820, {0xb2a19d8, 0xc00159fdd0}, {0xc008075120?, 0xc0080751b0?}, 0xc0080751a8)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0x186
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0xb2a19d8, 0xc00159fdd0}, {0x82e9e60, 0xc0054607e0}, 0xc00521a9c0, 0xc00521aa20)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x151
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xb2a19d8, 0xc00159fdd0}, {0x82e9e60, 0xc0054607e0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0xb2a19d8, 0xc00159fdd0}, {0x0?, 0x0?}, {0x82e9e60, 0xc0054607e0}, {0x81b7e00, 0xc0057f5b00}, 0xc008075348?, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x76
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8({0xb2a19d8, 0xc00159fdd0}, {0x83dc4a7, 0x21}, {0x82e9e60, 0xc0054607e0}, {0x81b7e00, 0xc0057f5b00}, 0x0, 0xc003664a10, ...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x5ee
github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1({0xb2a19d8, 0xc00159fdd0}, {0x83dc4a7, 0x21}, {0x82e9e60, 0xc0054607e0}, {0x81b7e00, 0xc0057f5b00}, 0x0, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0x19a
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0xb2a19a0, 0xc006d75680}, 0xc005460480, {0x0, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x463
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907

It seems like there is a possible race in the test itself, so I wouldn't just skip it under race and close the issue.

[cockroach-teamcity]

sql/rowexec.TestDrainingProcessorSwallowsUncertaintyError failed on master @ 0dce29441187eda187aad472543997b06c2b01ea:

Fatal error:

panic: test timed out after 41m37s
running tests:
	TestDrainingProcessorSwallowsUncertaintyError (40m57s)
	TestDrainingProcessorSwallowsUncertaintyError/vectorize=false (40m50s)
	TestDrainingProcessorSwallowsUncertaintyError/vectorize=false/dummy=true (40m50s)

Stack:

goroutine 966426 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45

Log preceding fatal error

=== RUN   TestDrainingProcessorSwallowsUncertaintyError
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestDrainingProcessorSwallowsUncertaintyError2037115885
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)

pkg/sql/rowexec/processors_test.go:521: (TestDrainingProcessorSwallowsUncertaintyError)
	NOTICE: .GetFirstStoreID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().GetFirstStoreID() instead.

pkg/sql/rowexec/processors_test.go:522: (TestDrainingProcessorSwallowsUncertaintyError)
	NOTICE: .GetFirstStoreID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().GetFirstStoreID() instead.

pkg/sql/rowexec/processors_test.go:528: (TestDrainingProcessorSwallowsUncertaintyError)
	NOTICE: .AdvSQLAddr() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().AdvSQLAddr() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
=== RUN   TestDrainingProcessorSwallowsUncertaintyError/vectorize=false/dummy=false
=== RUN   TestDrainingProcessorSwallowsUncertaintyError/vectorize=false/dummy=true

Parameters:

• attempt=1
• race=true
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!