roachtest: acceptance/multitenant-multiregion failed about cockroach

[rafiss]

The error here is

  | 	Error:      	Received unexpected error:
  | 	            	no time out detected because of dead region.
  | 	            	(1) secondary error attachment
  | 	            	  | dial tcp 35.230.47.244:30258: connect: connection refused
  | 	            	  | (1) dial tcp 35.230.47.244:30258
  | 	            	  | Wraps: (2) connect
  | 	            	  | Wraps: (3) connection refused
  | 	            	  | Error types: (1) *net.OpError (2) *os.SyscallError (3) syscall.Errno
  | 	            	Wraps: (2) assertion failure
  | 	            	Wraps: (3) attached stack trace
  | 	            	  -- stack trace:
  | 	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAcceptanceMultitenantMultiRegion.func3
  | 	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant.go:279
  | 	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAcceptanceMultitenantMultiRegion.Group.GoCtx.func6
  | 	            	  | 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168
  | 	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
  | 	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
  | 	            	  | runtime.goexit
  | 	            	  | 	src/runtime/asm_amd64.s:1695

@fqazi Do you think we should add "connection refused" to the IsQueryTimeoutErr check?

[fqazi]

@rafiss I don't think thats the issue here, the lease query kept timing out but the region probe was 😬 :

We see these until the test times out

operation "count-leases-region" timed out after 15.004s (given timeout 15s): count-leases: failed to read query result: query execution canceled
06:04:41 multitenant.go:273: test status: Running schema change with lease held...

[cockroach-teamcity]

roachtest.acceptance/multitenant-multiregion failed with artifacts on master @ 2d67111f0db7bec9bfed537542c60f37f8340f69:

(test_runner.go:1198).runTest: test timed out (10m0s)
test artifacts and logs in: /artifacts/acceptance/multitenant-multiregion/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[fqazi]

Looking at the failure I think the test the sql_instance table is not distributed in a regional by row manner for some reason. We rely on the span config reconciliation to do the right thing, so not sure where things are wrong yet.

[fqazi]

Digging further I think the query we are using to make sure the ranges are distributed is wrong, and this isn't a code issue

[cockroach-teamcity]

roachtest.acceptance/multitenant-multiregion failed with artifacts on master @ c4ab095c4f65b9140661ed57adddc690b1e3ce3f:

(test_runner.go:1216).runTest: test timed out (10m0s)
test artifacts and logs in: /artifacts/acceptance/multitenant-multiregion/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!