Comments (1)
arulajmani
commented on June 28, 2024
Looking at n5, we see lots of side-transport connection errors of the form:
teamcity-15465834-1717134592-109-n6cpu4-geo-0005> W240531 14:09:48.990622 1204 kv/kvserver/closedts/sidetransport/sender.go:833 ⋮ [T1,n5,ctstream=2] 300 failed to send closed timestamp message 1937 to n2: send msg error: ‹×›
We also see lots of failed follower reads because the CTS is lagging:
2.664ms 0.023ms event:kv/kvserver/replica_follower_read.go:118 [n5,s5,r21/6:‹×›] can't serve follower read; closed timestamp too low by: 3.527558563s; maxClosed: 1717165674.266061440,0 ts: 1717165677.293620003,0 uncertaintyLimit: ‹×›
We see further evidence of networking issues on n5:
teamcity-15465834-1717134592-109-n6cpu4-geo-0005> I240531 14:37:10.561956 3205 1@circuitbreaker/circuitbreaker.go:322 ⋮ [T1,n5] 135 circuitbreaker: ‹×› tripped: failed to connect to n6 at ‹×›: initial connection heartbeat failed: grpc: ‹×› [code 14/Unavailable]
teamcity-15465834-1717134592-109-n6cpu4-geo-0005> I240531 14:37:10.562005 3205 1@circuitbreaker/circuitbreaker.go:447 ⋮ [T1,n5] 136 circuitbreaker: ‹×› event: ‹×›
teamcity-15465834-1717134592-109-n6cpu4-geo-0005> W240531 14:37:10.562013 3205 2@rpc/nodedialer/nodedialer.go:196 ⋮ [T1,n5] 36 unable to connect to n6: failed to connect to n6 at ‹×›: initial connection heartbeat failed: grpc: ‹×› [code 14/Unavailable]
teamcity-15465834-1717134592-109-n6cpu4-geo-0005> W240531 14:37:10.562013 3205 2@rpc/nodedialer/nodedialer.go:196 ⋮ [T1,n5] 137 unable to connect to n6: failed to connect to n6 at ‹×›: initial connection heartbeat failed: grpc: ‹×› [code 14/Unavailable]
I'll close this out as a networking related infra flake.
from cockroach.
Related Issues (20)
- roachtest: unoptimized-query-oracle/disable-rules=half/rand-tables failed HOT 1
- roachtest: cluster_creation failed
- sqlstats: internal sql stats queries should have background qos level HOT 1
- sql: ALTER COLUMN TYPE on virtual column fails HOT 3
- Sentry: catch.go:29: while building cascade expression: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach... HOT 1
- sql: ALTER COULMN TYPE of stored computed column drops default value HOT 2
- Sentry: panic.go:884: panic: × (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Clos... HOT 1
- Sentry: error.go:89: unexpected error from the vectorized engine: runtime error: slice bounds out of range [2689:15] (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.... HOT 1
- Sentry: error.go:89: unexpected error from the vectorized engine: runtime error: slice bounds out of range [18:9] (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com... HOT 2
- Sentry: error.go:89: unexpected error from the vectorized engine: runtime error: slice bounds out of range [18:9] (1) Wraps: (2) assertion failure Wraps: (3) attached stack trace -- stack trace: |... HOT 2
- Sentry: error.go:89: unexpected error from the vectorized engine: runtime error: slice bounds out of range [2689:15] (1) Wraps: (2) assertion failure Wraps: (3) attached stack trace -- stack trace: ... HOT 2
- pkg/sql/logictest/tests/cockroach-go-testserver-23.1/cockroach-go-testserver-23_1_test: TestLogic_upgrade_skip_version failed [node unavailable in KV NodeLiveness even though it started up] HOT 2
- ldr: allow for multiple urls on job creation HOT 1
- sql/schemachanger: Need Size Estimation & Warnings for Schema Changes HOT 1
- kv/kvserver/loqrecovery: TestRejectBadVersionApplication failed
- kv/kvserver/protectedts/ptstorage: TestStorage failed HOT 1
- sql: crdb_internal.ranges_no_leases fetches all descriptors needlesly HOT 1
- schemachanger: zone config schema changes do not need to check out descriptors
- changefeedccl: warn or error when user specifies `resolved` < `min_checkpoint_frequency` HOT 1
- : failed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.