Comments (2)
Thanks to the logging @nvanbenschoten added, we can confirm that the merge transaction is indeed spinning. The logs are littered with:
W240626 12:47:43.130670 28 kv/kvserver/replica_command.go:960 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 28880 merge txn failed (attempt 12365): TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED): "merge" meta={id=ee3591bd key=/Local/Range/System‹/›‹"�aaa-testing›‹"›/‹RangeDescriptor› iso=Serializable pri=0.01508323 epo=0 ts=1719566794.578285532,0 min=1719566794.578285532,0 seq=5} lock=true stat=PENDING rts=1719566794.578285532,0 wto=false gul=1719566795.078285532,0
I240626 12:47:43.155019 28 kv/kvserver/replica_command.go:849 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 28881 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
W240626 12:47:43.615552 28 kv/kvserver/replica_command.go:960 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 28882 merge txn failed (attempt 12366): TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED): "merge" meta={id=4a806099 key=/Local/Range/System‹/›‹"�aaa-testing›‹"›/‹RangeDescriptor› iso=Serializable pri=0.03137873 epo=0 ts=1719566808.135986107,0 min=1719566808.135986107,0 seq=5} lock=true stat=PENDING rts=1719566808.135986107,0 wto=false gul=1719566808.635986107,0
I240626 12:47:43.650410 28 kv/kvserver/replica_command.go:849 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 28883 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
W240626 12:47:44.342121 28 kv/kvserver/replica_command.go:960 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 28884 merge txn failed (attempt 12367): TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED): "merge" meta={id=fa89842c key=/Local/Range/System‹/›‹"�aaa-testing›‹"›/‹RangeDescriptor› iso=Serializable pri=0.02738480 epo=0 ts=1719566821.626367241,0 min=1719566821.626367241,0 seq=5} lock=true stat=PENDING rts=1719566821.626367241,0 wto=false gul=1719566822.126367241,0
I240626 12:47:44.370516 28 kv/kvserver/replica_command.go:849 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 28885 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
We see from the log lines that the transaction keeps failing because of ABORT_REASON_TIMESTAMP_CACHE_REJECTED
. Seems like we're losing resolution in the timestamp cache's read summary. This can happen if we're either compressing the read summary for some reason or we're loading the read summary from the RangePriorReadSummaryKey
key.
The compression budget is set to 4MB for local keys by default, so it's likely not that.
The only place where we use the persisted read summary is in applySnapshot
. Give this is a single server test, that also seems surprising.
from cockroach.
I hit this in a kvserver race build that timed out. In the tmp logs, the cockroach cluster running this test was up for 20 minutes. https://teamcity.cockroachdb.com/viewLog.html?buildId=15874939&buildTypeId=Cockroach_Ci_Tests_Testrace&tab=artifacts#%2Ftmp;%2Fbazel-testlogs%2Fpkg%2Fkv%2Fkvclient%2Frangefeed
from cockroach.
Related Issues (20)
- roachtest: costfuzz/rand-multi-region failed
- Sentry: builder_state.go:1785: building declarative schema change targets for CREATE SCHEMA: schema name empty when resolving database prefix for a schema name (1) attached stack trace -- stack trac...
- roachtest: costfuzz/rand-multi-region failed
- roachtest: unoptimized-query-oracle/disable-rules=all/rand-tables failed
- Freshly created cluster stops working after w few hours HOT 3
- pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [enum type does not exist during ALTER COLUMN SET DEFAULT] HOT 2
- ccl/jwtauthccl: TestJWTAuthWithCustomCACert failed
- roachtest: perturbation/metamorphic/decommission failed
- roachtest: unoptimized-query-oracle/disable-rules=all/seed-multi-region failed
- admission: add roachtest for disk-bound overload for snapshots
- security/ldap: clarify that cert verification error is due to missing LDAP CA cert
- security/LDAP: SHOW CLUSTER SETTING needs to redact LDAP bind password
- kvserver: wiring to initialize and use replica_rac2.Processor
- OSS-Fuzz issue 71224 HOT 1
- pkg/sql/schemachanger/schemachanger_test: TestConcurrentSchemaChanges failed
- kv/kvserver: TestPromoteNonVoterInAddVoter failed
- sql, kv: FOR UPDATE NOWAIT can starve or block under serializable HOT 3
- ccl/sqlproxyccl: TestDirectoryConnect failed
- Sentry: error.go:25: unexpected error: forecasted histogram had first bucket with non-zero NumRange or DistinctRange: (1) × Wraps: (2) issue #93892 Wraps: (3) attached stack trace -- stack trace: ...
- sql: 'infinity'::time should be greater than any other time HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.