streamnative / oxia

Oxia - Metadata store and coordination system

StreamNative Oxia

Oxia is a scalable metadata store and coordination system that can be used as the core infrastructure to build large scale distributed systems.

Why Oxia?

• Design optimized for Kubernetes environment
• Transparent horizontal scalability
• Linearizable per-key operations
• Able to sustain millions of read/write per second
• Able to store 100s of GBs

Features

• Key-Value interface
• Notification mechanism
• Ephemeral records
• Automated session management

Documentation

• Design goals
• Architecture
• Getting started with Oxia
• Go client API
• Deploy using Helm Chart
• Deploy using bare metal
• Developer docs
  • Replication protocol
    • Coordinator
    • Storage
  • Verifying correctness
  • Oxia's K8S resources

License

Copyright 2023 StreamNative, Inc.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

oxia's Issues

【Feature】add crc to wal

oxia use wal to storage the LogEntry in segment files, should we add crc to each record to protect data corrupted？

Some Questions about Oxia

Hi Team.
Recently I've seen your post and is very impressed about oxia is doing.
Couple of questions about oxia:

1. Is there a plan to replace bookeeper with oxia log storage?
2. If not, will we change bookeeper zk meta to oxia also?
3. What's is typical failure recovery time for one storage node down and Coordinator down?

【CI】enable more linter

Currently，there is only 7 linters is enabled:

• errcheck
• goimports
• gosimple
• govet
• ineffassign
• staticcheck
• unused

But there is more powerful linters，such as：

• goconst
• gocyclo
• misspell
• revive

and so on. Should we enable more linters？

Support back pressure in a performant way in `batch.Batcher`

The main event loop in batch.Batcher.run() completes (sends gRPC requests) synchronously. This means it blocks the event loop, which in turn blocks the input channel callC which is unbuffered. This results in the main oxia.Client methods blocking.

We could buffer the callC channel to allow some records to continue to be inserted (up to a limit) while the current batch is being completed. And/or we could complete the batch in a goroutine with some kind of barrier to prevent ending up with no back pressure at all.

Use fsnotify to watch conf file changes

Improve this part

oxia/coordinator/impl/coordinator.go

Lines 406 to 431 in d27833a

	func (c *coordinator) waitForExternalEvents() {
	refreshTimer := time.NewTicker(c.clusterConfigRefreshTime)
	defer refreshTimer.Stop()
	for {
	select {
	case <-c.ctx.Done():
	return
	case <-refreshTimer.C:
	if err := c.handleClusterConfigUpdated(); err != nil {
	c.log.Warn(
	"Failed to update cluster config",
	slog.Any("error", err),
	)
	}
	if err := c.rebalanceCluster(); err != nil {
	c.log.Warn(
	"Failed to rebalance cluster",
	slog.Any("error", err),
	)
	}
	}
	}
	}

Failed test with leaked iterator: TestLeaderController_DeleteShard

--- FAIL: TestLeaderController_DeleteShard (0.02s)
    leader_controller_test.go:996: 
        	Error Trace:	/home/runner/work/oxia/oxia/server/leader_controller_test.go:996
        	Error:      	Received unexpected error:
        	            	leaked iterators:
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/pebble.(*DB).Close
        	            	  | 	/home/runner/go/pkg/mod/github.com/cockroachdb/[email protected]/db.go:1314
        	            	  | oxia/server/kv.(*Pebble).Close
        	            	  | 	/home/runner/work/oxia/oxia/server/kv/kv_pebble.go:320
        	            	  | oxia/server/kv.(*db).Close
        	            	  | 	/home/runner/work/oxia/oxia/server/kv/db.go:132
        	            	  | oxia/server.(*leaderController).close
        	            	  | 	/home/runner/work/oxia/oxia/server/leader_controller.go:830
        	            	  | oxia/server.(*leaderController).Close
        	            	  | 	/home/runner/work/oxia/oxia/server/leader_controller.go:790
        	            	  | oxia/server.TestLeaderController_DeleteShard
        	            	  | 	/home/runner/work/oxia/oxia/server/leader_controller_test.go:996
        	            	  | testing.tRunner
        	            	  | 	/opt/hostedtoolcache/go/1.19.7/x64/src/testing/testing.go:1446
        	            	  | runtime.goexit
        	            	  | 	/opt/hostedtoolcache/go/1.19.7/x64/src/runtime/asm_amd64.s:1594
        	            	Wraps: (2) leaked iterators:
        	            	  | 0.0:
        	            	  |   000004:[__oxia/term#1,SET-__oxia/term#1,SET]
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
        	Test:       	TestLeaderController_DeleteShard

Nil pointer exception when getting notification on newly initialized leader

It appears a race condition on the leader initialization and receiving a GetNotifications() call from clients.

Jan 17 00:40:16.942 |INF| Received NewTerm request component='internal-rpc-server' peer='10.44.2.121:48644' req={ namespace='default' shardId='12' term='18' }
Jan 17 00:40:16.992 |INF| [JOB 1] WAL file /data/db/default/shard-12/000020.log with log number 000020 stopped reading at offset: 0; replayed 0 keys in 0 batches component='pebble' shard=12
Jan 17 00:40:17.004 |INF| Created leader controller component='leader-controller' namespace='default' shard=12 term=17
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x19e484a]
goroutine 109 [running]:
github.com/streamnative/oxia/server.(*leaderController).dispatchNotifications(0xc000256200, {0x22b0308, 0xc000690690}, 0xc00038e100, {0x22b61b0, 0xc000514200})
	/src/oxia/server/leader_controller.go:807 +0x16a
github.com/streamnative/oxia/server.(*leaderController).GetNotifications.func1()
	/src/oxia/server/leader_controller.go:773 +0x4f
github.com/streamnative/oxia/common.DoWithLabels.func1({0x22b02d0?, 0xc000908c00?})
	/src/oxia/common/pprof.go:46 +0x13
runtime/pprof.Do({0x22b0308?, 0xc000690690?}, {{0xc0003197a0?, 0x18d5980?, 0xc00067ce28?}}, 0xc00067cf30)
	/usr/local/go/src/runtime/pprof/runtime.go:51 +0x9d
github.com/streamnative/oxia/common.DoWithLabels({0x22b0308, 0xc000690690}, 0xc000908b70, 0xc00038e300)
	/src/oxia/common/pprof.go:42 +0x376
created by github.com/streamnative/oxia/server.(*leaderController).GetNotifications in goroutine 107
	/src/oxia/server/leader_controller.go:765 +0x2f0

[bug] Failed to trim wal: out of range

Apr  6 16:48:40.878249 ERR Failed to trim the wal error="failed to trim wal: out of range" component=wal-trimmer namespace=test shard=3 stack=[{"func":"(*trimmer).doTrim","line":"148","source":"wal_trimmer.go"},{"func":"(*trimmer).run","line":"95","source":"wal_trimmer.go"},{"func":"DoWithLabels.func1","line":"43","source":"pprof.go"},{"func":"Do","line":"44","source":"runtime.go"},{"func":"DoWithLabels","line":"39","source":"pprof.go"},{"func":"goexit","line":"1172","source":"asm_arm64.s"}]
Apr  6 20:36:10.218634 ERR Failed to trim the wal error="failed to trim wal: out of range" component=wal-trimmer namespace=test shard=3 stack=[{"func":"(*trimmer).doTrim","line":"148","source":"wal_trimmer.go"},{"func":"(*trimmer).run","line":"95","source":"wal_trimmer.go"},{"func":"DoWithLabels.func1","line":"43","source":"pprof.go"},{"func":"Do","line":"44","source":"runtime.go"},{"func":"DoWithLabels","line":"39","source":"pprof.go"},{"func":"goexit","line":"1172","source":"asm_arm64.s"}]
Apr  6 21:21:56.215243 ERR Failed to trim the wal error="failed to trim wal: out of range" component=wal-trimmer namespace=test shard=3 stack=[{"func":"(*trimmer).doTrim","line":"148","source":"wal_trimmer.go"},{"func":"(*trimmer).run","line":"95","source":"wal_trimmer.go"},{"func":"DoWithLabels.func1","line":"43","source":"pprof.go"},{"func":"Do","line":"44","source":"runtime.go"},{"func":"DoWithLabels","line":"39","source":"pprof.go"},{"func":"goexit","line":"1172","source":"asm_arm64.s"}]
Apr  6 21:31:56.206390 ERR Failed to trim the wal error="failed to trim wal: out of range" component=wal-trimmer namespace=test shard=3 stack=[{"func":"(*trimmer).doTrim","line":"148","source":"wal_trimmer.go"},{"func":"(*trimmer).run","line":"95","source":"wal_trimmer.go"},{"func":"DoWithLabels.func1","line":"43","source":"pprof.go"},{"func":"Do","line":"44","source":"runtime.go"},{"func":"DoWithLabels","line":"39","source":"pprof.go"},{"func":"goexit","line":"1172","source":"asm_arm64.s"}]
Apr  6 21:41:56.245300 ERR Failed to trim the wal error="failed to trim wal: out of range" component=wal-trimmer namespace=test shard=3 stack=[{"func":"(*trimmer).doTrim","line":"148","source":"wal_trimmer.go"},{"func":"(*trimmer).run","line":"95","source":"wal_trimmer.go"},{"func":"DoWithLabels.func1","line":"43","source":"pprof.go"},{"func":"Do","line":"44","source":"runtime.go"},{"func":"DoWithLabels","line":"39","source":"pprof.go"},{"func":"goexit","line":"1172","source":"asm_arm64.s"}]
Apr  6 21:51:56.227021 ERR Failed to trim the wal error="failed to trim wal: out of range" component=wal-trimmer namespace=test shard=3 stack=[{"func":"(*trimmer).doTrim","line":"148","source":"wal_trimmer.go"},{"func":"(*trimmer).run","line":"95","source":"wal_trimmer.go"},{"func":"DoWithLabels.func1","line":"43","source":"pprof.go"},{"func":"Do","line":"44","source":"runtime.go"},{"func":"DoWithLabels","line":"39","source":"pprof.go"},{"func":"goexit","line":"1172","source":"asm_arm64.s"}]

How to interact the standalone service?

After build from source and start the standalone service with ./bin/oxia standalone, it logs:

Apr 10 15:23:13.524376 INF Started Grpc server bindAddress=[::]:6648 grpc-server=public
Apr 10 15:23:13.524509 INF Serving Prometheus metrics at http://localhost:8080/metrics

However, I don't know how to play a hello world or health check at port 6648.

BTW, although I can test to browse http://localhost:8080/metrics, this port seems not export in the Docker startup instruction.

Flaky test: TestFollowerCursor

https://github.com/streamnative/oxia/actions/runs/4680271784/jobs/8291399457?pr=325

--- FAIL: TestFollowerCursor (0.31s)
    follower_cursor_test.go:56: 
        	Error Trace:	/home/runner/work/oxia/oxia/server/follower_cursor_test.go:56
        	Error:      	Not equal: 
        	            	expected: -1
        	            	actual  : 0
        	Test:       	TestFollowerCursor

Change module path from `oxia` to `github.com/streamnative/oxia`

current module path oxia is too short, and base on golang's suggest:

A module path should describe both what the module does and where to find it. Typically, a module path consists of a repository root path, a directory within the repository (usually empty), and a major version suffix (only for major version 2 or higher).

and other's golang module also formatted as github.com/streamnative/xxx, so should we change this module path to github.com/streamnative/oxia？

Flaky test: TestBatcher

I've seen it failing few times locally:

--- FAIL: TestBatcher (0.00s)
    batcher_test.go:69:
        	Error Trace:	/Users/mmerli/prg/oxia/oxia/internal/batch/batcher_test.go:69
        	            				/Users/mmerli/prg/oxia/oxia/internal/batch/asm_arm64.s:1172
        	Error:      	Not equal:
        	            	expected: <nil>(<nil>)
        	            	actual  : *errors.errorString(&errors.errorString{s:"shutting down"})
        	Test:       	TestBatcher
FAIL

Configure the Docker image entrypoint as `oxia`?

... then we can run:

docker run -p 6648:6648 streamnative/oxia standalone 

instead of:

docker run -p 6648:6648 streamnative/oxia oxia standalone 

I don't know what's the convention, but it seems the binary is what we want to deliver to our users. And users can always change the entry point, which is rare but viable.

Reserve more metadata in coordinator status configmap

oxia/coordinator/impl/metadata_configmap.go

Lines 133 to 138 in 281313c

	cm := &corev1.ConfigMap{
	ObjectMeta: metav1.ObjectMeta{Name: name},
	Data: map[string]string{
	"status": string(bytes),
	},
	}

This part of code will always create a new configmap and override the old one. But I think it should reserve the metadata part of the old configmap, for example ownerReferences filed is useful for cascading deletion. The labels field may also be useful.

[CI]: setup the release workflow for Oxia to push the chart to https://artifacthub.io/

Thanks

Gentlemen, this is the only remaining company channel I can use to say thank you for all the time we spent together working on this really fun project.
Best,
Andras

【Issue】ReadWriteSegment.Truncate didn't really discard entry

Backoff and timing out requests to service discovery

If the client is started and the server has not received the service discovery info, the client keeps trying forever to fetch the shards assignments, without any timeout and without backoff between attempts.

Test timeout

e.g. https://github.com/streamnative/oxia/actions/runs/3545294385/jobs/5953324062

panic: test timed out after 10m0s

goroutine 143 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.18.8/x64/src/testing/testing.go:2029 +0xbb
created by time.goFunc
	/opt/hostedtoolcache/go/1.18.8/x64/src/time/sleep.go:176 +0x48

goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc0002296c0, {0x1194159, 0x12}, 0x11da060)
	/opt/hostedtoolcache/go/1.18.8/x64/src/testing/testing.go:1487 +0x[75](https://github.com/streamnative/oxia/actions/runs/3545294385/jobs/5953324062#step:9:76)0
testing.runTests.func1(0x0?)
	/opt/hostedtoolcache/go/1.18.8/x64/src/testing/testing.go:1839 +0x9a
testing.tRunner(0xc0002296c0, 0xc0001c5b68)
	/opt/hostedtoolcache/go/1.18.8/x64/src/testing/testing.go:1439 +0x214
testing.runTests(0xc00028d040?, {0x17dfb20, 0xc, 0xc}, {0x40?, 0x7f450ad46e70?, 0x183c7e0?})
	/opt/hostedtoolcache/go/1.18.8/x64/src/testing/testing.go:1837 +0x7e5
testing.(*M).Run(0xc00028d040)
	/opt/hostedtoolcache/go/1.18.8/x64/src/testing/testing.go:1719 +0xa72
main.main()
	_testmain.go:135 +0x3aa

goroutine 38 [chan receive, 9 minutes]:
oxia/server.(*mockClientAddEntriesStream).Recv(0xc00000e060)
	/home/runner/work/oxia/oxia/server/mock_test.go:88 +0x51
oxia/server.(*followerCursor).receiveAcks(0xc0000e4900, {0x130d118, 0xc00000e060})
	/home/runner/work/oxia/oxia/server/follower_cursor.go:195 +0x1bb
created by oxia/server.(*followerCursor).runOnce
	/home/runner/work/oxia/oxia/server/follower_cursor.go:147 +0x34b

goroutine 9 [chan receive, 9 minutes]:
oxia/server.(*mockServerAddEntriesStream).Recv(0xc00000e168)
	/home/runner/work/oxia/oxia/server/mock_test.go:37 +0x4c
oxia/server.(*followerController).AddEntries(0x0?, {0x130d170, 0xc00000e168})
	/home/runner/work/oxia/oxia/server/follower_controller.go:173 +0x66
oxia/server.TestFollower.func1()
	/home/runner/work/oxia/oxia/server/follower_controller_test.go:59 +0x65
created by oxia/server.TestFollower
	/home/runner/work/oxia/oxia/server/follower_controller_test.go:59 +0x8d7

goroutine 50 [chan receive, 9 minutes]:
oxia/server.TestFollowerCursor(0xc0000d86[80](https://github.com/streamnative/oxia/actions/runs/3545294385/jobs/5953324062#step:9:81))
	/home/runner/work/oxia/oxia/server/follower_cursor_test.go:46 +0x845
testing.tRunner(0xc0000d8680, 0x11da060)
	/opt/hostedtoolcache/go/1.18.8/x64/src/testing/testing.go:1439 +0x214
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.18.8/x64/src/testing/testing.go:1486 +0x725

goroutine 27 [chan receive, 9 minutes]:
oxia/server.(*mockServerAddEntriesStream).Recv(0xc00012fd88)
	/home/runner/work/oxia/oxia/server/mock_test.go:37 +0x4c
oxia/server.(*followerController).AddEntries(0x0?, {0x130d170, 0xc00012fd88})
	/home/runner/work/oxia/oxia/server/follower_controller.go:173 +0x66
oxia/server.TestReadingUpToCommitIndex.func1()
	/home/runner/work/oxia/oxia/server/follower_controller_test.go:157 +0x65
created by oxia/server.TestReadingUpToCommitIndex
	/home/runner/work/oxia/oxia/server/follower_controller_test.go:157 +0x7c9

goroutine 51 [runnable]:
sync.(*Mutex).Lock(0xc0000e4900)
	/opt/hostedtoolcache/go/1.18.8/x64/src/sync/mutex.go:72 +0x7e
oxia/server.(*followerCursor).runOnce(0xc0000e4900)
	/home/runner/work/oxia/oxia/server/follower_cursor.go:154 +0x4f4
oxia/server.(*followerCursor).run(0xc0000e4900)
	/home/runner/work/oxia/oxia/server/follower_cursor.go:122 +0xa5
created by oxia/server.NewFollowerCursor
	/home/runner/work/oxia/oxia/server/follower_cursor.go:[82](https://github.com/streamnative/oxia/actions/runs/3545294385/jobs/5953324062#step:9:83) +0x7c9
FAIL	oxia/server	600.045s
ok  	oxia/server/kv	0.106s	coverage: [84](https://github.com/streamnative/oxia/actions/runs/3545294385/jobs/5953324062#step:9:85).3% of statements
ok  	oxia/server/util	0.024s	coverage: 100.0% of statements
ok  	oxia/server/wal	0.040s	coverage: [85](https://github.com/streamnative/oxia/actions/runs/3545294385/jobs/5953324062#step:9:86).3% of statements
?   	oxia/standalone	[no test files]
FAIL
make: *** [Makefile:7: test] Error 1
Error: Process completed with exit code 2.

Data race in TestAsyncClientImpl_NotificationsClose

==================
WARNING: DATA RACE
Write at 0x00c0004e4fa0 by goroutine 165:
  oxia/server.(*leaderController).close()
      /home/runner/work/oxia/oxia/server/leader_controller.go:831 +0x69c
  oxia/server.(*leaderController).Close()
      /home/runner/work/oxia/oxia/server/leader_controller.go:790 +0x84
  oxia/server.(*shardsDirector).Close()
      /home/runner/work/oxia/oxia/server/shards_director.go:189 +0x170
  oxia/server.(*Standalone).Close()
      /home/runner/work/oxia/oxia/server/standalone.go:145 +0xdb
  oxia/oxia.TestAsyncClientImpl_NotificationsClose()
      /home/runner/work/oxia/oxia/oxia/async_client_impl_test.go:201 +0x41b
  testing.tRunner()
      /opt/hostedtoolcache/go/1.19.7/x64/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.19.7/x64/src/testing/testing.go:1493 +0x47

Previous read at 0x00c0004e4fa0 by goroutine 201:
  oxia/server.(*leaderController).dispatchNotifications()
      /home/runner/work/oxia/oxia/server/leader_controller.go:762 +0x384
  oxia/server.(*leaderController).GetNotifications.func1()
      /home/runner/work/oxia/oxia/server/leader_controller.go:711 +0x94
  oxia/common.DoWithLabels.func1()
      /home/runner/work/oxia/oxia/common/pprof.go:43 +0x30
  runtime/pprof.Do()
      /opt/hostedtoolcache/go/1.19.7/x64/src/runtime/pprof/runtime.go:40 +0x122
  oxia/common.DoWithLabels()
      /home/runner/work/oxia/oxia/common/pprof.go:39 +0x4f0
  oxia/server.(*leaderController).GetNotifications.func2()
      /home/runner/work/oxia/oxia/server/leader_controller.go:706 +0x47

Goroutine 165 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.19.7/x64/src/testing/testing.go:1493 +0x75d
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.19.7/x64/src/testing/testing.go:[184](https://github.com/streamnative/oxia/actions/runs/4633142831/jobs/8198030286?pr=312#step:9:185)6 +0x99
  testing.tRunner()
      /opt/hostedtoolcache/go/1.19.7/x64/src/testing/testing.go:1446 +0x216
  testing.runTests()
      /opt/hostedtoolcache/go/1.19.7/x64/src/testing/testing.go:1844 +0x7ec
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.19.7/x64/src/testing/testing.go:1726 +0xa84
  oxia/oxia.TestMain()
      /home/runner/work/oxia/oxia/oxia/cache_test.go:43 +0x291
  main.main()
      _testmain.go:149 +0x404

Goroutine [201](https://github.com/streamnative/oxia/actions/runs/4633142831/jobs/8198030286?pr=312#step:9:202) (running) created at:
  oxia/server.(*leaderController).GetNotifications()
      /home/runner/work/oxia/oxia/server/leader_controller.go:706 +0x4c4
  oxia/server.(*publicRpcServer).GetNotifications()
      /home/runner/work/oxia/oxia/server/public_rpc_server.go:[202](https://github.com/streamnative/oxia/actions/runs/4633142831/jobs/8198030286?pr=312#step:9:203) +0x105
  oxia/proto._OxiaClient_GetNotifications_Handler()
      /home/runner/work/oxia/oxia/proto/client_grpc.pb.go:403 +0xf8
  github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1()
      /home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121 +0x[206](https://github.com/streamnative/oxia/actions/runs/4633142831/jobs/8198030286?pr=312#step:9:207)
  google.golang.org/grpc.(*Server).processStreamingRPC()
      /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1636 +0x1ef5
  google.golang.org/grpc.(*Server).handleStream()
      /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1717 +0xfae
  google.golang.org/grpc.(*Server).serveStreams.func1.2()
      /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:965 +0xec
==================

What about inline getting started doc into the README file?

It's a common path to understanding how to build and how play with a simple app.

Currently, we have the doc that located at https://github.com/streamnative/oxia/blob/main/docs/getting-started.md. This is fine but not quite straight forward. I propose and volunteer to merge it into the README file.

cc @merlimat @asafm thoughts?

Test timeout

coverage: 31.8% of statements
panic: test timed out after 10m0s

goroutine 209 [running]:
testing.(*M).startAlarm.func1()
        /opt/homebrew/Cellar/go/1.19.3/libexec/src/testing/testing.go:2036 +0xb4
created by time.goFunc
        /opt/homebrew/Cellar/go/1.19.3/libexec/src/time/sleep.go:176 +0x48

goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc0002c41a0, {0x1050ee76e, 0x12}, 0x10546ba00)
        /opt/homebrew/Cellar/go/1.19.3/libexec/src/testing/testing.go:1494 +0x57c
testing.runTests.func1(0x0?)
        /opt/homebrew/Cellar/go/1.19.3/libexec/src/testing/testing.go:1846 +0x94
testing.tRunner(0xc0002c41a0, 0xc000247b68)
        /opt/homebrew/Cellar/go/1.19.3/libexec/src/testing/testing.go:1446 +0x18c
testing.runTests(0xc000153180?, {0x105a14040, 0x12, 0x12}, {0xc000123d40?, 0x1046c21f0?, 0x105a291e0?})
        /opt/homebrew/Cellar/go/1.19.3/libexec/src/testing/testing.go:1844 +0x6c4
testing.(*M).Run(0xc000153180)
        /opt/homebrew/Cellar/go/1.19.3/libexec/src/testing/testing.go:1726 +0x874
main.main()
        _testmain.go:147 +0x3b8

goroutine 104 [chan receive, 9 minutes]:
oxia/server.TestFollowerCursor(0xc0004341a0)
        /Users/andrasbeni/oxia/server/follower_cursor_test.go:45 +0x748
testing.tRunner(0xc0004341a0, 0x10546ba00)
        /opt/homebrew/Cellar/go/1.19.3/libexec/src/testing/testing.go:1446 +0x18c
created by testing.(*T).Run
        /opt/homebrew/Cellar/go/1.19.3/libexec/src/testing/testing.go:1493 +0x560

goroutine 39 [chan receive, 9 minutes]:
oxia/server.(*mockRpcClient).Recv(0xc0002e8150)
        /Users/andrasbeni/oxia/server/mock_test.go:98 +0x48
oxia/server.(*followerCursor).receiveAcks(0xc00015a400, {0x10547bc88, 0xc0002e8150})
        /Users/andrasbeni/oxia/server/follower_cursor.go:195 +0x158
created by oxia/server.(*followerCursor).runOnce
        /Users/andrasbeni/oxia/server/follower_cursor.go:147 +0x278

goroutine 68 [chan receive, 9 minutes]:
oxia/server.(*mockServerAddEntriesStream).Recv(0xc0001c13e0)
        /Users/andrasbeni/oxia/server/mock_test.go:37 +0x44
oxia/server.(*followerController).AddEntries(0x0?, {0x10547bce0, 0xc0001c13e0})
        /Users/andrasbeni/oxia/server/follower_controller.go:178 +0x5c
oxia/server.TestFollower.func1()
        /Users/andrasbeni/oxia/server/follower_controller_test.go:53 +0x60
created by oxia/server.TestFollower
        /Users/andrasbeni/oxia/server/follower_controller_test.go:53 +0x78c

goroutine 83 [chan receive, 9 minutes]:
oxia/server.(*mockServerAddEntriesStream).Recv(0xc00000e3f0)
        /Users/andrasbeni/oxia/server/mock_test.go:37 +0x44
oxia/server.(*followerController).AddEntries(0x0?, {0x10547bce0, 0xc00000e3f0})
        /Users/andrasbeni/oxia/server/follower_controller.go:178 +0x5c
oxia/server.TestReadingUpToCommitIndex.func1()
        /Users/andrasbeni/oxia/server/follower_controller_test.go:152 +0x60
created by oxia/server.TestReadingUpToCommitIndex
        /Users/andrasbeni/oxia/server/follower_controller_test.go:152 +0x754

goroutine 105 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc0001b01d0, 0x0)
        /opt/homebrew/Cellar/go/1.19.3/libexec/src/runtime/sema.go:517 +0x16c
sync.(*Cond).Wait(0xc0001b01c0)
        /opt/homebrew/Cellar/go/1.19.3/libexec/src/sync/cond.go:70 +0xb0
oxia/server.(*quorumAckTracker).WaitForHeadIndex(0xc0004603c0, {0x105100bda?, 0x25?})
        /Users/andrasbeni/oxia/server/quorum_ack_tracker.go:120 +0xb4
oxia/server.(*followerCursor).runOnce(0xc00015a400)
        /Users/andrasbeni/oxia/server/follower_cursor.go:165 +0x438
oxia/server.(*followerCursor).run(0xc00015a400)
        /Users/andrasbeni/oxia/server/follower_cursor.go:122 +0x8c
created by oxia/server.NewFollowerCursor
        /Users/andrasbeni/oxia/server/follower_cursor.go:82 +0x7ac
FAIL    oxia/server     600.634s

【log】replace pkg's zerolog reference with slog

Currently we have upgrade golang's version to 1.21，we should use build in slog to replace it, and make zerolog as it's handler.

For server:

• reference slog in any pkgs
• initialize slog handler with zerolog when process initialize

For client:

• only use slog
• make the user to choice handler for slog

Support shard deletion in `batch.Manager`

In batch.Manager, in #40, there is no way to remove Batchers for shard that have be remove/split.

In internal.ShardManager it would be useful to be able to register a listener/callback or similar to enable batch.Manager to be able to register and listen for shard changes.