MIT 6.5840 (6.824) distributed-systems labs in Go: a full Raft consensus implementation — extended with PreVote and ReadIndex — plus a fault-tolerant KV service and a sharded KV store on top of it.

This repository contains my implementation of the MIT 6.5840 Distributed Systems lab series:

• Part A — leader election: randomized election timeouts, voting restricted to candidates with up-to-date logs.
• Part B — log replication: heartbeat-driven AppendEntries, index/term conflict handling with fast leader backtracking.
• Part C — persistence: term, votedFor, and the log are persisted at every change; votes only granted for safe (up-to-date) candidates.
• Part D — log compaction: state-machine snapshots; InstallSnapshot to catch up lagging followers.
• Extensions beyond the labs:
  • raft_prevote.go — PreVote: a node that times out first runs a pre-election without bumping its term or burning its vote, so a partitioned node cannot inflate the cluster's term; on partition recovery the old leader keeps its term.
  • raft_readindex.go — ReadIndex linearizable reads: the leader pins its commitIndex, confirms leadership with a quorum round of empty AppendEntries, waits until the index applies, then serves the read from the state machine — reads never go through the log.

The raft code is split by concern into raft_election.go, raft_replication.go, raft_persistence.go, raft_compaction.go, raft_application.go, raft_log.go, raft_prevote.go, raft_readindex.go. Design notes (in Chinese) are in src/doc/raft.md.

Get / Put / Append on a single raft group with exactly-once semantics: every client request carries a unique ID so retries can be deduplicated server-side, and state-machine snapshots keep the log short.

A small raft-replicated service that answers Join / Leave / Move / Query and reassigns shards to groups as evenly as possible.

The keyspace is split into shards served by groups; when shardctrler produces a new config (a group joined or left), a background task loop migrates the affected shards between groups, handles reconfiguration races, and cleans up moved shards.

Two Python helpers live in src/tools (and as copies next to the tests):

• dstest — stress-run tests: each argument is a regex matched against test names; every test runs for -n iterations (default 10), -p workers in parallel, optional --race, --loop for continuous runs.
• dslogs — colorize and filter the topic-tagged logs (VOTE, LEAD, TERM, CMIT, PERS, SNAP, …) emitted by the raft code.

Requires Go ≥ 1.22 (go.mod lives in src/).

cd src

# individual parts / whole labs
go test -v -race ./raft -run 'PartA|PartD'
go test -v -race ./raft ./kvraft ./shardctrler ./shardkv

# stress-run: 10 iterations of every matching test, in parallel, with the race detector
cd raft && python3 dstest 'PartA|PartB|PartC|PartD' -r

# colored, topic-filtered logs while debugging a test
go test -v -run TestReElectionPartA 2>&1 | python3 tools/dslogs --just LEAD,VOTE

src/
├── raft/           # consensus core (see above)
├── kvraft/         # fault-tolerant KV service
├── shardctrler/    # shard → group configuration service
├── shardkv/        # sharded KV store with live reconfiguration
├── labrpc/         # course-provided RPC plumbing (loss/reorder simulation)
├── labgob/         # course-provided gob codec (panic-safe for tests)
├── porcupine/      # linearizability checker used by the kv tests
├── models/         # KV model fed to the checker
├── doc/            # design notes (raft.md)
└── tools/          # dstest / dslogs developer tools

• Educational project for the MIT 6.5840 course; labrpc, labgob, and porcupine are course-provided infrastructure. Not affiliated with MIT.
• The course tests are reorganized/renamed here with Part* markers (e.g. TestInitialElectionPartA).