Distributed Key-Value Store

A distributed key-value store built from scratch in Go.

The project uses the Raft consensus algorithm to keep multiple nodes in sync while handling leader election, replication, persistence, failures, and recovery.

The main goal was to understand how distributed storage systems work internally instead of treating them as a black box.

What it does

The system runs as a cluster of nodes.

One node becomes the Raft leader. Writes go through the leader and are replicated to the other nodes. If the leader fails, the remaining nodes can elect a new leader and continue operating when a majority of the cluster is available.

The project also includes persistence, snapshots, log compaction, fault injection, authentication, metrics, and a command-line client.

Features

• Raft leader election
• Replicated writes
• Majority-based commits
• Automatic leader re-election
• Persistent write-ahead log
• Configurable WAL fsync policy
• Log segmentation and compaction
• State snapshots
• Strong and eventual consistency reads
• Dynamic node addition and removal
• Simulated network delays
• Simulated packet loss
• Simulated network partitions
• Token authentication
• HMAC request signing
• Rate limiting
• Audit logging
• Metrics
• Health and cluster status endpoints
• Command-line client
• Docker Compose cluster
• TLS support for Raft communication
• Raft and transport tests
• Benchmark programs

Architecture

                         Client
                           |
                         kvctl
                           |
                       HTTP API
                           |
             +-------------+-------------+
             |             |             |
          Node 1         Node 2        Node 3
          Leader        Follower      Follower
             |             |             |
             +-------------+-------------+
                           |
                    Raft Consensus
                           |
              +------------+------------+
              |            |            |
             WAL       Replication   Snapshots

Each node has its own persistent data.

Raft decides which node is the leader and which operations are committed. The key-value store applies committed operations to its local state.

Project Structure

kvstore/
├── cmd/
│   ├── server/        # Starts a KV store node
│   ├── kvctl/         # Command-line client
│   ├── bench_single/  # Single-node benchmarks
│   └── bench_suite/   # Benchmark suite
│
├── internal/
│   ├── api/           # HTTP API and middleware
│   ├── config/        # Server configuration
│   ├── observability/ # Logging and metrics
│   ├── raft/          # Raft implementation
│   ├── store/         # Key-value state machine
│   └── transport/     # Network and fault simulation
│
├── repair/            # Diagnostic and recovery tools
├── Dockerfile
└── docker-compose.yml

Requirements

• Go 1.22 or newer
• Docker and Docker Compose for running the multi-node cluster

Running a Single Node

From the kvstore directory:

go run ./cmd/server \
  -id n1 \
  -api 127.0.0.1:8001 \
  -raft 127.0.0.1:9001

The API will be available at:

http://127.0.0.1:8001

Running a 3-Node Cluster

The repository includes a Docker Compose configuration for a three-node cluster.

cd kvstore
docker compose up --build

The nodes expose these API ports:

Node 1 → http://localhost:8001
Node 2 → http://localhost:8002
Node 3 → http://localhost:8003

Each node also has a separate Raft communication port.

To stop the cluster:

docker compose down

To stop it and remove its persistent Docker volumes:

docker compose down -v

Using kvctl

kvctl is the command-line client included with the project.

Build it with:

go build -o kvctl ./cmd/kvctl

Check the node status:

./kvctl -url http://localhost:8001 status

Check node health:

./kvctl -url http://localhost:8001 health

Store a value:

./kvctl -url http://localhost:8001 put name "Likhith"

Read a value:

./kvctl -url http://localhost:8001 get name

Read using eventual consistency:

./kvctl -url http://localhost:8002 get name eventual

Delete a value:

./kvctl -url http://localhost:8001 delete name

List stored keys:

./kvctl -url http://localhost:8001 list

View cluster information:

./kvctl -url http://localhost:8001 cluster

View replication status:

./kvctl -url http://localhost:8001 replication

Create a snapshot:

./kvctl -url http://localhost:8001 snapshot

Compact the Raft log:

./kvctl -url http://localhost:8001 compact

Check the client version:

./kvctl version

HTTP API

The key-value API is intentionally simple.

Store a value

PUT /name
Content-Type: application/json

{
  "value": "Likhith"
}

Read a value

GET /name

Strong consistency is used by default.

For an eventual-consistency read:

GET /name?consistency=eventual

Delete a value

DELETE /name

Check node status

GET /status

Check node health

GET /health

View the cluster

GET /cluster

View replication status

GET /replication

View metrics

GET /metrics

Consistency

The project supports two read modes.

Strong consistency

A strong read goes through Raft's ReadIndex mechanism before accessing the local state.

This ensures that the node has confirmed its leadership before returning the value.

Client
  |
  v
Leader
  |
  v
Raft ReadIndex
  |
  v
Local State
  |
  v
Client

Eventual consistency

An eventual read accesses the node's local state without performing the Raft coordination required for a strong read.

This reduces coordination overhead but means the returned value can be behind the latest committed value.

Client
  |
  v
Any Node
  |
  v
Local State

Writes and Replication

Writes are handled by the current Raft leader.

Client
   |
   v
Leader
   |
   +----> Follower
   |
   +----> Follower
   |
   v
Majority reached
   |
   v
Commit
   |
   v
Apply to KV store

A write is committed after Raft obtains the required majority.

This allows the cluster to continue making progress when a minority of nodes becomes unavailable.

For example, a three-node cluster can continue operating with one failed node because two nodes still form a majority.

Failure Handling

The project includes a simulated network layer so distributed-system failures can be tested without physically breaking the network.

It can simulate:

• Network delays
• Dropped messages
• Network partitions

For example:

Node 1  <---->  Node 2
  |
  X
  |
Node 3

A partition can prevent nodes from communicating normally.

The purpose is to test whether Raft correctly handles leader changes, lost messages, unavailable nodes, and recovery.

Persistence and Recovery

The system does not rely only on memory.

Raft state is persisted using a write-ahead log.

The project also supports:

• WAL fsync configuration
• Log segmentation
• Snapshots
• Log compaction
• State restoration after restart

The basic recovery flow is:

Disk
 |
 v
WAL / Snapshot
 |
 v
Raft State
 |
 v
Key-Value State
 |
 v
Server Ready

This allows a node to restart without simply losing its previous state.

Security

The API supports optional security controls:

• Token authentication
• HMAC request signing
• Request rate limiting
• Audit logging

Raft communication can also be configured to use TLS.

Authentication and signing credentials are supplied through configuration or environment variables rather than being stored in the source code.

Observability

The server exposes operational information through:

/status
/health
/cluster
/replication
/metrics

These endpoints provide information such as:

• Current Raft role
• Current term
• Known leader
• Quorum status
• Replication lag
• Runtime metrics

This makes it possible to inspect the cluster without attaching a debugger or manually reading internal state.

Testing

Run all tests:

go test ./...

Run tests with Go's race detector:

go test -race ./...

Run benchmarks:

go test ./... -bench=. -benchmem

The repository contains tests for Raft behavior, persistence-related functionality, deterministic execution, and simulated network failures.

Benchmarks

The repository includes dedicated benchmark programs under:

cmd/bench_single
cmd/bench_suite

These are used to measure the behavior of the store under different workloads.

Benchmark results are intentionally kept separate from the implementation so they can be reproduced locally rather than presenting machine-specific numbers as universal performance claims.

Configuration

The server supports configuration through command-line flags and environment variables.

Important options include:

-id
-api
-raft
-data
-peers

-election-min
-election-max
-heartbeat
-request-timeout

-fault-min-delay
-fault-max-delay
-fault-drop-rate
-fault-partitions

-auth-token
-signing-secret
-rate-limit-per-sec

-raft-tls-cert
-raft-tls-key
-raft-tls-ca

-log-segment-bytes
-snapshot-every
-snapshot-max-log-entries
-wal-fsync

For example:

go run ./cmd/server \
  -id n1 \
  -api 127.0.0.1:8001 \
  -raft 127.0.0.1:9001 \
  -data ./data

Why I Built This

I wanted to understand what actually happens inside a distributed storage system.

Instead of stopping at a basic map[string]string server, the project goes deeper into:

• Consensus
• Leader election
• Replication
• Persistence
• Failure recovery
• Consistency
• Network failures
• Observability
• Security

The project is primarily a learning exercise in backend engineering and distributed systems, with the implementation kept small enough to understand and experiment with.

Limitations

This is an educational distributed storage system, not a production replacement for systems such as etcd, TiKV, or Redis.

The implementation is deliberately focused on understanding the core mechanisms rather than covering every concern required by a production database.

Roadmap

• Client-side automatic leader discovery
• Automatic retry after leader changes
• More extensive chaos testing
• More recovery tests
• More benchmark scenarios
• Improved snapshot testing
• Additional consistency tests
• Versioned releases and prebuilt binaries

Tech Stack

Language       Go
Consensus      Raft
Storage        In-memory KV + persistent WAL
API            HTTP / JSON
CLI            Go
Testing        Go testing + race detector
Deployment     Docker / Docker Compose
Security       Token Auth + HMAC + TLS
Observability  Metrics + Audit Logging