001Notes

System Design Interview Notes

On this page149

Deep-answer interview notes for system design questions: APIs, caches, Kafka, consistency, disaster recovery, and Spanner.

15 Jun 2026Interview Notes37 min readmedium confidence

Article details

Status: Building Publicly
Subcategory: System Design
Last reviewed: 26 Aug 2026
Prerequisites: HTTP and networking basics · Databases · Concurrency fundamentals

149 sections

SD.1 — All types of API

REST

Resource-oriented, stateless over HTTP. URLs identify resources, verbs (GET/POST/PUT/PATCH/DELETE) describe actions, status codes describe outcomes.

GET    /users/123              → 200 + user JSON
POST   /users                  → 201 + new user
PUT    /users/123              → 200 (full replace)
PATCH  /users/123              → 200 (partial)
DELETE /users/123              → 204

Wins: ubiquitous tooling, cacheable (HTTP cache), maps naturally to CRUD, debuggable from a browser. Loses: chatty for graph-shaped data (N+1 round trips), over-fetching, versioning is awkward. HATEOAS (links embedded in responses) is the “true” REST that nobody does.

GraphQL

Single endpoint, client specifies exact shape of data wanted:

query {
  user(id: 123) {
    name
    posts(last: 5) { title comments(last: 2) { author { name } } }
  }
}

Wins: no over-fetching, one round trip for nested data, strong type system (schema), subscription support. Loses: harder to cache (every query is different), N+1 server-side resolvers (mitigated by DataLoader pattern with batching), authorization is fiddly per-field, hard to rate-limit.

gRPC

Binary protocol over HTTP/2. Schema-first via Protocol Buffers (.proto).

service UserService {
    rpc GetUser(GetUserRequest) returns (User);
    rpc StreamUsers(stream UserFilter) returns (stream User);
}

Wins: very fast (binary, HTTP/2 multiplexing, header compression), strong types, code-gen for many languages, four streaming modes (unary, server-stream, client-stream, bidi). Loses: binary → harder to debug with cURL, browser support requires gRPC-Web proxy, schema evolution rules to learn. Internal microservice mesh? gRPC. Public API? REST or GraphQL.

WebSocket

Bidirectional, persistent TCP connection upgraded from HTTP. Both sides can push at any time.

client → server: HTTP Upgrade: websocket
server → client: 101 Switching Protocols
[binary or text frames in either direction]

Wins: real-time bidirectional, low overhead per message, browser-native. Loses: proxies/load balancers must support it, no built-in request/response correlation (you build it), stickiness needed (each WS lives on one server).

Server-Sent Events (SSE)

One-way push from server to client over HTTP. Server keeps the response open and writes event: framed messages.

HTTP/1.1 200 OK
Content-Type: text/event-stream

data: hello

event: tick
data: {"time": 123}

Wins: simpler than WS for server-push only, auto-reconnect via EventSource, plays nice with HTTP infrastructure, lightweight. Loses: one-way, limited to ~6 concurrent connections per origin in browsers (HTTP/1.1 limit; HTTP/2 lifts).

Polling / long polling

Polling: client asks “anything new?” every N seconds. Wasteful. Long polling: client asks; server holds the connection until something happens or timeout. Better latency, easier infra than WS.

Mostly legacy now; SSE/WS are better when supported. Useful in restrictive networks (corporate proxies blocking WS).

Webhooks

Server-to-server push. You register a callback URL; the source POSTs to it on events (Stripe charges, GitHub commits).

Wins: push at the right moment, no polling. Loses: need a publicly reachable HTTPS endpoint, idempotency required (retries), signature verification needed (HMAC).

SOAP

XML envelope over HTTP (or other transports). WSDL describes the contract.

<soap:Envelope>
  <soap:Body><GetUser><Id>123</Id></GetUser></soap:Body>
</soap:Envelope>

Legacy. Still in banking, telecom, SAP. WS-* extensions (WS-Security, WS-Atomic) cover enterprise needs that REST never claimed to.

JSON-RPC / XML-RPC

Procedure-call style:

{"jsonrpc":"2.0","method":"add","params":[1,2],"id":1}

Simpler than SOAP, easier than REST when your API is verb-y rather than resource-y (Ethereum nodes, Solana, Bitcoin). Solid for internal RPC.

MQTT

Pub/sub over TCP, originally for telemetry. Tiny header (~2 bytes), QoS levels 0/1/2. IoT, telematics.

Comparison table

API style	Transport	Direction	Shape	Use case
REST	HTTP/1.1 or 2	request/response	client-defined URL, server-defined body	public API, CRUD
GraphQL	HTTP	request/response (+ WS subscriptions)	client-defined query	client-driven aggregation
gRPC	HTTP/2	uni or bi-streaming	proto-defined	internal microservices
WebSocket	TCP after HTTP upgrade	full duplex	freeform	chat, games, trading
SSE	HTTP/1.1 or 2	server → client	freeform text events	dashboards, notifications
Webhook	HTTP	server → server	provider-defined	event integration
SOAP	HTTP/other	request/response	strict XML	enterprise legacy
MQTT	TCP	pub/sub	tiny binary	IoT

Choosing

External, public: REST (or GraphQL if you have many disparate clients).
Internal microservices: gRPC.
Real-time bidirectional: WebSocket.
Server push only: SSE.
Event integration: webhook.
IoT/embedded: MQTT.

SD.2 — API versioning

Why version

You can’t break existing clients. Mobile apps, partner integrations, SDKs in the wild — they were written against your old schema. Backward-incompatible changes (renaming a field, changing a type, removing an endpoint) force everyone to update simultaneously, which is impossible at scale.

Strategies

URL path:

GET /v1/users/123
GET /v2/users/123

Advantages: minimal routing logic, visible versions in logs, and direct service routing. Costs: cache fragmentation because /v1 and /v2 are different keys, plus unstable resource URLs.

Header:

GET /users/123
Accept: application/vnd.acme.v2+json

Pros: same URL, content negotiation is “the HTTP way.” Cons: invisible from a browser, harder to debug, edge cases with caching headers.

Query parameter:

GET /users/123?api_version=2

Advantages: clients can set it without changing the URL. Costs: clients can omit it, and caches can mix versions unless the header participates in the cache key.

Subdomain:

GET https://v2.api.acme.com/users/123

Pros: clean cut between versions; allows totally separate deployments. Cons: TLS cert sprawl, CORS pain.

No version, additive only:

Never remove or rename a field.
New fields are added; clients ignore unknown fields.
Use feature flags or capability negotiation. Pros: zero versioning effort if disciplined. Cons: schema accumulates legacy cruft forever; you can’t ever clean up.

Semantic versioning

MAJOR.MINOR.PATCH: bump MAJOR on breaking change, MINOR on additive, PATCH on bug fixes. For APIs, only MAJOR is typically exposed in the URL/header.

Compatibility patterns

Backward-compatible additions:

Add a new field — old clients ignore it.
Add a new endpoint.
Add a new optional query parameter.
Loosen a validation rule.

Breaking changes:

Remove a field/endpoint.
Rename a field.
Tighten validation.
Change a field’s type or units.
Change pagination semantics.

Anything in the breaking list requires a major bump (or a parallel endpoint).

gRPC specifics

Protobuf has rules:

Field numbers are forever. Never re-use a removed field number — old clients would misinterpret.
New fields default-init in old clients (ints to 0, strings to "").
reserved 5; reserved "old_field"; documents abandoned numbers/names.
Don’t change a field’s type — even int32 to int64 can break across older readers.

Deprecation policy

Public APIs need a documented sunset policy:

Announce deprecation (banner in docs, response header Sunset: <date>).
Provide migration guide.
Define a sunset date (often 12–24 months out).
Monitor usage; reach out to laggards.
Finally, return 410 Gone (or redirect to new version).

GitHub, Stripe, Twilio all use this pattern.

Real-world example: Stripe

Stripe pins each API key to the version that was current at signup (or the version explicitly chosen). New behavior ships in a new dated version. Server forwards old requests through a chain of upgraders. Cost: every change has to come with a forward/backward compatibility shim. Win: clients never have to update for backend changes.

Counter-example: version-by-host pain

Splitting v1 and v2 onto totally different services means duplicating storage, auth, observability. If most differences are tiny, run them in the same service with version-aware handlers.

SD.3 — MongoDB vs others — read/write performance

What MongoDB is

Document database storing BSON (binary JSON) documents in collections. Schemaless (or weak schema enforcement). Built-in sharding, replica sets for HA.

Write performance

Internals:

WAL: writes hit an in-memory journal first (default j:true waits for journal sync).
WiredTiger storage engine: B+ tree with snapshotted reads, document-level locking (was collection-level in MMAPv1).
Replica writes: primary fsyncs WAL, replicates to secondaries asynchronously.
Write concern {w:1}: primary ack only. {w:majority}: majority of replicas ack.

Throughput numbers (single shard, m5.xlarge equivalent):

w:1, j:false: 50K–100K writes/sec (fire-and-forget, dangerous).
w:1, j:true: 5K–20K writes/sec (durable to primary).
w:majority, j:true: 2K–10K writes/sec (durable to majority).

Comparison:

DB	Typical writes/sec (single shard, durable)	Why
Postgres	5K–30K	row-level locks, WAL, MVCC
MySQL (InnoDB)	5K–30K	similar to PG
MongoDB	5K–20K	document-level locking
Cassandra	50K–200K	log-structured, no read-before-write on insert
DynamoDB	10K–40K per partition	provisioned, request-routed
ScyllaDB	100K–500K	C++ rewrite of Cassandra, shard-per-core

Cassandra/Scylla write fast because they’re log-structured merge (LSM) trees — append to a memtable, no random IO, compact later. Postgres/Mongo are B-tree which require random writes for index updates.

Read performance

Mongo reads:

Primary key (_id) lookups: O(log n), microseconds with index.
Secondary index lookups: O(log n) on the index, then random read of the document.
Range queries: index scan + document fetch (can be sequential if index covers).
Covered queries (all fields in the index): no document fetch, fastest.
Aggregation pipeline ( $match →$ group → …): can be slow without proper index.

Per-document read ~50 μs cached, ~5 ms cold (random disk read). Comparable to Postgres.

Comparison:

DB	Read characteristics
Postgres	full SQL, complex joins fast with planner, MVCC reads
Mongo	denormalized docs avoid joins, $lookup is slow
Cassandra	single-partition reads fast, cross-partition is fan-out
Redis	in-memory, sub-ms
DynamoDB	partition+sort key reads fast; scan is slow
Elasticsearch	full-text search, aggregations on terms

Why Mongo wins / loses

Wins when:

Documents map naturally to your domain (catalog items, user profiles, events).
You need flexible schema (rapidly evolving fields).
You scale by sharding (built-in router mongos).
Geo queries (2dsphere index).

Loses when:

You need ACID across multiple documents (Mongo has multi-doc transactions since 4.0 but they’re slower).
Heavy joins / relational queries.
Aggregates over huge datasets (use a warehouse or column store).
Strong consistency across shards under contention.

Sharding

Mongo shards by shard key. Choosing badly:

Monotonic keys (timestamp, ObjectId by default) → all writes hit the latest shard (“hot shard”).
Low-cardinality keys → uneven distribution.

Good keys: hashed or compound with cardinality, often {tenantId, _id}.

Replica sets

3+ nodes. Primary takes writes, secondaries replicate via oplog. Primary failure: election (consensus among voters) elects new primary in seconds. Reads can be sent to secondaries with readPreference=secondary (eventually consistent) or secondaryPreferred.

Disk layout

WiredTiger compresses pages (Snappy default, zstd optional). Compression ratio 2–4× typical. Indexes uncompressed.

Counter-example: don’t use Mongo for relational data

If your access pattern is “join users with orders with payments with shipments,” Postgres is faster and more flexible. Mongo’s $lookup is slow because it doesn’t pipeline; it materializes intermediate documents.

SD.4 — Caches

What caching is for

Caches trade memory for latency. A read from RAM is ~100 ns; from SSD ~100 μs; from network ~1 ms; from disk ~10 ms. Caching keeps hot data on the fast side.

Cache tiers in a typical web stack

Tier	Latency	Capacity	What
CPU L1	1 ns	32 KB	per-core, code+data
CPU L2	4 ns	256 KB–1 MB	per-core
CPU L3	12 ns	30 MB	shared per socket
OS page cache	50 ns	RAM	file contents
App in-process	100 ns	RAM	object cache
Local Redis/memcached	100 μs	per-node RAM	cluster cache
Distributed cache (cluster)	500 μs	aggregate RAM	hot data
Database	1–10 ms	TB	source of truth
CDN	10–50 ms (first hop)	global	static assets, API responses
Browser	0 (local disk)	per-user	per-user assets

Local vs global cache

Local (in-process): each app server has its own LRU map. Pros: zero network. Cons: duplicated entries across nodes, no shared invalidation.

Distributed (Redis/memcached cluster): one logical cache, sharded across nodes (often consistent-hashed). Pros: dedup, cluster-wide invalidation. Cons: network hop, partition tolerance trade-offs.

Hybrid (two-tier): local L1 + distributed L2. Local cache holds last 10K hottest items; falls back to remote. Common pattern: local hit (1 μs) → remote hit (300 μs) → DB (5 ms). Invalidation via pub/sub: a writer publishes “key X changed,” everyone evicts their local copy.

Write strategies

Strategy	Behavior	Trade
Write-through	write goes to cache AND DB synchronously	safe; slow writes; cache always coherent
Write-back	write to cache, flush later	fast writes; data loss on cache failure
Write-around	write goes to DB only; cache populated on read	safe for write-heavy data; first read slow
Read-through	cache miss fetches from DB and populates	cleanest separation
Cache-aside	app manages cache miss + repopulate	most flexible; most bug-prone

Eviction policies

LRU (least recently used): default for most. Approximated via clock or LRU-2.
LFU (least frequently used): better for skewed workloads with steady hot sets.
TLRU (TTL-aware LRU): expire by both time and access recency.
FIFO: simple, suboptimal.
Random: surprisingly competitive at high hit rates.
ARC (adaptive replacement, IBM): self-tunes between LRU and LFU. Used by ZFS.
W-TinyLFU (Caffeine library): admits new items only if frequency justifies; near-optimal.

Indexing inside the cache

For a row cache: index by primary key. For a query cache: index by query hash. Adding a query parameter creates a new key. For a list cache: store the list under a key like users:by_country:US:page:1 and invalidate explicitly on writes.

Hot data

Top-K access skew: 90/10 rule (90% of requests hit 10% of keys). The cache should fit the hot working set, not the entire dataset. Cache-fit > cache-size.

Measuring: redis-cli --hotkeys, sample request logs, or sample LFU counters in the cache itself.

Cache stampede / dogpile

Hot key expires → 1000 simultaneous misses → DB hammered.

Mitigations:

Locking: one request rebuilds, others wait. Redis SET NX EX lock.
Soft expiry: key has a soft TTL inside the value; the first request after soft expiry triggers async refresh while others still serve the stale value (Netflix’s pattern).
Probabilistic early expiration: P(refresh) ramps up as expiry approaches; spreads refresh load.
Background warming: materialized views recomputed on a schedule.

Negative caching

Cache the “doesn’t exist” answer too, so 404 lookups don’t pummel the DB:

SET user:9999 "MISSING" EX 60

Reduces DDoS-via-scanning.

Cache invalidation patterns

TTL only: simplest; tolerates staleness up to TTL.
Explicit invalidate on write: safest; complexity in coordinating writers.
Versioned keys: include a version in the key; bumping the version invalidates all. user:123:v5 → bump to v6.
Pub/sub invalidation: writer publishes change; subscribers evict.

Distributed cache consistency

Cache and DB are not in the same transaction. Race:

T1 reads X (miss), reads DB → 5
T2 writes X = 6 to DB, invalidates cache (no-op since not in cache yet)
T1 inserts 5 into cache  ← stale

Fix: invalidate AFTER cache populate, or use compare-and-set with version. “Write-then-invalidate” + bounded staleness is usually good enough.

CDN

Edge cache for static assets and (with care) API responses. Headers:

Cache-Control: public, max-age=3600 — let everyone cache.
Cache-Control: private, no-store — never cache.
Vary: Accept-Encoding, Cookie — split cache by these headers.
Etag — opaque version; If-None-Match revalidation.

CDNs (Cloudflare, Akamai, Fastly) also do edge compute, image resizing, JS execution.

Counter-example: cache the wrong thing

Caching /feed?user=123 for an authenticated user across CDN edge → cross-user data leak. Always partition cache key by tenant/user.

SD.5 — Kafka deep dive

What Kafka is

A distributed, partitioned, replicated commit log. Not a queue (though you can use it as one) — it’s an append-only log that consumers tail.

Topology

Producer ──┐                           ┌── Consumer (group A, instance 1)
Producer ──┤                           ├── Consumer (group A, instance 2)
           │                           │
           ▼                           ▼
       ┌─────────┐  ┌─────────┐  ┌─────────┐
       │ Broker1 │  │ Broker2 │  │ Broker3 │
       └─────────┘  └─────────┘  └─────────┘
            ▲           ▲           ▲
            └─── controller (KRaft or ZK)

Topics and partitions

A topic is a stream of records. Each topic is split into partitions — independent ordered logs. Ordering is per-partition, not cross-partition.

topic "orders":
  partition 0: [r0, r1, r2, r3, ...]
  partition 1: [s0, s1, s2, ...]
  partition 2: [t0, t1, t2, ...]

The number of partitions determines parallelism: one consumer per partition per group at peak.

Partition assignment by key

Producer hashes the record key (murmur2 mod P) to choose partition. Same key → same partition → ordered for that key.

producer.send("orders", key="user_123", value=order_json)

If key == null, round-robin (or sticky-partitioner since Kafka 2.4 — batches to one partition then rotates).

Brokers

A broker is a Kafka server. Each partition has:

One leader broker (handles all reads and writes for the partition).
N-1 followers that replicate from the leader.
The ISR (in-sync replica set): followers that are caught up within a threshold.

Replication

Producer sends to leader; leader writes to local log; followers fetch and replicate.

acks=0  → fire-and-forget; producer doesn't wait. Data loss on failure.
acks=1  → leader writes locally and ACKs. Leader crash before replication = data loss.
acks=all → leader waits for all ISR to ACK. Safest. min.insync.replicas=2 means at least 2 ISR must exist for write to succeed.

acks=all + min.insync.replicas=2 + 3 replicas = tolerates one failure with no data loss.

Log layout

Each partition is a series of segments (files):

/var/kafka/orders-0/
  00000000000000000000.log     # records 0..N
  00000000000000000000.index   # offset → byte
  00000000000000000000.timeindex # time → offset
  00000000000123456789.log     # records N..M
  ...

Segments are rolled by size (segment.bytes, default 1 GB) or time. Old segments are deleted per retention.ms or retention.bytes.

Producer guarantees

Idempotent producer (enable.idempotence=true): broker dedups by (producer_id, sequence_number). Default in modern Kafka.
Transactions: producer can write to multiple partitions atomically — all-or-nothing visibility, exactly-once semantics when combined with Kafka Streams or transactional consumer.

Consumer groups

group "billing":
  instance A → partitions 0, 1
  instance B → partitions 2, 3

Each partition is consumed by exactly one instance in a group. Add more instances → rebalance (partitions reassigned). Group coordinator manages this.

Offsets

Consumer maintains its position per partition. Stored in the internal __consumer_offsets topic.

At-least-once (default): commit after processing. Failure between processing and commit = duplicate on restart.
At-most-once: commit before processing. Failure = data loss.
Exactly-once: combine transactional producer + read-committed consumer + offsets stored in the transaction.

Why Kafka is fast

Zero-copy sendfile: broker → consumer goes from disk page cache to NIC without copying through user space.
Sequential disk IO: append-only log writes are essentially sequential, even on spinning disks (~100 MB/s on HDD, 1 GB/s on NVMe).
Page cache leverage: hot tail of log sits in OS page cache; consumers reading the tail rarely hit disk.
Batching: producers batch records (linger.ms, batch.size); broker treats batches as units; consumers receive batches.
Compression (compression.type): gzip/snappy/lz4/zstd. Producer compresses; broker stores compressed; consumer decompresses. Saves bandwidth AND disk.

Log compaction

Alternative retention mode: keep only the latest record per key. Used for state topics (changelog of a KV store).

key=user:1, value={name: "Alice", balance: 100}  ← discarded after newer one
key=user:1, value={name: "Alice", balance: 150}  ← kept (latest for key)
key=user:2, value=...

Tombstone (value=null) marks the key for deletion at the next compaction.

Coordinator (KRaft / ZooKeeper)

Old: ZK held cluster metadata, leader election, ISR state. New (KRaft, Kafka 3.3+): Kafka uses its own Raft for the metadata log — single system, simpler operations.

Common configuration pitfalls

Too few partitions → can’t scale consumers. Repartitioning later is hard (key→partition mapping breaks ordering).
Too many partitions → controller overhead, longer rebalances, more files (file descriptors).
max.in.flight.requests.per.connection > 1 + retries: out-of-order on retry. With idempotent producer this is safe.
retention.ms too short: consumers fall behind, miss records.
unclean.leader.election.enable=true: allows out-of-sync replica to become leader → potential data loss.

Mirror Maker / replication across clusters

Kafka MirrorMaker 2 (kafka-mirror-maker): cross-cluster replication. Used for DR (active-passive), aggregation (multi-region → central), or migration.

Confluent Cluster Linking: native cross-cluster replication preserving offsets.

When NOT to use Kafka

Low-volume, low-latency request/response → use gRPC or HTTP.
True message queue with per-message ack and priority → RabbitMQ.
Strict ordering across an entire topic — partition-1 is the only way, and that single partition is your throughput ceiling.

SD.6 — Redis vs Memcached vs others

Redis

In-memory data structure server. Keys map to typed values: string, list, set, hash, sorted set, stream, bitmap, hyperloglog, geo, JSON (module), vector (RediSearch).

SET k v EX 60
LPUSH q x; RPOP q
ZADD scores 100 alice; ZRANGE scores 0 -1 WITHSCORES
HSET user:1 name alice age 30
XADD events * type click user 123

Persistence: RDB snapshots, AOF (append-only file), or both. Configurable trade between durability and write throughput.

Replication: master → replica async; replica can serve reads.

Cluster: sharded across 16384 slots; client computes CRC16(key) % 16384, routes to the slot’s master. Multi-key ops only within the same slot (use {hash_tag}).

Threading model: single-threaded command loop (no concurrency overhead inside a node). Multi-threaded I/O in 6.0+ for network. Add more shards for CPU scaling.

Latency: ~100 μs typical command, ~1 ms tail. CPU-bound at ~100K ops/sec/core.

Memcached

In-memory key-value cache. Strings only; no data structures.

Multi-threaded (vs Redis single-threaded). Scales vertically by adding cores.
Slab allocator: classes of fixed sizes; reduces fragmentation but wastes memory if your value sizes don’t align.
LRU eviction built in.
No persistence — true cache only.
No replication — cache restart loses data.
Sharding is client-side (consistent hashing across nodes).

Latency: ~100 μs typical. CPU-bound at ~500K ops/sec/node with multi-threading.

Decision

Want	Use
Just a cache, raw KV, multi-core	Memcached
Data structures (queues, sets, sorted sets, streams)	Redis
Persistence	Redis
Pub/sub or streams	Redis
Rate limiting, locks, atomic ops	Redis (Lua, INCR, SETNX, Redlock)
Geographic / vector / search	Redis with modules

Other in-memory systems

Aerospike: hybrid memory + flash. Submillisecond, much larger working set than RAM. Used by ad tech.

ScyllaDB: C++ Cassandra. Shard-per-core architecture, no GC pauses. Beats Cassandra by 5–10× per node.

Hazelcast / Apache Ignite: distributed in-memory data grid. Compute + caching together. Used in finance.

KeyDB: Redis fork with multi-threaded core. Drop-in for Redis with better single-node throughput.

Dragonfly: modern Redis-compatible, multi-threaded, single-binary. Reports 5–25× throughput over Redis.

Use case examples

Session store: Redis with EX 86400.
Leaderboard: Redis sorted set with score = points.
Rate limiter: Redis Lua script: INCR + EXPIRE with check.
Distributed lock: Redlock (multiple Redis instances) or SET NX EX.
Cache + pub-sub invalidation: Redis PUBLISH channel msg on writes.

Counter-example: misusing Redis as a primary store

Redis with AOF + replication is durable, but it’s not a database. Schema is your responsibility, no SQL, no joins, no transactional consistency across keys without scripting. Treat as a fast cache and ephemeral store; back persistent data with a real DB.

SD.7 — Zookeeper

What it is

Distributed coordination service. A small, replicated, hierarchical filesystem (“znodes”) with strong consistency and watch notifications. Built for metadata, not data.

ZAB protocol

ZooKeeper Atomic Broadcast — like Paxos/Raft. One leader, followers replicate. Writes go through leader; majority quorum ack required for commit. Reads can be served by any node (eventually consistent unless you sync).

Guarantees

Linearizable writes: all writes seen in same order by all clients.
FIFO client order: a client’s operations are seen in the order issued.
Sequential consistency for reads (not linearizable by default).

Read-after-write requires sync() before the read, which forces leader contact.

Znodes

Persistent: survives client disconnect.
Ephemeral: deleted when client session ends — used for liveness (“I am alive while my znode exists”).
Sequential: ZK appends a monotonic counter to the name. Useful for queues and leader election.

/services/db/leader-0000000042   (ephemeral sequential)
/services/db/leader-0000000043

Watches

Client registers a watch on a znode; ZK sends one-shot notification on change. Watch is removed after firing — re-register to keep listening. Notifications are guaranteed to arrive before any further reads see the new state (so you never miss a transition).

Use cases

Leader election: every candidate creates /election/lock- (ephemeral sequential). Lowest-numbered child is the leader; others watch their predecessor.
Distributed locks: same idea — create ephemeral sequential, lowest wins, others wait.
Service discovery: services register their address as ephemeral znodes; clients list /services/foo/.
Config: small config blobs; clients watch for changes.
Group membership: ephemerals under /group/ track live members.

Limits

Data size per znode: 1 MB (configurable down). Not for blobs.
Throughput: tens of thousands of writes/sec/cluster — fine for metadata, terrible for data.
Number of znodes: hundreds of thousands per cluster, then performance degrades.
5-node ensemble is typical (tolerates 2 failures). Even sizes (6, 8) don’t add tolerance — same as 5, 7.

Alternatives

etcd (CoreOS): Raft-based, gRPC API, used by Kubernetes. Same role as ZK with cleaner API.
Consul (HashiCorp): coordination + service discovery + DNS + KV. More features, slightly more complex.
KRaft (Kafka): Kafka now embeds its own Raft for metadata, removing ZK.

When NOT to use

As a database: it’s metadata-sized.
As a queue at scale: there’s a ZK queue recipe but it’s bottlenecked.
For pub-sub: watches are one-shot, not a stream.

Counter-example: ZK as a generic KV

Storing a million 1KB keys in ZK kills throughput, fills memory (ZK keeps all state in memory), and rebalances slowly. Use it for “where is the leader” and small config; back actual data with a real DB.

SQS (Amazon Simple Queue Service)

Managed distributed queue, pull-based.

Standard queue: at-least-once, unordered. Effectively unlimited throughput.
FIFO queue: in-order per MessageGroupId, exactly-once. 300 msg/s without batching, 3K with.
Visibility timeout: pulled message is invisible to other consumers for the timeout; consumer must delete to confirm.
Retention: up to 14 days.
DLQ: messages exceeding max receives go to dead-letter queue.

When: decouple producers from consumers, smooth bursts, AWS-native, low ops overhead.

Pub/sub fanout. Publish a message; SNS delivers to all subscribers.

Subscribers: HTTP endpoint, email, SMS, Lambda, SQS, Kinesis Firehose.
Fanout to multiple SQS queues for processing parallelism.
FIFO topics now exist too.

When: notify many destinations on one event, mobile push, broadcast.

SNS+SQS combo: publish once to SNS topic, fan out to multiple SQS queues, each processed by a separate consumer group. Decouples producers from consumer count.

RabbitMQ

AMQP message broker. Producer publishes to an exchange; exchange routes to queues by binding rules.

Exchange types:

Direct: routing key = binding key.
Topic: routing key wildcards (logs.error.*).
Fanout: all bound queues.
Headers: match on message headers.

Consumer modes:

Push (basic.consume): server pushes; ack-based.
Pull (basic.get): polling.

Features:

Priorities, TTLs per message, dead-letter, deferred delivery.
Manual ack (durable processing), auto-ack (fast).
Cluster (mirrored queues, quorum queues since 3.8).

Throughput: tens of thousands msg/s per node, single-digit ms latency.

When: complex routing, priority queues, per-message TTL, request/response over messaging, work distribution.

Kafka

See SD.5. Persistent log, replay, partition-level ordering, very high throughput.

When: high-throughput streaming, event sourcing, replayable history, change-data-capture, analytics pipelines.

Comparison

Aspect	SQS	SNS	RabbitMQ	Kafka
Model	queue	pub/sub	broker (exchange→queue)	log
Pull/push	pull	push	both	pull
Ordering	FIFO queue only	FIFO topic only	per queue	per partition
Throughput	unlimited (std)	unlimited	10K–100K msg/s	millions msg/s
Latency	~10ms	~10ms	~1ms	~5ms
Replay	no	no	no (msg gone after ack)	yes (retention)
Persistent log	no	no	optional	yes
Routing logic	basic	topic filters	rich exchanges	partition by key
Ops burden	zero (managed)	zero	medium	high
Per-message ack	yes	n/a	yes	offset-based
Priorities	no	no	yes	no

Decision tree

Cloud-native, simple decoupling, AWS → SQS.
Broadcast / fan-out to many destinations → SNS (often with SQS subscribers).
Per-message routing, complex enterprise messaging, priority → RabbitMQ.
High throughput, replayable, event streaming, analytics → Kafka.
CQRS / Event Sourcing → Kafka (log is your source of truth).
Tasks with retries and deadlines → RabbitMQ or SQS depending on platform.

Counter-example: Kafka as a queue

Using Kafka as a per-message ack queue is awkward — Kafka commits offsets, not individual messages. If you need “process this message exactly once with retry-with-backoff per message,” RabbitMQ or SQS is cleaner.

SD.9 — AWS-based Semantic DB (vector search)

What semantic search is

Map text/images to vectors (embeddings) via a model. Similarity search by approximate nearest neighbor (ANN) — cosine similarity, dot product, L2 distance.

AWS options

Aurora PostgreSQL with pgvector:

CREATE EXTENSION vector;
Type: vector(1536) for OpenAI ada-002 size.
Indexes: HNSW (graph-based, fast queries, slower build), IVFFlat (partition + scan, faster build, slower queries).
Pros: full SQL alongside vectors (filter + ANN in one query), ACID. Cons: not as scalable as dedicated vector DBs at billion-scale.

SELECT id, content FROM docs
 WHERE tenant_id = 42
 ORDER BY embedding <=> '[0.1, 0.2, ...]'::vector
 LIMIT 10;

Amazon OpenSearch Service:

k-NN plugin with HNSW and IVF.
Combines lexical (BM25) and semantic search → hybrid retrieval.
Distributed, scales to billions of vectors with proper sharding.

Amazon DocumentDB: added vector search 2024.

Amazon Bedrock Knowledge Bases: managed RAG. Point at S3, choose a chunking strategy, choose an embedding model (Titan, Cohere). Bedrock indexes into OpenSearch Serverless / Aurora / Pinecone / Redis / Mongo Atlas. Query via API.

Amazon Kendra: older, NLP-heavy enterprise search. Not strictly vector, but semantic in spirit.

Pinecone (third-party, AWS Marketplace): managed vector DB.

Embedding pipeline

Document → Chunker → Embedding model → Vector + metadata → Vector DB
Query    → Embedding model → ANN query → Top K → (rerank) → LLM context

Models on AWS:

Amazon Titan Embeddings G1/V2 (1536 or 1024 dim).
Cohere Embed (1024 or 384 dim).
Open source (sentence-transformers) on SageMaker.

Tuning

Chunk size: 200–1000 tokens. Smaller = more precise retrieval; larger = more context per chunk.
Overlap: 10–20% chunk overlap helps with boundary-spanning facts.
Embedding dim: higher = more accurate, more memory, slower. Often 384–1536.
HNSW M / ef: M (neighbors per node) tunes recall; ef (search beam) tunes latency vs recall.
Filtering: pre-filter or post-filter. Pre-filtering with a tight filter can hurt recall; post-filtering may waste work. pgvector + indexed metadata = pre-filter cleanly.

Hybrid retrieval

Combine BM25 (keyword) with vector (semantic). Reciprocal Rank Fusion:

score(doc) = 1/(k + rank_bm25) + 1/(k + rank_vector)

Beats either alone in most RAG benchmarks.

Reranking

After top-K ANN retrieval, run a cross-encoder (e.g., Cohere Rerank, BGE-reranker) on (query, candidate) pairs. ~10ms per candidate but much higher precision.

Counter-example: ANN isn’t always right

For “find papers by author Smith” — that’s a keyword match, not semantic. Hybrid retrieval helps. Vector search alone underperforms BM25 on entity-heavy queries.

SD.10 — Collaborative service scaling (Sheets, Zoom)

Two different domains: text/state collaboration vs real-time media. Both need to scale to millions but the architecture differs dramatically.

Collaborative document (Google Sheets, Figma)

Core problem: multiple users edit the same document; merge concurrent edits without losing intent.

Operational Transformation (OT)

Each edit is an operation (insert, delete). The server transforms incoming operations against concurrent operations so the result is consistent.

Server state: "abc"
User A inserts 'X' at position 1: "aXbc"
User B inserts 'Y' at position 2 (in original): "abYc"
Server transforms B against A: position 2 → 3 (shifted by A's insert): "aXbYc"

Both clients converge. Used by Google Docs/Sheets, Etherpad. Server is authoritative; clients send ops, server transforms and broadcasts.

Pros: well understood, efficient. Cons: complex transformations grow with op types; centralized server needed.

CRDT (Conflict-free Replicated Data Types)

Operations commute; merging is associative + commutative. Each element has a unique ID; concurrent edits merge without coordination.

Examples: Yjs, Automerge, Riak DT. Used by Figma, Linear, Notion (partly).

Pros: peer-to-peer possible (no central server), offline-first, simpler than OT. Cons: payload overhead (IDs everywhere), tombstones for deletes, can be subtler about user intent.

Architecture pattern (OT/CRDT-based)

Client ─WS─► Edge ─►  Doc Service (shard by doc_id)
                         │
                         ├── In-memory document state
                         ├── Op log (Kafka or DB append-only)
                         └── Snapshots (S3 every N ops)

Sticky routing: doc_id → specific server (consistent hash). All ops for a doc go to one place to maintain ordering.
Op log: durable record of all operations. Snapshot to S3 every N ops for fast load.
Presence channel: who’s editing, where their cursor is. Separate pub/sub (Redis), eventual consistency OK.
Permission check: at WS open; cached for session.
Scale: shard by doc_id. Each doc = one shard’s responsibility. Add shards as docs grow.

Hot docs (one document with 10K viewers) need fan-out: read replicas, or a tree of presence proxies.

Real-time media (Zoom, Google Meet)

Topologies

Mesh (peer-to-peer): every participant connects to every other. Bandwidth scales O(N²). Works only for 2–4 participants.
MCU (Multipoint Control Unit): server decodes all streams, composites into one output, encodes, sends to each. CPU-heavy, bad for scale.
SFU (Selective Forwarding Unit): server receives each participant’s streams, forwards subsets to other participants without re-encoding. Bandwidth efficient, scales to ~50 participants per meeting per SFU. Industry standard for Zoom, Meet, Teams.

Simulcast / SVC

Each publisher sends multiple resolutions (simulcast: 3 separate streams) or one scalable stream (SVC, Scalable Video Coding). SFU forwards the layer matching the receiver’s bandwidth/window size. Lets a phone watch a lecture at 480p while a desktop watches at 1080p, with the publisher only encoding once.

Scaling SFUs

Sharding per meeting: each meeting lives on one SFU. Add SFUs to scale meetings, not within a meeting.
Cascading SFUs: for very large meetings (10K viewers), one SFU forwards to others which forward to participants. Tree topology.
Selective routing by region: route participant to nearest SFU; SFUs interconnect across regions.

Transport

WebRTC for client-server. UDP with SRTP encryption, congestion control via Google Congestion Control (GCC) or BWE. Fallback to TCP/TLS through TURN if UDP blocked.

Latency

Mesh: best (direct). 50 ms regional.
SFU: +20 ms (one server hop).
MCU: +50 ms (decode/encode).

Recording / VOD

Composite recording: an additional SFU client that records all streams (or composites). Streamed to S3 or live to RTMP.

Live streaming layer

Beyond N=50 active participants you typically switch to a passive viewer layer:

SFU broadcasts active speakers + a single composited mixed feed to RTMP/HLS.
Viewers consume the HLS feed via CDN.
Two-way participants stay in the SFU; thousands of viewers via standard streaming.

Counter-example: doc sharding granularity

Shard collab by user, not doc, and you create a hot spot when 100 users edit one doc — they’re spread across servers, ops fan out and ordering becomes hard. Shard by doc.

SD.11 — WebRTC, RTSP, RTP, WebSocket, SSE, polling, QUIC, UDP

RTP (Real-time Transport Protocol)

UDP-based protocol for delivering audio/video. Each packet carries: sequence number, timestamp, payload type, SSRC (source ID).

Why UDP: real-time media tolerates loss better than retransmission delay. Late packets are useless.

RTCP (companion): control protocol — reports loss, jitter, sender info.

SRTP: secure RTP. Encrypted payload + authentication.

RTSP (Real Time Streaming Protocol)

A control protocol for media streams. Think of it as remote control: PLAY/PAUSE/TEARDOWN over TCP. Media itself flows over RTP/UDP (or RTSP-tunneled TCP).

Used in IP cameras, surveillance, set-top boxes. Largely supplanted by WebRTC and HLS for new deployments.

WebRTC

Browser-native real-time media + data over UDP. Includes:

ICE (Interactive Connectivity Establishment): finds best path between peers using STUN (discover public IP) and TURN (relay through server when NAT prevents direct).
DTLS-SRTP: encrypts media.
RTP/RTCP under the hood.
Data channels: SCTP over DTLS for arbitrary binary data (game state, file transfer).
getUserMedia: camera/mic access.

Use case: browser-to-browser video calls, low-latency game state, peer-to-peer transfer.

Pain: NAT traversal, TURN servers for clients behind symmetric NAT (10-30% of users), complex signaling (you build it).

WebSocket

Persistent TCP connection, bidirectional, framed messages. Upgrade from HTTP.

GET /chat HTTP/1.1
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Key: ...

Server replies 101 Switching Protocols, the same TCP becomes a duplex frame channel.

Use case: chat, collaborative apps, trading dashboards, anything needing bidirectional persistent.

SSE (Server-Sent Events)

HTTP response that never closes. Server writes data: ...\n\n chunks. Client uses EventSource:

const es = new EventSource('/events');
es.onmessage = (e) => console.log(e.data);

Pros: simple, auto-reconnect, works through standard HTTP proxies. Cons: server → client only.

Polling

Client: GET /messages?since=1234
Server: 200 OK [messages]
sleep 5s
repeat

This is operationally simple but wastes polling requests. Use it when the network prevents server-initiated delivery.

Long polling

Server holds the request open until something happens or timeout (~30s). Mostly historical; SSE/WS are better.

QUIC

Transport protocol over UDP. Multiplexed streams, built-in TLS 1.3, 0-RTT connection resume, no head-of-line blocking like TCP.

HTTP/3 runs over QUIC. Browsers and big sites (Google, Cloudflare, Facebook) deploy it.

Why QUIC over TCP:

Connection establishment is 1-RTT (or 0-RTT for resumed); TCP+TLS is typically 2-RTT.
Multiplexed streams independent — one packet loss doesn’t stall others (TCP+HTTP/2 has HOL blocking at TCP level).
Connection ID survives IP change (mobile handoff). TCP would have to re-establish.
User-space implementation → faster iteration vs kernel TCP stack.

Cost: higher CPU than TCP (per-packet crypto), middlebox unfriendliness (some networks block UDP).

UDP congestion control

UDP itself has none — apps that use it heavily must implement their own:

BBR (Google): models bandwidth + RTT, drives toward optimal. Used by QUIC, YouTube.
CUBIC: TCP default; window grows cubically after loss.
GCC (Google Congestion Control): used by WebRTC. Adapts to packet loss + RTT changes.

Without congestion control, UDP apps cause collapse on shared links — RTP/SRTP rely on RTCP feedback for sender-side rate control.

Decision

Need	Use
Browser-to-browser video	WebRTC
Camera/IPTV control	RTSP
Bidirectional persistent app data	WebSocket
Server → browser stream of events	SSE
Fallback for restrictive networks	long polling
Modern web transport	HTTP/3 / QUIC
Streaming to passive viewers	HLS/DASH over HTTP

SD.12 — Inter-region consistency

The fundamental problem

Speed of light: New York ↔ London is ~30 ms one way, ~70 ms RTT. A strongly consistent write that requires majority across these regions takes a minimum of 70 ms.

Replication patterns

Single-region with async cross-region replicas

Primary in us-east-1; async replicas in eu-west-1 lagging seconds to minutes. Reads in eu-west-1 may be stale. On region failure: promote a replica, accept the lag as data loss.

Multi-master (active-active) with last-writer-wins

Each region has a writable copy. Writes asynchronously replicate. Conflicts resolved by timestamp (last writer wins). Examples: DynamoDB Global Tables, Cassandra cross-DC.

Pros: low write latency anywhere. Cons: conflicts can lose data silently; LWW is meaningful only if clocks are tight (NTP-grade ms might not be enough).

Multi-master with CRDT merge

Same architecture but merge instead of LWW. Counters, sets, sequences merge deterministically.

Multi-region with strong consistency

Spanner, CockroachDB, FaunaDB. Cross-region writes go through Paxos/Raft groups spanning regions; commit waits for majority.

Pros: linearizable across regions; no conflicts. Cons: write latency = inter-region RTT (often 100+ ms).

Tunable: pin the leader/Paxos majority to one region for low-latency writes there, with reads from other regions hitting their local replicas + a “stale read” tolerance.

Reading

Local read: always cheap. May be stale by replication lag.
Read with consistency token: client provides a token saying “I want to read at or after timestamp T.” If local replica isn’t caught up, it waits or forwards.
Bounded staleness: explicitly allow up to N seconds of staleness.
Strong read: round-trip to leader. Slow but always fresh.

Cross-region conflict handling

CRDT (sets, counters, registers, sequences): no conflict, always converge.
LWW: simple but lossy.
Vector clocks / dotted version vectors: detect conflicts; punt to app code to resolve.
Custom merge: shopping cart “union” semantics.

Read-your-writes guarantees

Critical UX: after writing, the user expects to read what they wrote. Patterns:

Send subsequent reads to the same region for a session.
Carry a write-timestamp; reads from any region wait for that timestamp to land.
Write-through cache: keep the just-written value in a local cache.

Region failure / failover

Active-active: lose data not yet replicated to the surviving region.
Active-passive: failover requires DNS/load balancer redirect + replica promotion. Total downtime depends on detection + promote time.
Quorum across 3+ regions: tolerate one region failure with zero data loss.

Concrete examples

DynamoDB Global Tables: multi-master, LWW. Eventual consistency. ~1 second cross-region propagation.
Aurora Global Database: physical replication, one writable region, several read-only replicas. Failover takes ~1 minute.
Spanner: synchronous Paxos across regions. Writes cost ~70–100 ms inter-region.
Cockroach: similar to Spanner without TrueTime; HLC-based.
Azure Cosmos DB: pick your consistency level per request (strong, bounded staleness, session, consistent prefix, eventual).

SD.13 — Disaster recovery per system

RTO and RPO

RTO (Recovery Time Objective): how long can you be down? (5 min, 1 hour, 1 day)
RPO (Recovery Point Objective): how much data can you lose? (0, 5 min, 1 hour)

Zero RTO + zero RPO is the most expensive setup. Define the actual business requirement before designing.

Tiers

Tier	RTO	RPO	Approach
Backup/restore	hours-days	hours	nightly snapshots; restore on DR site
Pilot light	hours	minutes	minimal infra always running; scale up on failover
Warm standby	minutes	seconds	scaled-down full stack always running
Multi-site active-active	seconds	zero (or near)	live traffic across regions

Per-system patterns

Postgres / MySQL

WAL streaming to standby (pg_basebackup + pg_walreceiver).
Read replica → promote on primary failure.
AWS RDS: Multi-AZ (synchronous standby in another AZ; failover ~60s). Cross-region read replicas (async; promote for DR).
WAL archived to S3 (archive_command) for point-in-time recovery.

Kafka

Cross-cluster mirror (MirrorMaker 2 or Confluent Cluster Linking) to a DR cluster.
Consumers on the DR cluster lag the source; on failover they continue from the last replicated offset.
Trade-off: throughput cost of replication; offset mapping complexity (MM2 uses checkpoint topics).

Redis

Replica in another AZ/region (async). On failover: redirect clients.
Persistence (AOF + RDB) for restart recovery.
For zero-data-loss + zero-downtime: Redis Cluster across AZs with synchronous fsync ≠ true (Redis doesn’t have synchronous replication; small RPO is inherent).

MongoDB

Replica set across 3 AZs.
Cross-region: use ZK-style 5-node sets (3+2) or dedicated DR replicas. writeConcern: majority keeps writes durable to majority before ack.

Cassandra/Scylla

Multi-DC native: define DCs in topology, replication strategy NetworkTopologyStrategy with replica count per DC.
Writes always asynchronous to other DCs (with tunable LOCAL_QUORUM consistency).

Object storage (S3)

Cross-region replication built in. Asynchronous, eventual.
Versioning + MFA-delete for ransomware protection.

DNS / load balancing

Route 53 health checks with failover routing policy.
Multi-region active-active with latency-based routing.

Application servers

Stateless services: deploy in multiple regions; LB across.
Stateful (session affinity): externalize session state to Redis / DB.

Testing DR

The most important part. Plans untested are plans that don’t work. Game days: deliberately fail a region in production. Chaos engineering (Chaos Monkey, Gremlin) makes this routine.

Backups vs DR

Backups answer “we lost data — restore it.” DR answers “the region is gone — keep running.” You need both.

Counter-example: replication lag = data loss window

DR setup with async cross-region replica + 30 second replication lag means you can lose 30 seconds of writes on regional failure. If business requires zero loss, you must use synchronous cross-region replication and eat the latency.

SD.14 — Eventual consistency

Definition

Given no further writes, all replicas of a value will eventually converge to the same state.

This is a liveness property (something good eventually happens), not a safety property (something bad never happens). Mid-flight, replicas can diverge arbitrarily.

CAP theorem reminder

Network partition → must choose:

CP: refuse writes that can’t be safely replicated. Linearizability preserved; availability lost.
AP: accept writes locally, reconcile when partition heals. Available; consistency degraded.

Eventual consistency systems are AP. Spanner-like systems are CP (they pause writes during quorum loss).

Anti-entropy mechanisms

How divergent replicas catch up.

Read repair

When a read sees inconsistent replicas, repair the lagging one in the background. Common in Cassandra. Free if reads are frequent; useless for cold data.

Hinted handoff

Coordinator can’t reach a replica during write → stores a “hint” locally. When the replica comes back, coordinator replays hints. Used by Cassandra, DynamoDB.

Merkle trees

Per-replica hash tree over the data. Periodically exchange roots; descend the tree to find which ranges differ; reconcile those. Cassandra nodetool repair. Dynamo paper section 4.6.

Gossip

Each node periodically gossips its state (or a digest) to a few random peers. Information spreads exponentially. Used for membership, schema versions, partition state. Compatible with anti-entropy.

Conflict resolution

When two replicas hold different “current” values for the same key:

Last-Writer-Wins (LWW): highest timestamp wins. Simple, lossy. Requires synchronized clocks (or logical clocks).
Vector clocks: detect causal vs concurrent. Concurrent → present both to the app; let the user/app decide (e.g., shopping cart union).
Dotted Version Vectors: improvement over vector clocks (no per-client entries).
CRDTs: designed so merge is deterministic and lossless. Counters, sets, sequences. Used in Riak, Redis CRDT, Yjs.

Tunable consistency

DynamoDB/Cassandra style:

Write at QUORUM (>N/2 replicas), read at QUORUM: linearizable for that key (since W+R > N).
Write at ONE, read at ONE: fast, eventually consistent.
Write at ALL: no failure tolerance.

Read-your-writes / monotonic reads / session guarantees

Pure EC is weak. Stronger guarantees within a session:

Read-your-writes: client sees its own writes immediately.
Monotonic reads: client never sees time go backward.
Monotonic writes: client’s writes are applied in order.
Causal consistency: causally related ops seen in causal order.

Implement via sticky sessions, version tokens, or vector clocks.

Examples of EC in the wild

DNS: records propagate over hours.
Email: delivery is async; you may see “sent” before recipient receives.
DynamoDB default reads: eventually consistent (cheap); strong-consistent reads cost more.
S3 eventual consistency (historical): list-after-write was eventual. Fixed to strong consistency in 2020.
Apache Cassandra: tunable per-op consistency.

Counter-example: assuming EC = “occasionally late”

Under partition, “eventually” can be hours or days. You must build the system to be correct with stale reads, not just “fast 99% of the time.”

SD.15 — Google Spanner

The big idea

Globally distributed, externally consistent (linearizable), SQL database that gives you ACID across regions. The trick that makes it possible: TrueTime.

TrueTime

A clock API that returns TT.now() = [earliest, latest] — a bounded interval. Spanner knows the true time is somewhere in there. Bound is typically ~7 ms.

Mechanism: every datacenter has GPS antennas + atomic clocks. Each server polls multiple time masters; the API integrates their reports + bounds the worst-case error.

Commit Wait

A transaction acquires a timestamp from TrueTime. Before committing, Spanner waits until TT.now().earliest > assigned_timestamp — i.e., until every other node in the world will see this timestamp as in the past.

This guarantees that if T1 commits before T2 starts (in real time), T1’s timestamp < T2’s timestamp. Hence external consistency: external observers see writes in real-time order.

Cost: writes pay one ~7 ms wait. Worth it for the guarantee.

Architecture

Universe = many zones (datacenters).
Each zone has spanservers managing tablets (key-range partitions of a table).
Each tablet’s replicas (typically 5) form a Paxos group spanning zones.
Cross-tablet transactions use two-phase commit across Paxos groups (one group is the coordinator).

Key features

SQL (with extensions): joins, indexes, transactions.
Schema changes are non-blocking and online.
Interleaved tables: child rows physically stored next to parent → cheap joins.
Read-only transactions: served from a snapshot at a chosen timestamp without acquiring locks. Can use a stale timestamp for performance.
Read-write transactions: 2PC + Paxos; expensive.

Performance

Single-row reads: ~few ms.
Read-only multi-row transactions: ~10–20 ms.
Single-row writes: ~10–50 ms (Paxos majority).
Multi-region transactions: 50–200 ms.

So: real ACID across continents, at a real (and predictable) cost.

Comparison with CockroachDB

Cockroach is open-source Spanner-inspired but uses Hybrid Logical Clocks (HLC) instead of TrueTime. HLC combines wall clock + logical counter; doesn’t require GPS/atomic clocks but loses Spanner’s bounded-error advantage.

Cockroach provides serializable isolation (not external/linearizable) by default. Can be configured stronger but at higher cost.

Comparison with FaunaDB

Uses Calvin protocol: deterministic transactions ordered via a single global log. Different theoretical model; similar in spirit (strong consistency at scale).

Comparison with regular RDBMS

Postgres/MySQL: single-leader, vertical scale, replicas async. Strong within one node; eventual across replicas.
Spanner: distributed leader sets, linearizable across the planet, much more complex ops.

When to use Spanner

Multi-region, globally distributed, ACID requirement (financial, regulated).
Workloads outgrowing a Postgres node.
Cloud (it’s Google managed; AWS-native equivalents: limited; Cockroach Dedicated runs on AWS).

When NOT to use Spanner

Single-region: a Postgres or Aurora cluster will be faster and cheaper.
Analytics: column stores (BigQuery, Snowflake) crush Spanner for scans.
Heavy writes with low budget: pay-per-operation pricing adds up.

Counterexample: treating TrueTime as a perfect clock

TrueTime gives bounded uncertainty, not zero uncertainty. The commit-wait is the price you pay. If your application can tolerate slightly stale reads (use stale-read timestamps), you can dodge the cost where it doesn’t matter.

Primary references

Google Site Reliability Engineering book