DEV Community: System Design Simple

Design Instagram's feed: system design in 5 steps

System Design Simple — Mon, 09 Nov 2026 14:07:59 +0000

A home feed must return the newest posts from every account a user follows, in time order, within a latency budget, while reads outnumber posts by about a hundred to one. The design question is where the work of merging those posts happens: on every read, on every write, or split between the two, and what each choice costs in operations, memory and tail latency. Interview question: "Design the home feed of a photo app for 100 million daily users."

Prerequisites

Requests per second and peak load. Capacity is planned for the busiest second: divide a daily count by 86,400, then multiply by a stated peak factor, as in the ticket sale spike that separates performance from scalability.

Big-O cost. O(log N) means the work grows with the logarithm of the input size: doubling N adds one step. It says how cost scales, not how long it takes.

Binary heap. An array kept as a tree where each parent is no larger than its children. Peek is O(1), pop and push are O(log k), building from k items is O(k).

Definitions

Definition

System design: deciding the components, their interfaces and where data lives, so the system meets stated requirements at a stated load.

Definition

Functional and non-functional requirements: what a user can do (post, follow, read) versus how well it must be done (latency, availability, freshness, scale).

Definition

Timeline: the ordered list of post ids a feed is built from, computed per request or stored per user.

Definition

Fan-out on read (pull): build the feed at request time by reading the recent posts of every followed account and merging them.

Definition

Fan-out on write (push): when a post is published, insert its id into the stored timeline of every follower, so a feed read is one range lookup.

Definition

Write amplification: storage writes caused by one logical write; under push, the author's follower count.

Definition

Cursor pagination: each page returns an opaque token naming the last item served; the next request asks for items strictly after it.

Definition

Percentile latency (p50, p99): the latency that 50% or 99% of requests are at or below.

How it works: the design process

System design is an engineering loop: what must it do, how well, how much, draw it, measure it. "Draw it" is four artefacts: an API, a data model, a high-level architecture and a deep dive on the one part the numbers say is hard.

The five questions. Step four, draw it, is the API, data model, architecture and deep dive below.

1. Requirements

Functional: publish, follow and unfollow, read the newest posts from followed accounts in reverse time order, 20 per page; stories, likes and search are out of scope. Non-functional (assumed): feed p99 under 200 ms at the server, a new post visible within a few seconds, and the feed readable while the write path is degraded, which ranks availability above freshness. Testable targets are covered in turning functional and non-functional requirements into SLOs.

2. Estimates

With the assumed inputs of the worked example, the peak is about 35,000 feed reads and 347 posts a second. The ratio, about 100 reads per post, is the most important number in this design: it says the expensive work should move to the rarer operation.

3. API

GET  /v1/feed?limit=20&cursor=<opaque>
     200 {"items": [{"post_id": "...", "author": {...}, "created_at": "...",
                     "caption": "...", "media_url": "..."}],
          "next_cursor": "eyJ0cyI6MTcwMDAwMDAwMDAwMCwiaWQiOjQyfQ"}
POST /v1/posts                      201 {"post_id": "..."}
PUT  /v1/users/{id}/following/{author_id}     204
DELETE /v1/users/{id}/following/{author_id}   204

The cursor is opaque (base64 of a timestamp and id) so its encoding can change; limit is capped server side; the response carries hydrated posts, so the phone makes one round trip.

4. Data model

posts     (post_id BIGINT PRIMARY KEY, author_id BIGINT, created_at TIMESTAMPTZ,
           caption TEXT, media_key TEXT, deleted_at TIMESTAMPTZ NULL)
           INDEX (author_id, created_at DESC, post_id DESC)
follows   (follower_id BIGINT, followee_id BIGINT, created_at TIMESTAMPTZ,
           PRIMARY KEY (follower_id, followee_id))
           INDEX (followee_id, follower_id)      -- "who follows X", for fan-out
timeline  Redis sorted set  timeline:{user_id}
           member = post_id, score = created_at in milliseconds   -- derived, rebuildable

Posts and follows are the source of truth; the timeline is derived data that can always be rebuilt from them. It stores ids, not content, so an edit needs no fan-out; the read path hydrates 20 ids in one multi-get. The follow graph is indexed both ways: "whom do I follow" for pull, "who follows this author" for push.

5. High-level design and 6. deep dive

Read path: client, load balancer, feed service, timeline store, post cache. Write path: post service, queue, fan-out workers, timeline store. The deep dive is the fan-out between them, where the estimates put almost all of the cost; the next three sections are that deep dive.

7. Measure

Every cost above is a model. The last step replaces guessed costs with measured ones, one component at a time, and revises the design when a number moves.

Fan-out on read vs fan-out on write: the cost model

Fan-out on read: a k-way merge per request

Each author's posts are sorted newest first by the (author_id, created_at DESC) index, so a feed is a k-way merge of k sorted lists, k being the follow count.

Fan-out on read: the heads of the followed accounts' sorted lists merge into the top 20.

Merging all N items of k lists costs O(N log k). A feed needs only the top n, so the merge stops early: O(k) to build the heap plus n pops and pushes at O(log k) each, O(k + n log k) in total. With k = 150 and n = 20, log₂ 150 ≈ 7.2, so the merge is roughly 150 + 2 × 20 × 7.2 ≈ 439 elementary heap steps, against sorting all 150 × 20 = 3,000 candidate posts. The CPU is cheap; the cost is k separate list reads per feed, each a storage or cache round trip.

Fan-out on write: write amplification

Under push, a post is inserted into each of the author's f timelines: write amplification is f. An insert into a sorted set of length L is O(log L), so one post costs O(f log L); a feed read is one range query, O(log L + n).

The cost formulas and the crossover

Let R be feed reads per second, W posts per second, F the average follows per user and f̄ the average followers per author. Every follow edge has one follower and one followee, so across the whole graph f̄ = F. Counting storage operations per second:

fan-out on read    cost_read  = R × F × c_list  +  W × c_post
fan-out on write   cost_write = W × f̄ × c_ins  +  R × c_range

with f̄ = F, c_range ≈ c_list and c_post ignored, push wins when
   W × F × c_ins  <  R × (F − 1) × c_list
   R / W  >  (F / (F − 1)) × (c_ins / c_list)  ≈  c_ins / c_list   (F large)

Here c is the cost of one operation of each kind: push wins whenever reads per post exceed the cost of an insert relative to a list read. With R / W ≈ 100 and similar unit costs, push does about 52,083 inserts plus 34,722 range reads a second, about 86,805 operations, against about 5.2 million list reads: roughly 60 times fewer. Where reads and writes are close, the simpler pull model wins.

The per-author view makes this clearer. An author with f followers posting p times a day costs p × f inserts a day under push; under pull, f followers reading r times a day cost f × r list reads. Push wins when p × c_ins < r × c_list, and f cancels out. Follower count therefore has no say in which model does fewer operations, only in the shape of the work.

The pull cost: 150 list reads per feed at 35,000 feeds a second.

Timeline storage: Redis sorted sets and cursors

The commands

A Redis sorted set holds unique members, each with a floating-point score, kept in score order. Redis implements it as a skip list plus a hash table, so an add is O(log N) and a range read needs no sort. One key per user, member = post id, score = creation time in ms:

# redis-cli
ZADD timeline:42 1700000000123 9001          # O(log N): insert, or update the score
ZREMRANGEBYRANK timeline:42 0 -501           # keep only the newest 500 (ranks are ascending)
ZRANGE timeline:42 0 19 REV WITHSCORES       # newest 20; same as the older ZREVRANGE
ZRANGE timeline:42 (1700000000123 -inf BYSCORE REV LIMIT 0 20   # next page after a cursor

ZRANGE is O(log N + M) for M returned elements; since Redis 6.2 its REV and BYSCORE options replace the deprecated ZREVRANGE and ZREVRANGEBYSCORE. ZREMRANGEBYRANK with ranks 0 to -501 deletes everything except the 500 highest scores, which caps the timeline length. A worker pipelines both per follower, so the cap holds on every insert.

Watch the precision of scores. They are IEEE 754 doubles, exact for integers only up to 2⁵³ ≈ 9.0 × 10¹⁵. A millisecond timestamp (about 1.7 × 10¹²) fits; a 64-bit time-ordered id does not, and ids above 2⁵³ collapse onto neighbours. Keep the id as the member.

Memory estimate

raw post ids only          100,000,000 users × 500 × 8 B    =   400 GB
id + score                 100,000,000 × 500 × 16 B         =   800 GB
skip list encoding         100,000,000 × 500 × ~80 B (assumed per-entry overhead)
                                                            = 4,000 GB
compact listpack, cap 128  100,000,000 × 128 × ~20 B (assumed)
                                                            =   256 GB

Redis stores small sorted sets in a compact listpack while they stay under zset-max-listpack-entries (128 by default) and zset-max-listpack-value (64 bytes); above that it converts to the skip list. So the cap is a memory lever with a step in it: 500 forces the larger encoding, 128 keeps the compact one. Check the assumed per-entry figures with MEMORY USAGE timeline:42. A second lever: keep timelines only for recently active users and rebuild the rest with one pull merge.

Cursor pagination vs offset

Offset pagination (LIMIT 20 OFFSET 1000) has two defects. Cost: PostgreSQL still computes the rows OFFSET skips, so page 51 reads 1,020 rows to return 20. Correctness: if 3 posts arrive at the head between pages, offset 20 points 3 items earlier and you see 3 duplicates.

A cursor names the last item served; in SQL it is a row comparison the index answers directly:

SELECT post_id, created_at FROM posts
WHERE author_id = $1 AND (created_at, post_id) < ($ts, $id)
ORDER BY created_at DESC, post_id DESC LIMIT 20;

Each page is O(log N + n) at any depth, and new posts cannot shift it. Ties need care: Redis orders equal scores by member bytes, so a score-only cursor with an exclusive bound can skip a post from the same millisecond. Put score and id in the cursor, use an inclusive bound and drop ids at or before the cursor's.

Where posts and timelines live starts with the shape of the data, not with a product name: SQL vs NoSQL, choose the data shape first.

The hybrid, hot keys and invalidation

Why large accounts break pure push

Follower count cancels out of the total, not out of latency. One post by a 50,000,000-follower account is 50,000,000 inserts into 50,000,000 keys, all due at once. At the assumed peak insert rate of 52,083 a second that is 50,000,000 ÷ 52,083 ≈ 960 seconds of fan-out capacity: the last followers see it about 16 minutes late, and ordinary posts queue behind it.

The hybrid

Push for authors below a follower threshold (assumed 100,000), pull above it. A read is one range read of the stored timeline merged with the recent posts of the few large accounts followed (say 5, assumed): the same k-way merge with k = 6 instead of 150. Pick the threshold so that f ÷ (insert rate available to one post) stays within the freshness target. Public talks on timeline systems describe this shape, including the one in Further reading.

Hot keys

Pulling moves the load from many keys to one. If those 50,000,000 followers load the feed 10 times a day (assumed): 50,000,000 × 10 ÷ 86,400 × 3 ≈ 17,361 reads a second at peak on one key and one shard. Fixes: an in-process cache in each feed server (assumed 200 servers, 1 second TTL: at most about 200 shard reads a second), with request coalescing so only one request refills an expired entry, or replicas of the key read in turn, the idea behind scaling out behind a load balancer applied to one key. A pulled list is the same for every reader, so it caches well; per-user timelines cannot be shared.

Invalidation: deletes, unfollows, privacy

Any change to posts, follows or privacy can leave a timeline stale, and fanning a delete out costs the same amplification as the post. So the read path is the authority:

Delete or edit. Hydration drops ids whose post has deleted_at set; edits appear for free. A later ZREM only tidies the timeline; correctness does not depend on it.
Unfollow. Filter items against the current follow set at read, and let a worker remove that author's ids from one timeline.
Follow. Backfill the author's newest ids with ZADD on the follow event.
Block or private. Read-time filtering is mandatory: a stale entry here is a privacy leak.

A worker can insert a post just after an unfollow cleaned the timeline; the read filter makes that race harmless. Where a race must never give a wrong answer, such as a seat sold twice, you need an atomic conditional write, as in the race condition between two fans and one seat.

Ranking as a separate stage

A ranked feed keeps this pipeline as candidate generation (the newest 500 ids) and adds a scoring stage that ranks them and returns 20. The cursor then names a ranked snapshot, not a timestamp, and scoring needs its own latency budget; fan-out does not change.

Measuring: method, percentiles and the tail

Method

A benchmark is evidence only if its method is stated. The lab behind this article (labs/sd-3/run.mjs): write the guess down first (150 list heads at about 0.5 µs plus about 25 µs of JSON, rounded to 100 µs, so 10,000 feeds a second); fix the data from a seed (10,000 users, 20 posts and 150 follows each); check the heap result against a full sort; warm up 1.5 seconds so the JIT compiles the hot path; loop the handler for 4 seconds on one core and count; record the machine (Apple M2 Pro) and runtime (Node.js 22.13.1).

The guess is written down before the run.

The run gave 73,526 feeds a second, 1,000,000 ÷ 73,526 ≈ 13.6 µs per feed, or 7.35 times the guess. That error is on the safe side, but it is outside the factor of 3 an estimate is allowed. Intuition is weakest at micro-costs like these, so they need a run.

Measured: 73,526 feeds a second on one core.

Scope is part of the result: merge and serialisation only, in memory, with no HTTP, TLS, network or storage. The conclusion is narrow: the merge is not the bottleneck; measure storage reads next.

One component measured; network and storage are next.

Throughput is not latency: utilisation and queueing

A per-core rate measures capacity; response time also includes waiting in a queue. In the simplest queueing model (M/M/1: random arrivals, one server, exponential service times), mean response time is S ÷ (1 − ρ), where S is the service time and ρ the utilisation. With S = 13.6 µs: 27.2 µs at ρ = 0.5, 68 µs at 0.8, 136 µs at 0.9. Waiting grows without bound as ρ approaches 1, so plans size for about 50% utilisation; the lab also simulates a queue in front of the handler and reports p50 and p99 as load rises. Applying the lab's sizing rule of 50% utilisation: 34,722 ÷ (73,526 × 0.5) ≈ 0.94 cores of this handler at peak, against 34,722 ÷ (10,000 × 0.5) ≈ 6.94 from the guess.

p99 and fan-out: the tail at scale

A feed that waits for 150 list reads is as slow as the slowest of them. If each read independently has a 1% chance of exceeding its own p99, the chance that at least one of 150 does is 1 − 0.99¹⁵⁰ ≈ 77.9%. This effect, described in "The Tail at Scale", is a second argument for push: one read instead of 150. Where fan-out is unavoidable, use hedged requests (send a duplicate after the p95 delay, take the first answer) or serve partial results.

Worked example

The scenario from the video: a photo app, you follow 150 people, and opening the app shows the 20 newest posts from them. All inputs are assumed for teaching; none are any company's real traffic.

From a daily count to a peak rate.

ASSUMED   100,000,000 daily users · 10 feed loads each · peak 3× average
          150 follows each (so 150 followers on average) · 20 posts per page
          1 user in 10 posts once a day · timeline cap 500 · large account 50,000,000

reads     100,000,000 × 10                    = 1,000,000,000 feeds a day
          1,000,000,000 ÷ 86,400              ≈ 11,574 per second
          11,574 × 3                          ≈ 34,722 ≈ 35,000 per second at peak
posts     100,000,000 ÷ 10                    = 10,000,000 a day
          10,000,000 ÷ 86,400 × 3             ≈ 347 per second at peak
ratio     34,722 ÷ 347.2                      = 100 reads per post

pull      150 × 35,000                        = 5,250,000 list reads/s
          (150 × 34,722 unrounded             ≈ 5,208,333)
push      347.2 × 150                         ≈ 52,083 inserts/s
          plus 34,722 range reads/s
merge     log2(150) ≈ 7.2;  150 + 2 × 20 × 7.2 ≈ 439 heap steps, not a 3,000-post sort

big post  50,000,000 ÷ 52,083                 ≈ 960 s of peak fan-out
hot key   50,000,000 × 10 ÷ 86,400 × 3        ≈ 17,361 reads/s on one key
tail      1 − 0.99^150                        ≈ 77.9% of pulled feeds hit a slow read

MEASURED  73,526 feeds/s on one core (merge only, in memory)
          1,000,000 ÷ 73,526                  ≈ 13.6 µs per feed
          73,526 ÷ 10,000                     ≈ 7.35 × the guess
          34,722 ÷ (73,526 × 0.5)             ≈ 0.94 handler cores at 50% busy

Conclusion from the numbers: CPU for the merge is negligible; 5.25 million storage reads a second and a 77.9% tail exposure are not. Push for ordinary authors, pull for the few huge ones, and measure storage next.

Code

All three programs use only the Python standard library. First, fan-out on read: the early-stopping k-way merge with heapq, checked against a full sort. Save as feed.py and run python3 feed.py.

import heapq
import random
from itertools import islice

random.seed(42)

# Synthetic data: 1,000 users, 20 posts each, 150 follows each.
USERS, POSTS_EACH, FOLLOWS_EACH, FEED_SIZE = 1_000, 20, 150, 20

# timeline[user] = that user's own posts, NEWEST FIRST: (timestamp, post_id)
timeline = {}
for u in range(USERS):
    posts = [(random.randint(0, 604_800), f"p{u}_{k}") for k in range(POSTS_EACH)]
    posts.sort(reverse=True)            # newest first
    timeline[u] = posts

follows = {u: random.sample([v for v in range(USERS) if v != u], FOLLOWS_EACH)
           for u in range(USERS)}


def home_feed(user):
    """Fan-out on read: merge 150 newest-first lists, keep the top 20."""
    heap = []
    # 1. Put the NEWEST post of each followee on the heap.
    #    heapq is a min-heap, so store -timestamp to pop the newest first.
    for author in follows[user]:
        posts = timeline[author]
        if posts:
            ts, pid = posts[0]
            heap.append((-ts, author, 0))
    heapq.heapify(heap)                 # 150 items, O(150)

    feed = []
    # 2. Pop the newest, then push that author's next post. 20 times.
    while heap and len(feed) < FEED_SIZE:
        neg_ts, author, i = heapq.heappop(heap)
        feed.append(timeline[author][i])
        if i + 1 < len(timeline[author]):
            next_ts, _ = timeline[author][i + 1]
            heapq.heappush(heap, (-next_ts, author, i + 1))
    return feed


# Check against the slow, obvious answer: collect everything and sort.
everything = sorted((p for a in follows[0] for p in timeline[a]), reverse=True)
assert home_feed(0) == everything[:FEED_SIZE]

# The standard library can do the same merge in one line.
merged = heapq.merge(*(timeline[a] for a in follows[0]), reverse=True)
assert list(islice(merged, FEED_SIZE)) == everything[:FEED_SIZE]

print(home_feed(0)[:3])

Observe: it prints the three newest posts, and both asserts prove the heap equals the 3,000-post sort and heapq.merge.

Second, fan-out on write with the large-account threshold and the hybrid read. Save as fanout_write.py.

import heapq
from collections import defaultdict, deque
from itertools import islice

FEED_KEEP = 500        # ASSUMED: keep only the newest 500 post ids per feed
CELEBRITY = 100_000    # ASSUMED: at or above this many followers, skip fan-out

followers = defaultdict(set)    # author -> users who follow them
following = defaultdict(set)    # user -> authors they follow
feed = defaultdict(lambda: deque(maxlen=FEED_KEEP))  # user -> (ts, id), newest first
own_posts = defaultdict(list)   # author -> (ts, id), newest first


def follow(user, author):
    followers[author].add(user)
    following[user].add(author)


def publish(author, ts, post_id):
    """Fan-out on write: push the new post id into every follower's feed."""
    own_posts[author].insert(0, (ts, post_id))
    if len(followers[author]) >= CELEBRITY:
        return 0                        # celebrity: readers pull it later
    for user in followers[author]:
        feed[user].appendleft((ts, post_id))   # oldest falls off past 500
    return len(followers[author])       # how many writes this one post cost


def read_feed(user, n=20):
    """Hybrid read: the ready-made feed plus a small merge of celebrities."""
    celebs = [own_posts[a] for a in following[user]
              if len(followers[a]) >= CELEBRITY]
    merged = heapq.merge(feed[user], *celebs, reverse=True)
    return list(islice(merged, n))


for u in range(150):
    follow(u, "author_a")
print(publish("author_a", 1_700_000_000, "p1"))   # prints 150
print(read_feed(7))                               # [(1700000000, 'p1')]

Observe: publish returns 150, the write amplification of that post. Change CELEBRITY to 100 and it returns 0, yet read_feed(7) still shows the post, now through the read-time merge. In production the loop runs in a worker behind a queue partitioned by author, and timelines live in Redis.

Third, every number in the worked example, recomputed. Save as numbers.py; change an assumption and the whole chain updates.

import math
# ASSUMED inputs (teaching numbers, not any company's traffic)
DAU, LOADS, PEAK = 100_000_000, 10, 3
FOLLOWS, PAGE = 150, 20
POSTERS = 0.10            # 1 in 10 daily users posts once a day
CAP = 500                 # newest post ids kept per timeline
DAY = 24 * 60 * 60

reads_avg = DAU * LOADS / DAY
reads_peak = reads_avg * PEAK
posts_peak = DAU * POSTERS / DAY * PEAK
print(f"feed reads   avg {reads_avg:,.0f}/s  peak {reads_peak:,.0f}/s")
print(f"posts        peak {posts_peak:,.1f}/s")

# Fan-out on read: one list read per followed account per feed.
print(f"read model   {FOLLOWS * reads_peak:,.0f} list reads/s (rounded peak 35,000: {FOLLOWS*35_000:,})")
# Fan-out on write: average followers = average follows (each edge has two ends).
print(f"write model  {posts_peak * FOLLOWS:,.0f} timeline inserts/s + {reads_peak:,.0f} range reads/s")
print(f"reads per post  {reads_peak / posts_peak:,.0f}")

# Merge cost for the top PAGE of k lists: heapify k, then PAGE pops and pushes.
k = FOLLOWS
print(f"log2(150) = {math.log2(k):.2f};  ~{k} + 2*{PAGE}*{math.log2(k):.1f} = {k + 2*PAGE*math.log2(k):.0f} heap steps vs sorting {k*20:,} posts")

# One post from a 50M-follower account at the platform's peak insert rate.
big = 50_000_000
print(f"50M inserts at {posts_peak*FOLLOWS:,.0f}/s = {big/(posts_peak*FOLLOWS):,.0f} s")

# Timeline memory, three ways (bytes per entry are ASSUMED).
for name, entries, b in [("raw ids, 500", CAP, 8), ("id+score, 500", CAP, 16),
                         ("skiplist ~80 B, 500", CAP, 80), ("listpack ~20 B, 128", 128, 20)]:
    print(f"memory {name:22s} {DAU*entries*b/1e9:,.0f} GB")

# Hot key: a 50M-follower account read at request time.
hot = big * LOADS / DAY * PEAK
print(f"hot key reads at peak {hot:,.0f}/s; with 1 s local cache on 200 servers: 200/s")

# Tail amplification: one feed waits for all 150 list reads.
p_slow = 1 - 0.99 ** FOLLOWS
print(f"P(at least one of 150 reads beyond its p99) = {p_slow:.1%}")

# Measured (labs/sd-3, M2 Pro, Node 22.13.1, one core, in memory).
measured, guess = 73_526, 10_000
S = 1e6 / measured
print(f"measured {measured:,}/s = {S:.1f} us per feed; guess off by {measured/guess:.2f}x")
print(f"merge cores at peak, 100% busy: {reads_peak/measured:.2f}; at 50%: {reads_peak/(measured*0.5):.2f}; guess at 50%: {reads_peak/(guess*0.5):.2f}")
for rho in (0.5, 0.8, 0.9):
    print(f"M/M/1 rho={rho}: response {S/(1-rho):.1f} us")

Observe: it prints every figure in the worked example. The measured rate is an input, not measured by this script.

Failure modes and edge cases

Fan-out backlog. Workers fall behind. Alert on queue lag (age of the oldest unprocessed post), not length; separate queues by author size, add workers or lower the threshold.
Cold or evicted timelines. A missing key on read: pull-merge for that request and write the result back.
Timeline store loss. Timelines are derived, so rebuild them; plan that throughput, because a lost shard sends all its users to pull at once.
Out-of-order inserts. Retries arrive late. Scoring by creation time keeps order, and ZADD of an existing member only updates its score, so retries are idempotent.
Threshold crossing. Older posts sit in timelines, newer ones are pulled; both merge by score. Deduplicate by post id.
Hot shard. Track per-key request rates; use the local cache or replicas.
Deep scroll. Past the cap of 500, pull-merge backwards from the cursor: slow, but rare.

Trade-offs

Option	Good for	Costs you	Use when
Fan-out on read	No stored timelines; follows, unfollows and deletes are correct at once	F list reads per feed (5,250,000/s here); feed p99 set by the slowest of F reads	Small scale, or reads close to writes
Fan-out on write	One range read per feed; p99 of one read	Write amplification = followers; timeline memory (256 GB to 4 TB here, assumed overheads); stale entries to filter	Reads far exceed writes and no author has a huge following
Hybrid	Fast reads and bounded fan-out bursts	Two paths, a threshold to tune, a small merge and a hot key per large author	A social graph with a few enormous accounts
Offset pagination	Simple, jump to page N	Linear cost in depth; duplicates when the head grows	Static, small result sets
Cursor pagination	Constant cost per page; stable while new items arrive	No random page access; cursor needs a tie-breaker	Any feed or infinite scroll

Common mistakes

Mistake: Drawing boxes before the rates. Instead: Compute R ÷ W first; a ratio of 100 selects push.

Mistake: Blaming large accounts for more total operations. Instead: Follower count cancels out; the problem is the 960-second burst and the hot key.

Mistake: Copying post content into timelines. Instead: Store ids and hydrate, so edits need no fan-out.

Mistake: Paginating a live feed with OFFSET. Instead: A (score, id) cursor: constant cost, no duplicates.

Mistake: A 64-bit id as a sorted set score. Instead: Scores are doubles exact to 2⁵³; score by ms time.

Mistake: Quoting a micro-benchmark as capacity. Instead: Say what was measured, what was not, and what comes next.

In an interview

Clarify. Time order or ranked, page size, freshness, scope.
Assumptions. Derive about 35,000 reads and 347 posts a second, a ratio of 100.
Structure. API with a cursor, data model, read and write paths.
Deep dive. Pull vs push costs and the crossover, the hybrid, sorted sets with a cap.
Trade-offs. Memory vs read cost, freshness vs availability, hot keys, tail latency.
Wrap up. Measure storage p99, queue lag and memory per timeline first.

A 60-second answer

"Scope: post, follow, a time-ordered feed of 20, p99 under 200 ms, seconds of staleness allowed. At 100 million daily users, 10 loads each and a 3× peak, that is about 35,000 reads a second against about 347 posts, a ratio of 100. Pull costs 150 list reads per feed, 5.25 million a second, and inherits the slowest of 150 reads. So I push each post id into followers' Redis sorted sets, capped with ZREMRANGEBYRANK, about 52,000 inserts a second; a read is one ZRANGE plus a hydrating multi-get. Accounts over about 100,000 followers are pulled at read time behind a short local cache, since one 50-million-follower post is 16 minutes of fan-out. Cursors are (time, id); deletes and unfollows are filtered at read. Then I measure storage p99 and queue lag."

Practice

BeginnerConvert 1,000,000,000 feed loads a day into a peak rate with a 3× peak factor.

1,000,000,000 ÷ 86,400 ≈ 11,574 a second on average; × 3 ≈ 34,722, about 35,000 a second.

IntermediateWhat is the cost of producing the top n posts from k sorted lists with a heap, and why not sort?

O(k + n log k): build a heap of k heads, then n pops and pushes. Sorting all N candidates is O(N log N), mostly for posts never shown.

IntermediateWhy does offset pagination show duplicates on a live feed?

New posts arrive at the head, so every item shifts down. If 3 posts arrive between page 1 and page 2, offset 20 starts 3 items earlier and repeats them.

AdvancedPer author, follower count cancels out of the push vs pull operation count. Why use a threshold at all?

Because the work arrives as a burst: 50,000,000 inserts for one post is about 960 seconds of peak fan-out capacity, delaying that post and everyone queued behind it.

AdvancedEach list read has a 1% chance of exceeding its p99. What fraction of pulled feeds over 150 lists see at least one slow read?

1 − 0.99¹⁵⁰ ≈ 77.9%. The component's tail becomes the feed's typical case; push reduces the read to one lookup.

AdvancedWhy is 500 entries per timeline a worse memory choice in Redis than 128, beyond the 3.9× entry count?

Above zset-max-listpack-entries (128 by default) the set converts from a compact listpack to a skip list plus hash table with several times the per-entry overhead: 256 GB becomes 4 TB here, under assumed overheads.

AdvancedThe merge ran at 73,526 feeds a second on one core. How many cores does the feed service need?

Unknown from this run. For the merge alone, 34,722 ÷ (73,526 × 0.5) ≈ 0.94 cores at 50% utilisation. Network and storage were not measured.

Cheat sheet

Term or formula	One-line meaning
Design process	Requirements, estimates, API, data model, high-level, deep dive, measure
Read to write ratio R ÷ W	Push wins when it exceeds c_ins ÷ c_list (about 100 here)
K-way merge	O(N log k) for all items; O(k + n log k) for the top n
Write amplification	Writes caused by one post = follower count
ZADD	O(log N) insert or score update; idempotent per member
ZRANGE … REV / BYSCORE	O(log N + M) newest-first range; replaces ZREVRANGE
ZREMRANGEBYRANK key 0 -501	Keep only the 500 highest scores
Score precision	Double, exact to 2⁵³; score by ms time, not by 64-bit id
Cursor	(score, id) of the last item; constant cost, no duplicates
Hot key	One key read by many; local cache with a short TTL or replicas
M/M/1 response time	S ÷ (1 − ρ); plan near 50% utilisation
Tail amplification	1 − (1 − p)^k chance a fan-out of k waits on a slow part

System design requirements: SLOs, budgets, capacity

System Design Simple — Mon, 09 Nov 2026 14:04:27 +0000

A design is checked against its requirements, and "fast and reliable" cannot be measured, so it cannot decide anything. Requirements turn into indicators, targets, budgets and capacity numbers, and those numbers choose the storage, the consistency model and the node count. Interview question: "Design a ride app for one million rides a day. What do you pin down before you draw?"

Prerequisites

Rates per second. A server's load is measured in requests per second. A day has 24 × 60 × 60 = 86,400 seconds, so any daily count divided by 86,400 is its average rate: 1 million a day is 11.6 a second.

Peak versus average. The busiest window divided by the average is the peak factor, and capacity is sized for the peak plus headroom. The article on performance vs scalability for a ticket rush covers how a load balancer and a waiting room absorb a burst.

Percentiles. The mean hides the slow tail, so latency targets are percentiles: p99 is the value that 99% of measurements are at or below.

Definitions

Definition

Functional requirement: a behaviour the system must provide, stated as an observable action and its result. It is true or false for a given build.

Definition

Non-functional requirement (NFR): a quality the system must have while doing that (latency, availability, freshness, durability, correctness under retry). It is a matter of degree, so it needs a number, a measurement point and a window.

Definition

SLI (service level indicator): a measured ratio of good events to valid events, for example "match requests answered within 5 s ÷ all valid match requests", taken at a stated point such as the load balancer.

Definition

SLO (service level objective): an internal target for an SLI over a window, such as 99% over a rolling 30 days. SLA (service level agreement): a customer contract with a looser target and a penalty, such as service credits.

Definition

Error budget: the amount of failure an SLO permits, (1 − SLO) × window, counted in minutes or in failed requests. Overspending it moves priority from features to reliability.

Definition

Geospatial index: a structure that maps positions to cell ids so "drivers near this point" reads a few cells, not every driver.

Definition

Idempotency: an operation applied twice has the same effect as applied once. Payment APIs use an idempotency key: a unique id the client sends so the server recognises a retry.

How it works: from questions to numbers

Requirements work has four outputs, and each feeds a later decision. Functional requirements decide which components exist, scope which ones are left out, non-functional requirements how each is built (storage engine, replication mode, cache policy), and estimates how many you run and which one gets the deep dive.

Four outputs: what it does, what it will not, how well, how much.

Functional vs non-functional. A functional requirement can be checked with one request in a test environment. A non-functional one can only be checked statistically, over many requests and time. Break a functional requirement and a feature is missing; break a non-functional one and the feature exists but is too slow, wrong under retry or down at the busiest hour. Late NFRs cause most redesigns, because they can change the type of store as well as the endpoints.

Write functional requirements as actions with outcomes, including the failure outcome. "Rider requests a ride" is incomplete. "Rider requests a ride and, within the match SLO, either gets a driver or is told none is available" defines a timeout, a terminal state and something an SLI can count. "Rider pays" becomes "exactly once, even if the app retries".

Scope is a requirement too: four core actions kept, three features stated as out.

Scope with reasons. Ratings and chat do not touch the request, match, track, pay path. Scheduled rides need a durable timer service and supply planning, a separate design. Stating the reason invites an objection early, while changing scope is still cheap.

The question checklist. Ask only the questions whose answers would change a component:

Actors and actions: who calls the system, and which three to five actions matter?
Scale and shape: events per day, peak factor, growth per year, one region or many?
Latency: which action has a deadline, at which percentile, measured where?
Freshness: how old may displayed data be? This sets cache TTLs and acceptable replication lag.
Correctness: what must never be lost, doubled or reordered (money, seats, stock)?
Availability and degradation: downtime per month, and which partial outage is acceptable?
Read/write ratio: mostly read, mostly written, or both?
Retention and compliance: what is kept, how long, who may see it?
Background load: does anything send data while nobody is using the product, such as idle drivers?

SLIs, SLOs, SLAs and error budgets

An NFR becomes operational when it has four parts: what is counted (the SLI), where it is measured, the target and the window. The Google SRE workbook recommends SLIs as good events ÷ valid events, so every SLI sits between 0% and 100%.

Each gauge becomes an SLI with a target and a window.

NFR as spoken	SLI (good ÷ valid)	Measured at	SLO (assumed)
Find a driver quickly	matches answered (driver or "none available") within 5 s ÷ valid requests	API gateway	99% over 30 days
Car moves smoothly on the map	renders showing a position at most 8 s old ÷ all renders	rider app, sampled	99% over 30 days
Booking is up	bookings without a server error ÷ valid bookings	load balancer	99.9% over 30 days
Never charge twice	rides with exactly one capture ÷ paid rides	ledger, daily	100%: an invariant, no budget

Valid excludes requests the system should reject (malformed input, a client cancelling), or a bad client burns your budget. The measurement point decides what is visible: a server timer misses DNS, TLS and mobile network time; a client timer includes them but is noisy. The freshness target is 8 s, two update intervals, because jitter or one lost packet would break a 4 s target constantly. "Send every 4 s" is an input; the SLO measures what the rider sees.

Percentiles and the tail. If one action fans out to n parallel backend calls, each slow 1% of the time, it is slow whenever any call is: P(slow) = 1 − 0.99ⁿ. For n = 10 that is 9.6%, for n = 100 it is 63.4%. So a component inside the match path needs a tighter target (p99.9) than the user-facing SLO (p99).

Error budget arithmetic. A 30-day window is 30 × 24 × 60 = 43,200 minutes. The budget is (1 − SLO) × 43,200:

SLO        budget per 30 days            budget per 365 days
99%        0.01 × 43,200 = 432 min (7.2 h)      0.01 × 8,760 h = 87.6 h (3.65 days)
99.9%      0.001 × 43,200 = 43.2 min            0.001 × 8,760 h = 8.76 h
99.95%     0.0005 × 43,200 = 21.6 min           0.0005 × 8,760 h = 4.38 h
99.99%     0.0001 × 43,200 = 4.32 min           0.0001 × 525,600 min = 52.56 min
99.999%    0.00001 × 43,200 = 0.432 min (25.9 s) 0.00001 × 525,600 min = 5.26 min

Request-based budgets track user impact more closely, because 10 minutes down at 3 a.m. and at the evening peak are not equally bad: 30 days × 1,000,000 = 30,000,000 bookings, and 99.9% permits 0.001 × 30,000,000 = 30,000 failures. An error budget policy states what happens when it runs out, typically a freeze on risky releases.

Burn rate is spend relative to plan: 1 uses the budget in exactly 30 days. Spending 2% of it in one hour is 0.02 × 720 h = 14.4, which empties the budget in 720 ÷ 14.4 = 50 hours. Alerting on burn rate keeps pages proportional to user harm.

Availability of a chain. Components in series must all be up, so availabilities multiply. Gateway 99.99%, matcher 99.95% and database 99.99% give 0.9999 × 0.9995 × 0.9999 ≈ 99.93%, which is lower than the weakest part. Redundant copies in parallel fail only if all fail: 1 − (1 − A)ⁿ, so two 99.9% replicas give 1 − 0.001² = 99.9999%. That assumes independent failures; replicas sharing a power feed, a deploy pipeline or a config fail together, close to a single copy. Each added nine allows ten times less downtime and costs redundancy and slower releases, so pick the target from user need; the SRE book's "Embracing Risk" chapter argues this in detail.

Capacity estimation, step by step

Estimation is a pipeline of single multiplications or divisions, with each assumption labelled:

Actors to events. DAU (daily active users) × actions per user = events per day. Here rides per day are given; the hidden events are location updates.
Events to average rate. Events per day ÷ 86,400.
Average to peak. × peak factor (3 assumed).
Concurrency. Little's law: L = λ × W. Rides in progress = ride arrival rate × ride duration. This turns rates into open connections and memory.
Storage. Rate × bytes per record × retention × replicas.
Bandwidth. Peak rate × bytes on the wire (payload plus headers).
Nodes. ceil(peak demand ÷ (per-node capacity × target utilisation)) + 1 spare. Per-node capacity comes from a load test.

Why a target utilisation of about 50%. In the simplest queueing model (M/M/1) the queue wait is ρ ÷ (1 − ρ) service times at utilisation ρ: 1 at 0.5, 4 at 0.8, 9 at 0.9. 50% leaves room for a higher peak and a failed node.

Reference latencies. From the widely cited latency table (see the sources): a memory reference is about 100 ns, a random 4 KB SSD read 150 µs, a round trip inside one data centre 500 µs, a disk seek 10 ms, California to the Netherlands and back 150 ms. So an in-memory store is limited by round trips and request handling, not the write itself. 25,000 writes a second, each waiting for its own 0.5 ms round trip in turn, needs 25,000 × 0.5 ms = 12.5 s of waiting per second: use at least 13 concurrent connections, or pipeline 100 commands per round trip for 250 round trips a second.

See the ticket rush article on load balancing for spreading connections over gateway nodes, and why that fixes scalability rather than performance.

How the location load shapes the design

Average rates on one scale: bookings barely show next to location updates.

Read/write ratio picks the architecture. A read-heavy system is solved with caches and replicas: write once, serve many copies. A write-heavy one cannot be cached out of trouble; every write must land. The feed in fan-out on read vs fan-out on write is read-heavy. Location is the opposite: every online driver writes every 4 s, a watching rider reads about one update per update, and the matcher searches about 35 times a second. Most written values are never read before a newer one replaces them.

Split current position from history. They have opposite requirements:

Latest location: one value per driver, overwritten, useful for seconds. 100,000 drivers × 100 B ≈ 10 MB. The size is small and the rate is the problem, so it goes in an in-memory store such as Redis.
Trail history: append-only, rides only, for receipts and disputes. About 30 GB a day, written in batches to a log or write-optimised table.

TTL as a liveness signal. Store each driver's latest position with an expiry, for example 30 s (assumed, about seven missed updates). A phone that loses signal drops out of search by itself. Redis caveat: a geo index is one sorted set and expiry applies to whole keys, not members. So use a per-driver key with a TTL plus a sweep of the geo set by last-seen time, or one geo key per cell per short time bucket, so buckets expire whole.

Write coalescing and batching. Coalescing drops superseded values of one key: a phone that reconnects and flushes ten buffered points needs only the newest in the latest store. Batching groups many values into one write: buffering each ride's trail for 60 s turns 15 writes into one, so 10,417 points a second at peak become about 694 writes. A crash can lose up to 60 s of trail, fine if the fare comes from a durable event at ride end.

Geospatial indexing. "Drivers within 2 km" over 100,000 points is a scan unless positions are bucketed:

Geohash interleaves longitude and latitude bits and encodes 5 bits per base-32 character. A 6-character geohash is 30 bits, 15 for longitude and 15 for latitude: 360° ÷ 2¹⁵ by 180° ÷ 2¹⁵, about 1.22 km by 0.61 km at the equator. Points either side of a cell edge may share no prefix, so a search includes the 8 neighbours. Redis stores a geo index as a sorted set scored by a 52-bit geohash and scans the 1 + 8 cell ranges covering the shape (GEOADD, O(log N) per member; GEOSEARCH, O(N + log M)).
H3 covers the Earth with hexagons at 16 resolutions. Resolution 8 cells average 0.737 km², an edge of about 0.53 km. All six neighbours sit at the same centre distance, so a cell plus one ring (7 cells, about 5.2 km²) is a near-circular search area; k rings hold 3k(k + 1) + 1 cells.

The match radius picks the cell size: 1 to 3 km in a city points at geohash length 6 or H3 resolution 7 to 8, so a search reads a handful of cells instead of the whole fleet.

Consistency per data type

Set consistency for each kind of data rather than for the whole system, so data that tolerates staleness can stay on the cheap path.

One app, two consistency rules.

Data	Requirement	Consequence
Driver position	Stale-tolerant: 8 s old or one lost update is fine	In memory, last write wins, asynchronous replication, TTL
Ride state (requested, accepted, finished)	One driver per ride; transitions in order	Conditional update, as for one seat
Payment	Exactly once, durable, auditable	Idempotency key, strongly consistent store, append-only ledger
Trail history	Durable eventually; minutes of delay fine	Batched appends to a log or table

If an offer reaches two drivers, acceptance is the same race as a seat sale: one conditional update that checks and sets in one statement, explained in how an atomic update stops two fans buying one seat.

Payments add retries: a client that times out cannot know whether the charge happened. It sends one unique key per payment attempt with every retry; the server stores the key with the result and answers a repeat with the stored result. Stripe's API documentation on idempotent requests describes this contract. Write the key in the same transaction as the charge, or a crash between them reopens the double charge.

Worked example

The video's scenario: a ride app in one country, and a large venue emptying at night into a sharp peak. Scope: request, match, track, pay. Given: 1,000,000 rides a day; everything else is assumed and labelled.

The on-ride location arithmetic, every assumption labelled.

GIVEN    rides per day                        1,000,000
ASSUMED  ride length                          20 min = 1,200 s
ASSUMED  location interval                    4 s
ASSUMED  peak factor                          3
ASSUMED  drivers online at peak               100,000 (on a ride or idle)

RIDES      1,000,000 ÷ 86,400                 ≈ 11.6/s avg, × 3 ≈ 34.7/s peak
TRAIL      1,200 s ÷ 4 s = 300 points a ride; × 1,000,000 = 300,000,000 a day
           300,000,000 ÷ 86,400               ≈ 3,472/s avg, × 3 ≈ 10,417/s peak
IN RIDE    Little's law: 11.6/s × 1,200 s     ≈ 13,889 avg, × 3 ≈ 41,667 peak
           check: 41,667 ÷ 4 s                ≈ 10,417/s (matches TRAIL peak)
ALL GPS    100,000 drivers ÷ 4 s              = 25,000 writes/s at peak
           25,000 ÷ 34.7                      ≈ 720 location writes per new ride
LATEST     100,000 × 100 B (assumed per row)  = 10 MB in memory
HISTORY    300,000,000 × 100 B                = 30 GB a day
           (30 GB + 1 GB of 1 KB ride records, assumed) × 365 × 3 replicas ≈ 33.9 TB a year
INGRESS    25,000 × 200 B × 8 bits            = 40 Mbit/s at peak
SOCKETS    41,667 rides × 2 phones + 58,333 idle drivers ≈ 141,667 open
NODES      store:   25,000 ÷ (10,000 × 0.5) = 5, + 1 spare = 6   (10,000/s per node assumed)
           gateway: 141,667 ÷ (50,000 × 0.5) = 5.7 → 6, + 1 = 7 (50,000 sockets per node assumed)
BUDGET     booking SLO 99.9% over 30 days     = 43.2 min, or 30,000 failed bookings

Booking is about 35 writes a second, easy for one relational database. Location is 25,000 writes a second of data worthless after seconds. Bandwidth is trivial at 40 Mbit/s; the difficulty is the number of writes and sockets, so the deep dive goes to location.

Code

The calculator behind every number above: assumptions in one dictionary, the article's formulas as functions. Plain Python 3, no libraries.

# Requirements calculator for a ride app: load, storage, servers, SLO budgets.
# Every value in ASSUMED is a guess you say out loud and invite correction on.
# Plain Python 3, no libraries. Change one value, run again, watch what moves.
import math

ASSUMED = {
    "rides_per_day": 1_000_000,     # given by the interviewer
    "ride_minutes": 20,             # assumed average ride length
    "update_every_s": 4,            # assumed: driver phone sends GPS every 4 s
    "peak_factor": 3,               # assumed: busiest hour is 3x the average
    "online_drivers_peak": 100_000, # assumed: drivers online at peak, on a ride or idle
    "bytes_per_update": 100,        # assumed: stored row incl. index overhead
    "wire_bytes_per_update": 200,   # assumed: same update plus protocol headers
    "bytes_per_ride": 1_000,        # assumed: one ride record
    "replicas": 3,                  # assumed: copies of every stored byte
    "writes_per_node": 10_000,      # assumed: measured by a load test, one store node
    "conns_per_node": 50_000,       # assumed: open WebSockets one gateway node holds
    "target_util": 0.5,             # plan nodes at 50% so queues stay short
    "trail_batch_s": 60,            # history buffered per ride and flushed every 60 s
}
DAY = 24 * 60 * 60                  # 86,400 s
MONTH_MIN = 30 * 24 * 60            # 43,200 min in a 30-day window


def load(a):
    per_ride = a["ride_minutes"] * 60 / a["update_every_s"]
    rides_s = a["rides_per_day"] / DAY
    on_ride_peak = rides_s * a["ride_minutes"] * 60 * a["peak_factor"]   # Little's law x peak
    loc_peak = a["online_drivers_peak"] / a["update_every_s"]           # every online driver
    trail_s = rides_s * per_ride
    return {
        "updates per ride": per_ride,
        "trail updates per day": a["rides_per_day"] * per_ride,
        "trail updates/s (avg)": trail_s,
        "trail updates/s (peak)": trail_s * a["peak_factor"],
        "new rides/s (avg)": rides_s,
        "new rides/s (peak)": rides_s * a["peak_factor"],
        "rides in progress (peak)": on_ride_peak,
        "all location writes/s (peak)": loc_peak,
        "location writes per new ride (peak)": loc_peak / (rides_s * a["peak_factor"]),
        "latest-location store MB": a["online_drivers_peak"] * a["bytes_per_update"] / 1e6,
        "history GB per day": a["rides_per_day"] * per_ride * a["bytes_per_update"] / 1e9,
        "stored TB per year (replicas)": (a["rides_per_day"] * (per_ride * a["bytes_per_update"]
                                          + a["bytes_per_ride"])) * 365 * a["replicas"] / 1e12,
        "history writes/s batched (peak)": trail_s * a["peak_factor"] / (a["trail_batch_s"] / a["update_every_s"]),
        "ingress Mbit/s (peak)": loc_peak * a["wire_bytes_per_update"] * 8 / 1e6,
        "open connections (peak)": 2 * on_ride_peak + (a["online_drivers_peak"] - on_ride_peak),
    }


def nodes(demand, per_node, util):
    # N for the load, +1 so one node can fail at peak (N+1)
    return math.ceil(demand / (per_node * util)) + 1


def budget_minutes(slo, window_min=MONTH_MIN):
    return (1 - slo) * window_min


def serial(*parts):            # every part must be up
    return math.prod(parts)


def parallel(a, n):            # any one of n independent copies is enough
    return 1 - (1 - a) ** n


def p_some_slow(n, p_fast=0.99):  # one request fans out to n backends
    return 1 - p_fast ** n


def geohash_cell_m(length):    # cell size at the equator
    bits = 5 * length
    lon_bits, lat_bits = (bits + 1) // 2, bits // 2
    m_per_deg = 111_320
    return 360 / 2 ** lon_bits * m_per_deg, 180 / 2 ** lat_bits * m_per_deg


if __name__ == "__main__":
    a = ASSUMED
    out = load(a)
    for k, v in out.items():
        print(f"{k:<38}{v:>16,.1f}")
    print(f"{'store nodes (peak writes, N+1)':<38}"
          f"{nodes(out['all location writes/s (peak)'], a['writes_per_node'], a['target_util']):>16}")
    print(f"{'gateway nodes (connections, N+1)':<38}"
          f"{nodes(out['open connections (peak)'], a['conns_per_node'], a['target_util']):>16}")
    for slo in (0.99, 0.999, 0.9995, 0.9999, 0.99999):
        print(f"{'error budget min, 30 days, %.3f%%' % (slo * 100):<38}{budget_minutes(slo):>16,.2f}")
    print(f"{'failed bookings allowed at 99.9%':<38}{(1 - 0.999) * a['rides_per_day'] * 30:>16,.0f}")
    print(f"{'serial 99.99 x 99.95 x 99.99 %':<38}{serial(0.9999, 0.9995, 0.9999) * 100:>16.4f}")
    print(f"{'two parallel copies of 99.9 %':<38}{parallel(0.999, 2) * 100:>16.4f}")
    for n in (1, 10, 100):
        print(f"{'P(slow) fan-out to %d at p99' % n:<38}{p_some_slow(n) * 100:>15.1f}%")
    for n in (5, 6, 7):
        w, h = geohash_cell_m(n)
        print(f"{'geohash length %d cell, m' % n:<38}{w:>8,.0f} x {h:>6,.0f}")

Real output of python3 estimate.py:

updates per ride                                 300.0
trail updates per day                    300,000,000.0
trail updates/s (avg)                          3,472.2
trail updates/s (peak)                        10,416.7
new rides/s (avg)                                 11.6
new rides/s (peak)                                34.7
rides in progress (peak)                      41,666.7
all location writes/s (peak)                  25,000.0
location writes per new ride (peak)              720.0
latest-location store MB                          10.0
history GB per day                                30.0
stored TB per year (replicas)                     33.9
history writes/s batched (peak)                  694.4
ingress Mbit/s (peak)                             40.0
open connections (peak)                      141,666.7
store nodes (peak writes, N+1)                       6
gateway nodes (connections, N+1)                     7
error budget min, 30 days, 99.000%              432.00
error budget min, 30 days, 99.900%               43.20
error budget min, 30 days, 99.950%               21.60
error budget min, 30 days, 99.990%                4.32
error budget min, 30 days, 99.999%                0.43
failed bookings allowed at 99.9%                30,000
serial 99.99 x 99.95 x 99.99 %                 99.9300
two parallel copies of 99.9 %                  99.9999
P(slow) fan-out to 1 at p99                       1.0%
P(slow) fan-out to 10 at p99                      9.6%
P(slow) fan-out to 100 at p99                    63.4%
geohash length 5 cell, m                 4,892 x  4,892
geohash length 6 cell, m                 1,223 x    611
geohash length 7 cell, m                   153 x    153

What to observe: set update_every_s to 2 and the location rates double (50,000 writes a second, store nodes 6 to 11) while new rides stay at 34.7 a second. Set target_util to 0.8 and both node counts fall to 5, paid for in queueing delay. online_drivers_peak moves the store more than rides_per_day does: idle drivers are the hidden load.

A requirements template to fill top to bottom:

REQUIREMENTS: <system>

1. FUNCTIONAL (3 to 5 actions, each with its failure outcome)
   - Rider requests a ride; gets a driver or "none available" within the match SLO
   - Driver accepts; exactly one driver per ride
   - Rider and driver see the car position
   - Rider pays exactly once, even when the app retries

2. OUT OF SCOPE (with reason)
   - Ratings, chat: not on the core path
   - Scheduled rides: needs a durable timer service, separate design

3. SLOs (SLI = good / valid, where measured, target, window)
   - Match: answered within 5 s / valid requests, at gateway, 99%, 30 days
   - Freshness: shown position at most 8 s old / renders, in app, 99%, 30 days
   - Booking availability: non-5xx / valid, at load balancer, 99.9%, 30 days
     budget 43.2 min or 30,000 failed bookings per 30 days
   - Payments: one capture per paid ride, invariant, daily reconciliation

4. ESTIMATES (assumptions labelled)
   - Given 1,000,000 rides/day; assumed 20 min, 4 s interval, peak x3, 100,000 online drivers
   - Rides 11.6/s avg, 34.7/s peak; location 25,000 writes/s peak
   - Latest store 10 MB; history 30 GB/day; 33.9 TB/year with 3 replicas
   - Sockets about 141,667; store 6 nodes, gateway 7 nodes (N+1, 50% utilisation)

5. CONSISTENCY PER DATA TYPE
   - Position: stale-tolerant, in memory, TTL
   - Ride state: conditional update, one winner
   - Payment: idempotency key, strongly consistent, ledger append-only

6. OPEN QUESTIONS
   - Trail retention and who may read it
   - One region or many; peak per region

Failure modes and edge cases

The SLI counts the wrong thing. A match SLI counting only successful matches looks perfect in a supply shortage, because "none available" is excluded. Detect it by comparing requests with completed rides; count a fast "none available" as good and track supply separately.

Correlated failures. Replicas in one zone, or regions receiving the same bad deploy, fail together and break the parallel formula. Fix with separate failure domains and staged rollouts; until then, estimate with the single-copy figure.

A local peak above the global one. A venue emptying pushes one area far past the city's factor. Watch per-cell rates and size hot areas on their own.

Hot shards. Partitioning by cell puts downtown on one shard. Detect with per-shard command rate; use finer cells in dense areas, or shard writes by driver id and fan searches out to the shards holding the relevant cells.

Ghost drivers. If liveness depends on a sweep job that stalls, offline drivers stay searchable. Detect by offers timing out unanswered; fix with TTL liveness and a last-seen filter in the search.

Retries without idempotency. A timeout retry on a plain charge captures twice. Detect with daily reconciliation of captures per ride; scope the key per payment attempt, not per ride, so a deliberate second payment still works.

Trade-offs

Option	Good for	Costs you	Use when
Time-based SLO (bad minutes)	Simple to explain, matches SLA wording	An outage at 3 a.m. counts the same as one at peak	Low, steady traffic; contracts
Request-based SLO (bad requests)	Weights failures by real user impact	Needs good request counting at a stable point	Most user-facing APIs
Location every 2 s instead of 4 s	Smoother map, fresher matching	Twice the writes: 50,000/s at peak	Freshness SLO is being missed
Geohash cells	Built into Redis, simple prefixes	Uneven cell shapes, edge cases need 8 neighbours	Nearby search only
H3 cells	Equal-distance neighbours, good for area analytics	An extra library, conversion at the edges	Search plus supply and demand per area
Batched trail writes	About 15 times fewer storage writes	Up to one batch of trail lost on a crash	Trail is for display, fare comes from a durable event

Common mistakes

Mistake: Writing NFRs as adjectives ("fast", "highly available"). Instead: Give each an SLI, a measurement point, a target and a window.

Mistake: Treating SLO and SLA as the same number. Instead: Keep the SLO tighter, so a burning budget is caught before the contract is breached.

Mistake: Assuming redundant copies fail independently. Instead: Use 1 − (1 − A)ⁿ only across separate failure domains.

Mistake: Counting only the user-visible action. Instead: Count background writes: 35 bookings a second hide 25,000 location writes.

Mistake: Planning nodes at 100% of measured capacity. Instead: Plan at about 50% plus one spare; queueing delay grows as ρ ÷ (1 − ρ).

Mistake: Storing current location and its history in the same table. Instead: Latest position in memory with a TTL; the trail appended in batches.

In an interview

Clarify: actors, core actions, scope, and questions that change components.
Assumptions: say each one out loud and write "assumed" next to it.
Structure: functional list, out of scope, SLOs with windows, estimates.
Deep dive: the location path, where the numbers point.
Trade-offs: consistency per data type, interval vs cost, cell size.
Wrap up: what you would add, and which assumption to verify first (online drivers at peak).

The four slots filled before any component is drawn.

A 60-second answer

Functionally: request a ride and get a driver or a clear "none available", one driver per ride, live position, pay exactly once; ratings, chat and scheduled rides out. SLOs over 30 days: 99% of matches answered within 5 s at the gateway, 99% of map renders at most 8 s old, booking 99.9% available (43.2 minutes or 30,000 failed bookings of budget), and payments as an invariant with daily reconciliation. Estimates: one million rides a day is 11.6 a second, about 35 at a 3 times peak; with 100,000 drivers online sending every 4 s, location is 25,000 writes a second, about 720 per new ride. Bookings fit one relational database with conditional updates. The deep dive is location: latest position in memory with a TTL and a geohash or H3 index, the trail batched to durable storage, payments behind an idempotency key in a strongly consistent store.

Practice

BeginnerWhat is the error budget of a 99.95% SLO over 30 days?

0.0005 × 43,200 min = 21.6 minutes.

IntermediateGateway 99.99%, matcher 99.9%, database 99.99%, all in series. What is the end-to-end availability, and what limits it?

0.9999 × 0.999 × 0.9999 ≈ 0.9988, about 99.88%. The matcher limits it; improving the other two barely moves the result.

IntermediateWhy is the freshness SLO 8 s when phones send every 4 s?

Network jitter and the occasional lost update would break a 4 s target constantly. Allowing two intervals measures what the rider sees while tolerating one lost update.

IntermediateRides last 30 minutes instead of 20. How many rides are in progress at peak?

Little's law: 11.6 rides/s × 1,800 s ≈ 20,833 on average, × 3 ≈ 62,500 at peak.

Advanced2% of a 30-day error budget burned in one hour. What is the burn rate and how long until the budget is gone?

Burn rate 0.02 × 720 h = 14.4. At that rate the whole budget lasts 720 ÷ 14.4 = 50 hours.

AdvancedWhy can you not simply set a TTL on each driver inside a Redis geo set?

A geo index is one sorted set, and expiry applies to whole keys. Use a per-driver key with a TTL for liveness and sweep the set by last-seen time, or bucket geo keys by cell and short time window so whole buckets expire.

AdvancedA match call fans out to 20 cell shards in parallel, each slow 1% of the time. How often is the match slow, and what do you change?

1 − 0.99²⁰ ≈ 18.2%. Search fewer shards (partition so nearby cells share a shard), require p99.9 from each shard, or send a hedged request to a replica when a shard is slow.

Cheat sheet

Term or formula	Meaning
Functional requirement	An action and its outcome, checkable with one request
Non-functional requirement	A quality, checkable only statistically over many requests and time
SLI	good events ÷ valid events, at a stated measurement point
SLO	Internal target for an SLI over a window: 99% over 30 days
SLA	Customer contract, looser than the SLO, with a penalty
Error budget	(1 − SLO) × window: 99.9% over 43,200 min = 43.2 min
Burn rate	Share of budget used ÷ share of window elapsed; 14.4 empties 30 days in 50 h
Serial availability	A1 × A2 × … × An
Parallel availability	1 − (1 − A)ⁿ, independent failures only
Fan-out tail	P(slow) = 1 − pⁿ; 100 calls at p99 gives 63.4%
Little's law	L = λ × W: in progress = arrival rate × time in system
Node count	ceil(peak ÷ (per-node capacity × utilisation)) + 1
Queueing delay (M/M/1)	wait ≈ ρ ÷ (1 − ρ) service times: 1 at 50%, 9 at 90%
Geohash	Interleaved lat/lon bits, 5 per character; length 6 ≈ 1.22 × 0.61 km
H3	Hexagon cells, 16 resolutions; res 8 ≈ 0.737 km², ring k = 3k(k + 1) + 1 cells
Idempotency key	Client-made unique id; a repeat returns the stored result

Auto mode is now the default in the Claude apps: what it blocks, how to switch back

System Design Simple — Mon, 09 Nov 2026 08:35:46 +0000

If Claude has stopped asking you "Allow?" before every step, this is why. On October 22, 2026 Anthropic said that sessions in the Claude web, desktop and mobile apps would start in auto mode by default on Pro, Max and Team plans, the same default Claude Code got in August. Instead of you approving each action, a separate automated check does it. This review covers what that check blocks, the numbers Anthropic published, what people said, the honest limits, and how to switch back.

What Anthropic announced

The official post quoted the August announcement for Claude Code and extended it to the apps. It had about 2.8 million views and 21,000 likes as of November 4.

Post on X by @claudeai (official), 22 Oct 2026 (US time). Screenshot; open the original post.

Anthropic added the same news as a dated update on top of its auto mode article:

The October 22 update on Anthropic's auto mode article, captured November 4, 2026.

Two details in that update matter: it applies to new sessions, and to "most" Claude apps, not every surface. Enterprise organisations get an email before their default changes; the API and the cloud platforms stay opt-in for now.

How the check works

In manual mode, Claude stops and asks you before each tool action. In auto mode, Anthropic's article says each tool call goes through a classifier "targeted at blocking actions that are irreversible, destructive, or aimed outside your environment".

	Manual mode	Auto mode
Who decides each step	You click approve or deny	A separate classifier
Safe actions	Wait for your click	Go ahead
Risky actions	Wait for your click	Blocked; Claude looks for a safer route or asks you
When Claude is stuck	Keeps asking	Claude Code falls back to manual after 3 blocks in a row or 20 in a session
Rules you pinned yourself	Apply	Still apply; pinned defaults are not changed

The point is not that a model is more careful than you. It is that people stop reading prompts once there are many of them. Anthropic's data says users approve 97% of permission prompts, so the approve button had become a habit, not a check.

The numbers behind the change

Anthropic ran a controlled study with 1,053 paid developers who did not know which behaviour was being tested:

Anthropic's chart from its auto mode article, captured November 4, 2026.

89% vs 13.6%: auto mode caught 89% of dangerous commands; human review caught 13.6%.
800 vs 6: auto mode blocked 800 commands that humans had approved; humans blocked 6 that auto mode allowed.
Real sessions: among flagged high-severity cases, 6.3% of manually approved sessions contained an unrequested harmful action, against 2.4% in auto mode.
Prompt injection: in an independent evaluation of 720 attempts, none succeeded against Claude models in auto mode.

These are Anthropic's own reported figures. We have not reproduced them.

What auto mode does not promise

Anthropic is careful here, and so should you be. Its red-team testing with an outside firm lowered the classifier's miss rate on synthetic attacks from 12% to 7%, and the article says that 7% "should not be taken as the miss rate on real traffic". In other words: fewer mistakes than tired humans, not zero.

The clearest summary came from Addy Osmani, who works on Claude Code at Anthropic, in a long post about prompting (about 352,000 views):

From "You're probably overprompting Claude" by @addyosmani, 29 Oct 2026. Screenshot; open the original post.

That is the right mental model: neither mode is a guarantee, so the evidence matters. Ask Claude to show the command and its output, or a screenshot, before you accept that something is done.

How people reacted

We looked only at the posts with the most reach (as shown on X on November 4, 2026). We left out replies that were only about rival products.

Post	Reach	What it said
@claudeai (official)	2.84M views 21,036 likes	The announcement
@addyosmani (Anthropic)	353K views 2,833 likes	Auto mode is "not a guarantee of safe behavior", and neither is approving everything; ask for evidence
@erdyai (reply)	56K views 785 likes	Relief: no more clicking "allow" 40 times in one session
@testingcatalog	26K views 510 likes	Explains the change: Claude relies on its safety check instead of asking for every step
@0xerik (reply)	33K views 305 likes	Many developers were already using auto mode before it became the default

The pattern: the most-liked reactions were relief at fewer clicks. We found no high-reach post reporting a real incident caused by the new default; the strongest caution came from inside Anthropic itself, which is a good sign about how the change is being framed.

How to switch back, or pin a mode

Per session: the official post says you can "switch back to manual anytime". In the Claude Code CLI, press Shift+Tab to cycle permission modes; in the desktop app, use the mode menu.
Your own rules still win: deny rules and anything you pinned in your settings are not changed by the new default. If a command must never run, keep a deny rule for it.
For a team: admins can set an organisation-wide default with defaultMode in managed settings, or turn auto mode off with disableAutoMode.
For scripts: headless runs choose their mode explicitly with --permission-mode, so set it on purpose rather than relying on a default.

Two related changes in the same apps: Claude Duet, where Claude messages you first and still asks before it acts, and Claude self-assessment, where Claude reports possible misuse in its own conversations.

Our honest review

What is genuinely good. The change matches how people really behave. A prompt you approve 97% of the time is not protecting you; a check that blocks the dangerous 3% consistently is. Keeping your pinned rules untouched, falling back to manual when Claude is stuck, and warning Enterprise admins first are the right details.

What to watch. A default changes what new users never learn. People who start in auto mode may never see what Claude is actually running, so make a habit of asking for the command and output. And "most Claude apps" means the behaviour can differ by surface for a while; check the mode indicator before a session that touches anything important.

What we did. We use Claude Code daily to build this channel's videos. For anything that must never happen we rely on deny rules and hooks, not on the mode, which is the same advice Anthropic gives. We have not measured the classifier ourselves.

Verdict: a sensible default with honest numbers behind it. Keep it on for everyday work, keep your deny rules for the things that must never break, and ask for evidence before you trust a "done".

The full setup we use (permissions, hooks, headless runs) is in How to use Claude like a professional.

Sources: Anthropic's posts on X and its auto mode article (published August 7, 2026, updated October 22, 2026), captured November 4, 2026. Figures are Anthropic's own reported results. Reach figures are as shown on X that day. Public reactions are summarised with links to the original posts; short quotes are the posters' own words. Defaults can change after publication.

How to create an AI agent skill: one SKILL.md, reused by two agents

System Design Simple — Mon, 09 Nov 2026 08:35:09 +0000

To create an AI agent skill, make a folder with one file, SKILL.md: front matter with a name and a one-line description, then plain instructions. The description is the important part, because the agent reads only that line until a task matches it, and only then loads the rest.

This article is for developers who find themselves typing the same instructions into a coding agent every week: the same checks, the same file layout, the same warnings. The everyday problem is that the agent forgets all of it between sessions, so every run starts from zero and comes out slightly different. A skill writes those instructions down once, in a file the agent loads when the job comes up, so the work runs the same way each time and your teammates get the same behaviour.

We run 23 skills in the studio that makes this site, and the same files serve two different coding agents. Below is the format, a worked example, and the mistakes we made with our first skills.

The format

---
name: plain-english-check
description: Check a draft for plain English before publishing. Use when someone asks to review, proofread or simplify an article.
---

# Plain English check

1. Read the text once without editing.
2. Mark each AI tell: forced triads, staged openers, filler.
3. Rewrite only the marked lines. Keep every fact and number.
4. Check: no new claims, no lost numbers.

That is a complete, if small, skill (a shortened example, not one of ours word for word). In Claude Code, a project skill lives at .claude/skills/<name>/SKILL.md. Extra files (scripts, examples, a style guide) can sit in the same folder, and the steps can point to them. Project skills travel with the repository, so everyone who clones it gets them; keep personal habits out of the project folder and team rules in it. Check the skill into version control like code, and review changes to it in pull requests, because a changed step changes what every agent does next time.

Why does the description of a skill decide whether it loads?

An agent cannot read 23 long instruction files before every task; that would fill its context. So it keeps a short list of names and descriptions, and loads a full skill only when your request matches one.

The description does the routing. It is the line the agent matches your request against, so write it for matching, not as a summary of the file. Compare:

Description	What happens
"Video helper"	Rarely loads, because nothing matches it
"Render an approved episode into its Short. Use when the owner says make the video or render episode sd-N."	Loads exactly when someone says "make the video"

The second one is our real make-video skill. It names the job and the words people use to ask for it.

Matching a request to the right capability by a short description is also how MCP tool discovery works, where a client finds new apps with no changes to its own code.

One file, two agents

Our studio runs two coding agents. Claude Code reads .claude/skills natively. Codex CLI runs our cloud automation, and its prompt carries one extra line: when the task names a SKILL.md file, read it and follow it.

So the same make-video/SKILL.md (735 words) drives a render whether Claude or Codex does the work. When we change the steps, both agents get the change on their next run, because they read the same file. This is why we like the format: the file is plain Markdown, so it is not tied to one vendor's feature set, and it still works if you change agents.

Our studio runs two coding agents; how we use both AI coding agents together is covered in its own piece, and this article is about the files they share.

How we write a skill now

Do the job by hand once and write down the steps you actually took.
Name the trigger in the description: the words a person types when they want this job.
Write steps. Numbered, one action each, with the exact command or file path, not paragraphs of advice.
Add the checks at the end: what "done" looks like, and what must never change.
Point to files instead of pasting them: scripts and style guides live next to SKILL.md.
Test with a plain request that does not name the skill. If it does not load, fix the description, not the steps.

Mistakes we made

Vague descriptions. Our first skills said what they were, not when to use them, so they loaded late or never.
Too long. A skill that tries to cover three jobs loads for all three and confuses each. Split it.
Rules without checks. "Write simply" does nothing; "no sentence over 25 words, count them" works.

Our take

Most people will install a hundred skills and use four. What matters is simpler: does each skill have a description that fires at the right moment, and steps that end in a check? Four skills that load at the right moment will do more work than a hundred that sit unused.

Honest verdict

Good: plain Markdown, works with more than one agent, easy to review in a pull request, and it writes down the steps one person knows so anyone on the team can run them.

Worth knowing: a skill is only as good as its description, long skills cost context when they load, and an agent can still skip a step, so the checks at the end matter.

What it means for you

Pick the one job you explain to an agent most often. Write it as a SKILL.md today with a trigger-style description and a check at the end. Use it for a week, then fix the description based on when it failed to load.

KV cache and continuous batching: 1,000 people ask AI

System Design Simple — Mon, 09 Nov 2026 00:30:45 +0000

An AI server answers many people at once with two tricks: a KV cache, so each new token does not redo the work of the earlier ones, and continuous batching, so many requests share every generation step. On our toy model the cache cut one answer from 567 ms to 39 ms, and in a simulated queue of 1,000 requests the median wait for the first token fell from 452 s (7.5 minutes) to 14 s.

Why does writing one token at a time repeat work? {#tokens}

Definition

Token: a small piece of text, often part of a word, that a language model reads and writes one at a time.

Definition

Time to first token (TTFT): the time from sending a request until the first output token is ready. It includes waiting in the queue and processing the prompt.

A decoder-only transformer generates autoregressively: it predicts token t+1 from tokens 1 to t, appends it, and repeats. Inside each attention layer every token is projected into a query, a key and a value. The new token's query is compared with the keys of all earlier tokens, and the result weights their values.

The naive loop runs the whole sequence through the model at every step. For a prompt of P tokens and an answer of N tokens, the projections alone are done for N·P + N(N-1)/2 token rows instead of P + N-1. For our lab's case (P = 16, N = 48) that is 1,896 rows against 63. The keys and values of token 5 never change once computed, because attention in a decoder only looks backwards, so recomputing them is pure waste. Recomputing full causal attention at every step adds more on top, since each pass is already quadratic in sequence length, as the Transformer paper sets out.

What does a KV cache keep? {#kv}

Definition

KV cache: per layer and per attention head, the key and value vectors of every token already processed, stored so later steps can read them instead of recomputing them.

With the cache, generation has two phases. Prefill runs the prompt through the model once, in parallel, and writes its keys and values to the cache; this is mostly matrix-matrix work. Each decode step then handles one new token: it computes that token's key, value and query, appends the key and value to the cache, and attends over everything stored, which is mostly matrix-vector work.

The new token still does real math, and attention still reads every cached key, so a step at a long context costs more than one at a short context. The cache only removes the recomputation of old tokens. The Hugging Face cache documentation describes this split between computing the new token and reusing stored keys and values. If long chats are new to you, why a long AI chat forgets your name explains the context window this cache lives in.

How does continuous batching share the server? {#batching}

Definition

Static batching: start B requests together and run decode steps until the longest one finishes; short answers sit idle in their slot.

Definition

Continuous batching (iteration-level scheduling): rebuild the batch at every step; a finished request leaves at once and the next one in the queue is admitted with a prefill.

The Orca paper (OSDI 2022) introduced iteration-level scheduling for this reason: with request-level batching, requests that finish early wait for the slowest one, and new arrivals wait for the whole batch.

Batching helps fairness first. One-at-a-time serving makes person 1,000 wait for 999 complete answers, while 16 slots let many people start early. It can also raise throughput. On a GPU, a small decode step is usually limited by reading the weights from memory, so adding rows is cheap and tokens per second can rise sharply. Our single-threaded JavaScript implementation achieved 1.85x decode throughput at 16 rows versus one row at context 64. We did not profile its bottleneck or measure a GPU.

What our lab measured {#lab}

The lab (labs/ai-18, node run.mjs, Node 22, no packages) builds a tiny transformer with random weights. Part A times the model directly. Part B replays a queue of 1,000 requests using only those measured step times, interpolated inside the measured grid. The simulator admits requests in queue order, estimates mixed-length batches by summing shares of equal-length measurements, and pauses decode during prefill. It models neither KV memory limits nor network delay. So the person at the front of the line always starts first, and TTFT here means arrival-to-first-token-ready time; it does not measure delivery to a client. The code and our reference results are on GitHub, so you can run it yourself: kv-cache-bench.

# measured, one 48-token answer to a 16-token prompt, 5 runs per path
no KV cache     median 567.05 ms   (min 566.47, max 575.72)
with KV cache   median  39.19 ms   (min  39.14, max  39.22)
ratio           14.5x, identical tokens on both paths

# measured, one decode step at context 64
1 row   0.751 ms      16 rows   6.482 ms    -> 1.85x tokens per second

# simulated queue: 1,000 requests at t = 0, prompts 8 to 32, answers 16 to 96
policy                         median TTFT   p95 TTFT   all done   tokens/s
one at a time, no cache          452.2 s      851.5 s    892.3 s       64
one at a time, KV cache           23.9 s       45.1 s     47.3 s    1,203
static batches of 16, cache       21.2 s       40.5 s     42.8 s    1,331
continuous batching, 16 slots     13.8 s       26.4 s     27.8 s    2,044

The cache alone moved the median from 452 s to 24 s (18.9x). Continuous batching on top moved it to 14 s (1.73x more). The measured 14.5x for one answer is smaller than the 30x drop in projected rows because attention over the cache, the output layer and fixed overhead remain.

The video for this article: a notepad becomes the KV cache, stools become batch slots.

How KV cache memory limits concurrency {#memory}

Each request keeps its own KV cache for as long as it runs. Its raw KV tensor storage per token, excluding allocation overhead, is:

bytes per token = 2 (key and value) x layers x KV heads x head size x bytes per number

assumed example: 32 layers, 8 KV heads, head size 128, 16-bit numbers
2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KiB per token
a 4,096-token conversation = 4,096 x 128 KiB = 512 MiB for one request

Available KV memory limits how many tokens can remain resident. Compute capacity and latency targets can require a smaller batch; the vLLM tuning documentation walks through these scheduling and latency trade-offs. The PagedAttention paper (SOSP 2023) showed that storing the cache in fixed-size blocks, like virtual memory pages, cuts the waste from fragmentation and reserved space, so more requests fit. Long chats cost more to serve than short ones because the cache grows with every token. An agent's next request may contain a longer accumulated history, increasing its KV storage requirement. Re-sending an unchanged prefix can repeat prefill work unless the server reuses it; why an AI agent re-sends its history looks at that pattern.

Failure modes to watch in production: queue time dominating TTFT while the accelerator is idle (scheduler or admission limits too strict), out-of-memory pre-emption when long contexts arrive together (requests get evicted and recomputed), and long prompts stalling everyone's decode steps (servers split them with chunked prefill).

Trade-offs {#trade-offs}

Option	Good for	Costs you	Use when
No cache, one at a time	Teaching, tiny tests	Repeated projections grow quadratically with answer length; full attention recomputation adds further work; the queue is serial	Never in serving
KV cache, one at a time	Lowest latency for a single user	Memory per request; everyone else waits	A local model on your own laptop
Static batches	Simple offline jobs	Padding, short answers wait for long ones	Batch scoring with similar lengths
Continuous batching	Many users, mixed lengths	A scheduler, cache memory management	Any shared chat or API server

What we saw when we ran it {#we-ran-it}

This is a separate, later run from the one in the lab table above, on 2026-11-06 at 15:37 UTC, so its timings differ a little from that table. We ran node labs/ai-18/run.mjs on an Apple M2 Pro with Node v22.13.1, on one thread, and the whole run took 8.6 s. The screen first printed the direct timings: a 48-token answer took a median of 39.6 ms with the cache on and 577.8 ms with it off, and the same tokens came out in every run. Then it replayed 1,000 simulated requests arriving at once.

Our terminal after the run, with the cache timings and the queue table.

The line that surprised us was static batching. Batches of 16 with the cache on only moved the median simulated wait for a first token to be ready from 24.14 s to 21.41 s. Continuous batching with the same 16 slots brought it down to 13.96 s, and the last request in line had its first token ready at 27.90 s, where one at a time with no cache it waited 903.42 s. That last gap mixes two changes, the cache and the scheduling. On this tiny random-weight model on one CPU thread, with no memory limits or network delay in the simulation, the cache gave the biggest single gain and letting finished requests leave the batch right away gave the next one; a real server has to check both against its own memory and traffic.

The Hugging Face Transformers cache explanation page.

Our cache path follows the split that page describes: compute the new token, and reuse the stored keys and values of earlier ones.

Our take {#our-take}

The KV cache made the larger difference in this lab. It removes repeated projections and feed-forward work on earlier tokens, though dense attention still reads a growing history. Batching then decides who waits, and on our toy model the two together took the median wait from minutes to seconds. These timings come from one CPU core, random weights and a simulated workload, so they describe this implementation only. If you run models on your own machine, as in answering with no network, you already use the cache; batching matters once more than one person shares the server.

The lab is a simulation built on a toy model on one machine; it describes general serving patterns from public papers and docs, not any company's private system.

apt CVE-2026-109518: how to check and fix Debian, Ubuntu

System Design Simple — Mon, 09 Nov 2026 00:24:52 +0000

If a Debian or Ubuntu machine runs an apt older than the first fixed package for its release, a party that can change its HTTP traffic could make apt install a package that never passed verification. The flaw is CVE-2026-109518, a vulnerability that enables a man-in-the-middle attack through how apt reuses downloaded files; the fix is to update apt and confirm the installed version is the listed one or later.

This guide is for protecting applications you own or maintain.

Debian published DSA-6607-1 and Ubuntu published USN-8996-1 on 2026-11-04, and The Hacker News covered it the same day. Neither advisory reports exploitation.

How apt checks that a package is genuine

apt does not check a signature on each .deb. It checks one signed metadata file and follows hashes down from it (apt-secure). A hash here is a SHA256 fingerprint of a file's bytes: change one byte and the fingerprint changes. The archive key is the OpenPGP key the repository owner signs that metadata with.

# chain of trust, top to bottom
InRelease      signed Release metadata: SHA256 of every index, Date, optional Valid-Until
               (or Release plus a detached Release.gpg signature)
  └─ Packages  index; lists Filename and SHA256 of every .deb
       └─ hello_1.0.deb   accepted only if its SHA256 matches the Packages entry

With authentication and freshness checks enabled, apt rejects a bad signature, an index whose hash is not in the signed block, a .deb whose hash is not in the index, and metadata past its Valid-Until. Every link vouches for exact bytes. The guarantee covers the file that was hashed, so anything that later changes what sits behind that file name falls outside it.

How CVE-2026-109518 swapped a verified package

The signature and hash checks worked. According to the APT team and DSA-6607-1, a party answering apt's HTTP requests could send redirects so that two different package downloads pointed at the same URI, plus transient errors so apt retried. apt merged the two downloads using existing hard links in its download directory, and a chosen payload could end up behind a file that had already passed its hash check.

The hard link is the detail that matters. A file's bytes live in an inode, and a file name is only a pointer to that inode. Two hard links are two names for one inode. Writing through one name changes the bytes you read through the other, and neither path is replaced or renamed, so a check that only watches paths sees nothing move. A verified name and an unverified name that share an inode are the same file.

APT's upstream patch opens download targets exclusively. When resuming, it retains a read handle to the old file, unlinks that destination, creates a fresh file, and copies the partial contents into it. Later writes then affect the new inode.

Optional: the same gap in an updater you own

Application code that verifies and then installs from a path has the same check-then-use gap. The fragment below is illustrative and is not APT's code:

// install.mjs (unsafe: hashes one read of the path, installs a later read)
import { readFileSync } from 'node:fs';
import { execFileSync } from 'node:child_process';
import { createHash } from 'node:crypto';
const sha256 = (b) => createHash('sha256').update(b).digest('hex');

if (sha256(readFileSync(path)) !== expected) throw new Error('hash mismatch');
execFileSync('dpkg', ['-i', path]); // a write through another hard link, or a swap, lands here

// install.mjs (fixed, illustrative fragment: install exactly the bytes that were verified)
import { readFileSync, writeFileSync, mkdtempSync } from 'node:fs';
import { execFileSync } from 'node:child_process';
import { createHash } from 'node:crypto';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const sha256 = (b) => createHash('sha256').update(b).digest('hex');

const bytes = readFileSync(path);
if (sha256(bytes) !== expected) throw new Error('hash mismatch');
const dir = mkdtempSync(join(tmpdir(), 'verified-'));     // new directory, mode 0700
const safe = join(dir, 'pkg.deb');
writeFileSync(safe, bytes, { flag: 'wx', mode: 0o600 });  // new inode; fails if the name exists
execFileSync('dpkg', ['-i', safe]);

This example assumes other processes with the same privileges are trusted. Directory permissions do not isolate those processes from each other.

Which apt versions are affected?

Release	First fixed apt package	Record
Debian 13 trixie	3.0.4	DSA-6607-1
Debian 12 bookworm	2.6.2	DLA-4880-1
Debian forky, sid	3.3.6	security tracker
Ubuntu 26.10	3.3.3ubuntu0.1	USN-8996-1
Ubuntu 26.04 LTS	3.2.1ubuntu0.1	USN-8996-1
Ubuntu 24.04 LTS	2.8.4ubuntu0.1	USN-8996-1
Ubuntu 22.04 LTS	2.4.15ubuntu0.1	USN-8996-1
Ubuntu 20.04, 18.04	2.0.12+esm1, 1.6.19+esm1	Ubuntu Pro

You are fixed when the installed version is the listed version or a later version for the same release. The bug also needs apt to retry downloads. According to the APT team announcement, Acquire::Retries has defaulted to 3 since apt 2.3.2, which covers Debian 12 and Ubuntu 22.04 onward. Older releases are exposed only if a config file or a command line turned retries on.

Is my apt repo config safe?

Updating fixes the CVE. Per-repository settings decide how much of the chain is switched on (sources.list):

# /etc/apt/sources.list.d/vendor.list (unsafe)
deb [trusted=yes check-valid-until=no] http://repo.example.com/debian stable main
# and the vendor key dropped into /etc/apt/trusted.gpg.d/

# /etc/apt/sources.list.d/vendor.sources (fixed)
Types: deb
URIs: https://repo.example.com/debian
Suites: stable
Components: main
Signed-By: /etc/apt/keyrings/vendor.asc

trusted=yes (deb822: Trusted: yes) accepts an unsigned or badly signed repo. check-valid-until=no (deb822: Check-Valid-Until: no) accepts an old index after it expires, which can hold back security updates. A key in trusted.gpg.d is global: it can authenticate any repository that has no narrower key selection. Signed-By limits a source to the key or keys it names.

How to update apt

# Debian / Ubuntu
sudo apt update && sudo apt install --only-upgrade apt
# Ubuntu 24.04 threshold shown; use the first fixed version for your release from the table
installed=$(dpkg-query -W -f='${Version}' apt) || exit 1
dpkg --compare-versions "$installed" ge '2.8.4ubuntu0.1' || {
  echo "not fixed: $installed" >&2
  exit 1
}
apt changelog apt | grep CVE-2026-109518   # supporting detail only

apt-cache policy apt shows versions but does not compare them; dpkg --compare-versions applies Debian's version ordering. A changelog entry alone does not prove which package is installed.

The update itself travels over apt. Our own suggestion, not from the advisories: run it from a network you trust, against an HTTPS mirror if your release offers one. If you must delay, Acquire::Retries "0"; in /etc/apt/apt.conf.d/ temporarily removes one precondition the tracker documents. It only holds if the real invocation keeps that value, so check scripts for -o Acquire::Retries=..., and remove it after updating, since flaky mirrors will fail more often.

What our lab showed

Our lab builds a throwaway signed repo with the same three layers (a signed InRelease, a Packages index and two package stand-in files that contain plain text), tampers with it six ways, and runs one verifier twice: every check on, then exactly one check off. It uses Ed25519 in a custom signature envelope in place of OpenPGP, installs nothing and runs offline. Run it with node labs/security/apt-signature-lab/apt-signature-lab.mjs. We ran it on this Mac on 2026-11-06 with Node.js v22.13.1.

The saved output of node labs/security/apt-signature-lab/apt-signature-lab.mjs, placed in a terminal-style page and captured in Chrome.

Both verifiers accepted the untouched repo. Of the six tampered repos, the unsafe verifier accepted all six and the fixed one rejected all six: a .deb changed after signing (REJECT hello_1.0.deb), a .deb and Packages changed together (REJECT index hash), text added outside the signed block (REJECT index hash), a repo signed by another trusted key (REJECT wrong key), an expired index replayed (REJECT expired) and a removed signature (REJECT unsigned). The read-only version check found apt not installed on this Mac.

These six cases demonstrate signature scope, hash validation, key selection and expiry checks. They do not test the handoff from verification to installation, so they do not reproduce CVE-2026-109518.

Our take: update apt first. Repository hardening remains useful because it controls which keys and metadata your machine accepts.

Fix checklist

Step	Done when
Update apt	installed is the listed version or a later version for the same release (`dpkg --compare-versions ... ge`)
Old releases	Ubuntu 20.04 and older on Ubuntu Pro packages, or an upgrade planned
Until updated	`Acquire::Retries "0"` in config and no `-o Acquire::Retries=` override in scripts
Unsigned repos	no `trusted=yes`, `Trusted: yes` or `allow-insecure` anywhere
Key pinning	every third-party source has `Signed-By`; `trusted.gpg.d` holds no vendor keys
Freshness	expiry checks enabled for active repositories; documented exceptions for historical archives; consider `Valid-Until-Max` where metadata omits expiry
Your own updaters	install from the verified bytes in a private file, never by re-reading the path

Test your own app

Search your hosts and images for settings that switch a link off, in both .list and deb822 .sources syntax:

grep -rniE "trusted=yes|Trusted: *yes|allow-insecure|check-valid-until=no|Check-Valid-Until: *no|AllowUnauthenticated" /etc/apt/ Dockerfile* 2>/dev/null
grep -rnE "Acquire::(Retries|Check-Valid-Until)" /etc/apt/apt.conf.d/
grep -rnE "Acquire::Retries=" scripts/ .github/ Dockerfile* 2>/dev/null   # per-invocation overrides
ls /etc/apt/trusted.gpg.d/
apt-config dump | grep -iE "retries|check-valid-until"

In your own code, search for a hash check followed by a second read of the same path, for example sha256(readFileSync(path)) and then exec(..., path). A unit test can prove the fixed shape: verify a file, create a hard link to it, write new bytes through the link before install, and assert that the installed bytes are the verified ones. In CI, run the version check above inside your base image; it exits non-zero when apt is older than the threshold, so the build fails. Our PHP security checklist, Node.js security guide and Python security guide use the same search, test and fix approach for application code.

Server failover: active-passive vs active-active

System Design Simple — Mon, 09 Nov 2026 00:24:20 +0000

In active-passive vs active-active failover, every request routed to a dead server fails until the load balancer stops routing there, so the loss depends on how fast it notices and on who is ready to take over. In our simulation at 50 requests per second, an active-passive setup that waited for three 1 s probes lost a median 111 requests, and an active-active setup that dropped a server on its first failed request lost 1.

What our lab measured {#lab}

Copies side by side only help if the switch is quick, and every second of the gap comes out of the downtime budget. labs/sd-11/run.mjs measures that gap in a discrete-event simulation: one Node.js process (v22.13.1, Apple M2 Pro), a virtual clock, nothing on a real port. It shows the shape of the effect and is not a benchmark. The code and every run's output are on GitHub, so you can run it yourself: failover-lab.

load        50 req/s for 30 s; primary killed at 10 s
servers     2, each with 2 workers and a queue; service 8 ms + exp(mean 22 ms); 1 ms per hop
failure     requests in flight fail as "reset"; later ones are "refused" at once (closed port)
A-P         all traffic to the primary; probe every 1,000 ms; switch after 3 misses;
            cold standby (assumed +30 ms on its first request, fading over 50)
A-A         round robin over two warm servers; drop a server on its first failed request; no retries
runs        5 per mode, seeds 1100 to 1104; median (min to max)

Measure	Active-passive	Active-active
Failed requests	111 (102 to 148)	1 in every run
First failure to first success	2,217.2 ms (2,101.1 to 2,994.8)	35.9 ms (0.9 to 80.3)
p95 before the kill	92.2 ms	79.1 ms
p95 in the first second after	259.2 ms (cold standby)	78.3 ms
p95 steady after	90.3 ms	87.8 ms (survivor)

Active-passive in our simulation: 111 failed requests and 2.2 s with no success (median of 5 runs).

The configurations used different detection policies, so 111 versus 1 measures detection as much as topology. Detection policy sets how long traffic keeps going to the dead server, and topology sets which servers are already serving, how warm they are and how much headroom they have.

How long does active-passive detection take? {#detection}

Assume probes start on a fixed schedule every I seconds, the probe timeout T is shorter than I, the crash lands at a uniformly random moment, and the dead server refuses connections at once. With a threshold of k misses, the first failing probe starts somewhere in the next I seconds and the k-th starts (k minus 1) × I after it:

best      (k - 1) × I      first probe fires right after the crash
worst     k × I            first probe fires just before the crash
expected  (k - 0.5) × I

lab, I = 1 s, k = 3:   2 s to 3 s, expected 2.5 s
failed ≈ rate × time:  50 × 2 = 100  to  50 × 3 = 150
median run:            50 × 2.2172 s ≈ 110.9    measured 111

All five gaps fell inside that window, plus a few milliseconds for the standby's first response. Almost every failure was "refused" during detection; only 0 to 4 per run were in flight at the crash.

A hung server adds the probe timeout T: one T on a fixed schedule, and k × T if each probe starts I after the previous one ends. With I = 1 s, k = 3 and T = 0.5 s, that is 2.5 to 3.5 s against 3.5 to 4.5 s.

Defaults are slower. HAProxy's inter 2 s and fall 3 give 4 to 6 s. A Kubernetes readiness probe at its defaults (10 s, threshold 3) needs 20 to 30 s before the endpoint update even starts spreading to nodes.

A probe that only proves the process is up misses real failures, and one that queries the database fails on every server at once when the database does. Compare probe results with request failures. When every server fails at once, check shared dependencies first, then decide whether removing every backend improves service.

The cold standby added its own cost: with the assumed +30 ms, two workers could serve about 33 req/s while 50 arrived, so a queue built and the first-second p95 reached 259.2 ms.

Why did active-active lose only one request? {#active-active}

Round robin gave each server one request about every 40 ms. The first request to the dead primary was refused, the error came back after two 1 ms hops, and the load balancer (how it picks a backend) removed the server well before its next turn. That is a passive health check with a threshold of one: real traffic acted as the probe, and the refusal made the failure obvious at once.

Active-active: the load balancer dropped the dead server after its first failed request, so only that one request failed.

By the formula, active-active with three 1 s probes would lose about 50 to 75 (half the traffic for 2 to 3 s), and active-passive with the first-error rule would lose about one; we did not run either. A retry through Nginx's proxy_next_upstream could hide that last failure. A refused connection sent nothing, so even a POST can move on; once a POST was sent, Nginx does not resend it by default.

Why does the survivor slow down? {#headroom}

In our active-active runs the survivor took double load, and its steady p95 went up. Utilisation U is arrival rate × mean service time ÷ workers:

S = 30 ms mean, c = 2 workers
before: 25 req/s each    U = 25 × 0.030 ÷ 2 = 0.375    p95 79.1 ms
after:  50 req/s on one  U = 50 × 0.030 ÷ 2 = 0.75     p95 87.8 ms

N servers at u each, one lost: survivors run at u × N ÷ (N - 1)
to keep survivors at or below 0.75:  N = 2: 0.375   N = 3: 0.50   N = 4: 0.5625

The lab's arrivals are evenly spaced, so the rise was only 8.7 ms. With random arrivals the M/M/2 model's mean wait goes from 4.9 ms at U = 0.375 to 38.6 ms at U = 0.75 (why waiting grows near full load). Without headroom, survivors fail their own checks and are removed in turn, as the SRE book on cascading failures describes.

Inside Nginx: shared failure state, retries and logs {#nginx}

Our active-active lane resembles an open source Nginx upstream with max_fails=1 and no retries, with two differences. The simulation removed the dead server for good, while Nginx makes the peer eligible again after fail_timeout; a continuing failure can therefore cause further failed attempts. And without a zone directive, each worker process keeps its own failure counters and "unavailable until" marks, so each worker receiving traffic may independently discover the failure. zone app 64k; moves that state into shared memory. Requests already assigned to the dead server still fail, because Nginx records the failure only when it releases the failed peer (ngx_http_upstream_free_round_robin_peer).

Connection errors, timeouts and invalid headers always count toward max_fails; a 500, 502, 503 or 504, or 429, counts only if its http_* value is listed in proxy_next_upstream. A server that answers 500 to everything stays in rotation by default.

# nginx.conf (http context): active-active, passive checks, bounded retry
upstream app {
    zone app 64k;   # share failure counters and down marks across workers
    server 10.0.0.1:8080 max_fails=1 fail_timeout=10s;
    server 10.0.0.2:8080 max_fails=1 fail_timeout=10s;
}
log_format upstream '$remote_addr "$request" status=$status rt=$request_time '
                    'ua="$upstream_addr" us="$upstream_status" '
                    'uct="$upstream_connect_time" urt="$upstream_response_time"';
server {
    listen 80;
    access_log /var/log/nginx/upstream.log upstream;
    location / {
        proxy_pass http://app;
        proxy_next_upstream error timeout;   # which failures are retried
        proxy_next_upstream_tries 2;         # at most 2 attempts (default 0 = no limit)
        proxy_next_upstream_timeout 3s;      # no new attempt after 3 s
        proxy_connect_timeout 1s;
        proxy_send_timeout 5s;               # inactivity between writes (default 60 s)
        proxy_read_timeout 5s;               # inactivity between reads (default 60 s)
    }
}

The send and read timeouts measure inactivity between two successive operations, so a response can continue indefinitely while it keeps making progress. If the first connection times out after 1 s and the second backend sends no response bytes, the read timeout adds about 5 s. These settings impose no overall response deadline (proxy_read_timeout); enforce one in the client or a layer in front.

When a request is retried, each $upstream_* variable holds one comma-separated entry per server tried. Illustrative lines, not captured from our lab:

refused, retried:    status=200 ua="10.0.0.1:8080, 10.0.0.2:8080" us="502, 200" urt="0.000, 0.031"
connect timeout:     status=200 ua="10.0.0.1:8080, 10.0.0.2:8080" us="504, 200" urt="1.001, 0.029"
no response bytes:   status=504 ua="10.0.0.1:8080" us="504" urt="5.002"   (POST, not retried)
all marked down:     status=502 ua="app" us="502"   error log: no live upstreams

A refused connection is an explicit rejection, often because no process is listening. A connect timeout means the connection did not complete before its deadline; investigate the backend and network. A comma in ua shows multiple upstream attempts. Check the final client status to see whether the retry succeeded (upstream variables).

Databases, virtual IPs and DNS {#beyond}

Our lab stores no data and moves no address. A database standby fed by asynchronous replication can miss writes acknowledged just before the crash (replication lag), and a node paused past its lease can still finish a write, so the storage must reject older fencing tokens or the old primary must be isolated before promotion (Kleppmann). A VRRP virtual IP adds its own timer, 3.609375 s at priority 100 and a 1 s interval (RFC 9568, section 6.1). DNS failover also depends on detection, record updates, remaining cache lifetimes and when clients reconnect.

Active-passive vs active-active: which to use {#trade-offs}

Option	Good for	Costs you
Active-passive, active checks	a single writer such as a database primary	seconds of errors (111 in our simulation), idle hardware, cold start
Active-active, passive checks	stateless app servers (1 failed in our simulation)	N+1 headroom on every server; survivor latency rises
Either, plus retries	hiding the last failed request	duplicate side effects without idempotency keys; retry storms

Our take {#our-take}

For stateless app servers we would run active-active, remove a backend on real request errors, add zone in Nginx, and keep each of two servers at or below 37.5% utilisation if the survivor must stay at or below 75%. Keep active-passive where only one node may write, with fencing. Either way, compute your detection window from the real interval, threshold and timeout, and fail over on purpose now and then, because a spare that has never served can hide a cold start or a stale config.

This article describes general patterns and our own small simulation. It does not describe any company's private systems. The simulated numbers show the shape of the trade-off, not production measurements.

Is Codex CLI still running after you close it? A 5-minute check

System Design Simple — Sun, 08 Nov 2026 11:03:37 +0000

Yes, Codex CLI can keep a process running after you close the terminal. Since version 0.157.0, an eligible interactive session starts a shared background server automatically, and one command turns that off: codex features disable daemon_auto_start.

This article is for anyone who uses Codex CLI in the terminal most days, especially on a shared build machine or a laptop where background processes and disk space matter. The everyday problem is simple: you close the terminal and assume everything stopped. Since September 25, 2026, the CLI can leave a server running and its package on your disk (338 MB on our Mac) without asking. These three checks show you what your machine is doing, and one command lets you choose the behaviour you want.

Nothing here is broken. Starting the server is the intended behaviour; the only question is whether you want it on your machine. A background server lets several sessions share one connection and recover faster. Still, it starts without asking you. We spent five minutes on our Mac checking what it does, how much disk it takes and how to switch it off. The steps are below.

What changed in Codex CLI 0.157.0

The 0.157.0 release notes, published on September 25, 2026, list it plainly: "Enabled automatic background-server startup for eligible interactive sessions, with recovery choices when server settings are incompatible." The pull request behind it is titled "Enable automatic daemon startup by default".

Before 0.157.0 the CLI stopped when you closed the terminal. From 0.157.0, an interactive session can start a small server in the background, and later sessions talk to that server.

Non-interactive runs such as codex exec are a different path. Our automation calls codex exec many times a day, and on this Mac we found no background server running between those calls.

The Codex CLI 0.157.0 release note that turned on automatic background-server startup (GitHub, captured 2026-11-07).

Check 1: is the feature on?

codex features list | grep daemon_auto_start

On our Mac, on November 7, 2026, with Codex CLI 0.164.0, this printed:

daemon_auto_start                        stable             true

stable is the feature's stage, true is its effective state. So the automatic start is on, which is the default.

Check 2: how much disk does it use?

The server ships as its own package inside your Codex folder:

du -sh ~/.codex/packages/app-server-daemon/releases

Our result: 338 MB, one folder named 0.164.0-aarch64-apple-darwin. On our Mac there was one folder, for 0.164.0. Your number depends on your version and on how many old versions have piled up in that folder.

Check 3: is it running right now?

pgrep -fl codex

If this prints nothing, no Codex process is running. If you see a server process after you closed every terminal, the background server is still alive. You can close it from your process manager, or turn the feature off below and open a fresh terminal.

How do I turn off the Codex CLI background server?

codex features disable daemon_auto_start
codex features list | grep daemon_auto_start

codex features disable writes the change to ~/.codex/config.toml, so it applies to every new session. To undo it later, run codex features enable daemon_auto_start.

If you only want to skip the server for one view, Codex CLI 0.167.0 (November 6, 2026) adds codex agents --no-daemon, which opens the agent command center with an embedded server instead. That does not change the default for normal sessions.

Why it works this way (the system design view)

This is the shared service pattern you already know from other tools: instead of every terminal opening its own connections and state, one long lived process holds them and short lived clients attach to it.

	One process per terminal	Shared background server
Start time for the next session	Full start every time	Faster, the server is already up
State across sessions	Lost when the window closes	Kept by the server
What runs when you are not using it	Nothing	The server
Disk	The CLI only	The CLI plus the server package (338 MB here)

Neither choice is wrong; it depends on the machine. A laptop on battery and a shared build machine usually want the left column. If you open many sessions a day, you will probably prefer the right one.

It is the same one door thinking as in our explainer on which server gets your request: many clients, one place that holds the state.

We use the same one process stays up idea in our server failover piece, where something is always running so the client never waits for a cold start.

Our take

Most write ups of this change just repeat the release note. What they skip is the team question: a default like this should be a decision you write down once, because a shared build machine, a laptop and a CI runner want different answers, and "it was on by default" is not a reason for any of them.

Honest verdict

Good: the background server is documented, the flag has a clear name, and one command switches it off or on. The new --no-daemon option shows the team is giving people more control.

Worth knowing: the automatic start is on by default, the server package takes real disk space (338 MB on our Mac), and the setting lives in ~/.codex/config.toml, where a new teammate will not look first.

What it means for you

Solo developer, one or two sessions a day: run the three checks once. Keep the default if faster starts matter to you; turn it off if you prefer nothing running in the background.
Shared or CI machines: decide the default for the machine, set it with codex features disable daemon_auto_start (or enable) in your setup script, and document it.
Low on disk: look at ~/.codex/packages/app-server-daemon/releases after upgrades; old version folders add up.

If you run both agents, see how we use Claude Code and Codex CLI together before you settle the defaults for each.

Codex CLI agent command center: what 0.167.0 changes when you run several agents

System Design Simple — Sat, 07 Nov 2026 16:41:39 +0000

Originally published on System Design Simple: https://systemdesignsimple.com/codex-cli-agent-command-center/

Codex CLI 0.167.0 opens an agent command center with codex agents --no-daemon, marks unread results, keeps drafts across backends and previews images.

Drafted with AI assistance, then checked and edited by a human engineer.

Short answer: Codex CLI 0.167.0, released November 6, 2026, adds an agent command center you open with codex agents --no-daemon. The command center and session picker mark unread results, drafts survive a backend switch, and local image attachments show a preview on supported terminals. It is a supervision release, not a new model.

Key fact	Detail
Released	November 6, 2026, 01:40 UTC (GitHub release rust-v0.167.0)
Open it with	`codex agents --no-daemon`
Unread markers	In the command center and the session picker
Background server default	Still on (`daemon_auto_start`), the flag is for this view

Codex CLI 0.167.0, released on November 6, 2026, adds an agent command center that you open with one command: codex agents --no-daemon. It marks results you have not read, keeps your draft when you switch backends, and previews local image attachments while you write. There is no new model in this release. What changed is supervision: one place to watch several coding agents at once, so you stop re-checking every session by hand.

This article is for developers who run Codex CLI for real work and have started leaving two or three agents running in parallel. If you split your work between the two tools, how we use Claude Code and Codex CLI together is covered in its own piece.

The official word

The release notes list the command center first: "Open the agent command center with codex agents --no-daemon. The command center and session picker now mark unread results."

The same release adds, in its own words:

Switch between connected backends from the terminal session picker and keep your drafts when switching.
Preview local image attachments while composing on supported terminals.
Expand patch details directly in the transcript while an approval prompt is open.
Saved chats stay visible after you switch model providers; if a chat cannot resume, you can read its transcript.
Ctrl+L clears the terminal display while keeping earlier messages available through Ctrl+T.
Google Cloud Vertex AI setup sits behind the gcp_setup_wizard flag.

Why is it hard to keep track of several coding agents at once?

One agent is easy to watch. Run three or four in parallel and the slow part stops being typing; it is remembering what each one finished and which result you have not seen yet. Switching to a terminal and scrolling back is a small job, and doing it four times is the whole job. That is the problem the command center solves: one screen, every agent, the ones with unread results marked.

Running agents	What costs you time
1	Waiting for it
3 to 4	Finding which one finished and what you have not read
3 to 4 with unread markers	Opening the marked ones first

Unread markers in two places, the command center and the session picker, turn "check every session" into "check the marked ones". It is not work being done faster, it is work you no longer repeat.

Try it in two minutes

npm install -g @openai/codex@0.167.0
codex --version  # codex-cli 0.167.0
codex agents --no-daemon

Start two or three short tasks in separate sessions first, so the command center has something to show.

Small changes that save time every day

Three smaller lines in the release notes are worth a look, because each one removes a daily annoyance.

Patch details inside the approval prompt. You can expand what a change does before you approve it, instead of approving blind or scrolling back through the transcript.
A transcript when a chat cannot resume. After switching model providers, saved chats stay visible, and a chat that cannot resume still shows you its transcript.
Ctrl+L with Ctrl+T. Ctrl+L clears the screen, and earlier messages stay one keystroke away through Ctrl+T. A clean screen no longer means a lost history.

None of these needs a setting; they start working as soon as you update.

Keeping that history one keystroke away matters, because it is the record the session continues from — why an agent re-sends its history.

What --no-daemon means

Since 0.157.0, interactive sessions start a shared background server automatically. The --no-daemon option opens this one view without relying on that server. It does not change the default: the daemon_auto_start flag is still on, and codex features disable daemon_auto_start is the setting for every session. We cover that check in our 5-minute background server audit.

Our take

Most release coverage will skip this one because there is no new model in it. The shift it shows is more useful: coding agents are moving from one long chat to a team you supervise. A command center with unread markers is the first interface that treats them that way. If you only run one agent at a time, this release is a minor update. If you run three, it is the first version that respects how you actually work.

Honest verdict

Good: one command to see everything, unread markers where you look, drafts kept across backends, and patch details before you approve. Worth knowing: image previews depend on your terminal, the Vertex AI setup is behind a flag, and the background server default is unchanged, so read --no-daemon as "this view only", not "server off".

What it means for you

One agent at a time: update when convenient. You will notice the image preview and the cleaner picker.
Two or more agents: update and make codex agents --no-daemon your first check when you return to your desk. Open unread results first, then pending approvals.
Teams: test draft keeping on your exact backend mix, and decide the background server default once, in writing, before the team updates.

Frequently asked questions

How do I open the Codex CLI agent command center?

Update to Codex CLI 0.167.0 or later and run codex agents --no-daemon. It shows your agent sessions and marks results you have not read yet.

What does --no-daemon do in codex agents?

It runs the agents overview without relying on the shared background server, which Codex CLI starts automatically since 0.157.0. It does not change that default; codex features disable daemon_auto_start does.

Do I keep my draft when I switch backends in Codex CLI?

Yes, from 0.167.0 you can switch between connected backends from the terminal session picker and keep your drafts when switching.

Does codex agents --no-daemon turn off the background server for all my sessions?

No. It opens this one view without relying on the shared background server. The default stays on, and codex features disable daemon_auto_start is the command that changes it for every session.