LLM Serving: GPU Memory, KV Cache, and Large-Scale Inference

1
2
3
4
5
Model weights = everything the chef knows about cooking
                Once learned, never forgotten. Shared across all dishes.

KV Cache      = sticky notes the chef writes while cooking one specific dish
                Trashed when that dish is done. Rewritten for every user, every conversation.

1
2
3
4
5
6
7
8
9
10
┌─────────────────────────────────────────────────┐
│              Single GPU VRAM (80GB)              │
├─────────────────┬───────────────────────────────┤
│  Model weights  │        KV Cache               │
│  ~35GB (INT4)   │    allocated on demand         │
│  Shared by all  │    ~4-16GB per active session  │
├─────────────────┴───────────────────────────────┤
│              Other overhead                      │
│    intermediate buffers, temp storage            │
└──────────────────────────────────────────────────┘

1
2
3
4
Step 1: Load model weights into VRAM (one-time cost, regardless of user count)
Step 2: User sends message → allocate KV Cache VRAM on demand
Step 3: Generation done → KV Cache freed (weights stay)
Step 4: New message arrives → KV Cache re-allocated

1
2
3
4
5
6
7
8
9
10
11
12
Real usage pattern:

User is online but idle ┌─── barely uses any VRAM ────┐
                        ↓
User types → AI generates │ Occupies KV Cache, but only seconds to minutes
(process takes 30-60s)    │
                        ↓
User reads results        ┌─── KV Cache freed ────────┘

Average 5 uses per day, 30 seconds each
= 150 seconds/day = 2.5 minutes of actual generation
= 99.97% of the time, this user uses zero VRAM

1
2
3
4
No matter how many users are chatting simultaneously
DeepSeek's model weights: one copy

User 1, User 2, ... User N all use the same model weights

1
2
3
4
5
6
7
Traditional (inefficient):
  User 1's request → done → User 2's request → done → ...
  GPU idle most of the time. Utilization < 30%

Continuous batching (efficient):
  Requests from users 1-100 → form a batch → compute together
  GPU utilization up to 70-90%

1
2
3
4
5
6
7
8
Traditional (contiguous):
  User A's cache: block 1  block 2  block 3  block 4 (must be contiguous, gaps unusable)
  User B's cache: wait for A to finish

vLLM (paged):
  User A's cache: page 1  page 5  page 8  (allocated on demand, no contiguity needed)
  User B's cache: page 2  page 3  page 9  (runs in parallel, no waiting)
  VRAM utilization: 20-30% → 80%+

1
2
3
bfloat16 (full precision): 2 bytes per number
INT8 (8-bit integer):      1 byte per number  →  50% less VRAM
INT4 (4-bit integer):      0.5 bytes per number →  75% less VRAM

1
2
3
4
5
6
7
8
9
10
11
12
Option A: Truncation
  → Drop everything beyond a limit. Keep only recent content.

Option B: Sliding Window
  → Keep only the last N tokens' cache (like human memory).

Option C: Summarization
  → Summarize long history, replace raw history with the summary.

Option D: Sparse Attention
  → Only attend to important tokens (keywords, named entities).
  → Less important tokens don't get full K and V.

1
2
3
4
5
6
Same input: "I like eating apples"

Token "I"     → routes to experts [E7, E42] (handles single words and tone)
Token "like"  → routes to experts [E3, E15] (handles emotion)
Token "eat"   → routes to experts [E22, E58] (handles actions)
Token "apple" → routes to experts [E31, E77] (handles concrete nouns)

1
2
3
4
5
6
7
8
9
The hospital has 200 specialists (284B params).
Each patient gets assigned 1-2 doctors at triage (13B activated).
This patient sees cardiology (E7), the next sees orthopedics (E31).

Do you only need a cardiologist on staff?
No. You don't know whether the next patient needs orthopedics or neurology.

All 200 doctors must be on duty (all 284B params loaded).
But each patient only sees 1-2 doctors (~13B activated per token).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
DeepSeek V4 Flash (MoE):
  Total params: 284B
  Activated params: ~13B (per token)
  Precision: INT4

VRAM needed (model weights):
  bf16: 284B × 2 bytes ≈ 568GB  ← total is large
  INT4: 284B × 0.5 bytes ≈ 142GB
  ✓ Need VRAM for all 284B (total VRAM)

Compute needed (per token):
  Dense model (70B): compute all 70B params per token
  MoE model (284B/13B): compute only ~13B params per token
  ✓ Saves compute, not VRAM

1
2
3
4
5
6
7
8
9
10
11
Spread 256 experts across 32 GPUs:

GPU 0:  experts [E0-E7]  + Attention layers (every GPU has these)
GPU 1:  experts [E8-E15] + Attention layers
GPU 2:  experts [E16-E23] + Attention layers
...
GPU 31: experts [E248-E255] + Attention layers

Token "apple" routes to E31 → it's on GPU 3
→ Send "apple"'s intermediate result over the network to GPU 3
→ GPU 3 computes, sends the result back

1
2
3
4
5
6
7
8
9
10
11
12
┌─────────────────────────────────────────────────────┐
│ What MoE saves and what it doesn't                   │
├────────────────────┬────────────────────────────────┤
│ Saves compute      │ 13B/284B activated per token    │
│ Saves per-GPU VRAM │ Each card stores only part of   │
│                    │ the experts (but needs many cards)│
│ Doesn't save total │ 284B is 284B. Total is that big  │
│ params             │                                  │
│ Doesn't save disk  │ Model file ~150-570GB            │
│ Doesn't save total │ All GPUs combined still need to  │
│ VRAM               │ hold all 284B params             │
└──────────────────────────────────────────────────────┘

1
2
3
4
5
6
7
8
9
10
11
Simple request ("write a hello world")
  → Route to cheap small model (Qwen-7B / DeepSeek-V3)
  → Low cost, fast, works great

Medium request ("optimize this SQL query")
  → Route to mid-tier model (DeepSeek-V4-Flash)
  → Medium cost

Complex request ("refactor the entire backend architecture")
  → Route to strongest model (Claude Opus / GPT-5)
  → High cost but best results

1
2
3
4
5
6
7
8
9
10
11
12
User's first conversation (Turn 1)
  → Load balancer sends to GPU-A, KV Cache lives on GPU-A
  → Conversation ends, user goes away

An hour later, user returns (Turn 2)
  → Load balancer sends to GPU-B (GPU-A is busy now)
  → GPU-B doesn't have Turn 1's KV Cache
  → Either recompute Turn 1, or degrade quality

If lucky enough to go back to GPU-A:
  → But GPU-A may have LRU-evicted the cache due to other users
  → Still a miss

1
2
3
4
5
6
7
session_id → gpu_id

{"sess_abc123": "gpu-07", "sess_def456": "gpu-15", ...}

Turn 1: user arrives → assigned to GPU-07
        → record sess_abc123 → GPU-07
Turn 2: user returns → lookup sess_abc123 → force route to GPU-07

1
2
3
4
5
6
A group of GPUs [GPU-07, GPU-08, GPU-09, GPU-10] forms a Pod

Turn 1 → Pod-6 (handled by GPU-07)
Turn 2 → still within Pod-6, assigned to GPU-09
        → GPU-07 and GPU-09 are connected via NVLink/InfiniBand
        → KV Cache can transfer over high-speed interconnect (microsecond latency)

1
2
3
4
5
6
7
8
GPU VRAM (HBM)    → fastest but expensive, small capacity ← default cache location
  ↓ eviction
CPU memory (DDR)  → medium speed, large capacity (~512GB/1TB) ← cold data
  ↓ eviction
NVMe SSD          → slow, massive capacity ← coldest data

User Turn 1 ends → KV Cache from GPU → CPU memory
User Turn 2 returns → KV Cache from CPU → GPU (tens of ms vs seconds to recompute)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
User A and User B both use the same system prompt:

System: "You are a programming assistant. Reply in Chinese..."

User A asks: "write a quicksort"           ↓
User B asks: "write a bubble sort"          ↓
                    ↓
System prompt is identical → KV Cache shared across users

User A triggers Prefill first:
  [System] → [You] → [are] → [a] → ... → KV Cache stored

User B's request later:
  System prompt detected as exact match → directly reuse User A's cache
  Only needs to compute "write a bubble sort"

1
2
3
4
5
6
7
8
9
Input: "Please write a quicksort algorithm for me"

System detects:
  "Please write a" → cache hit (other users asked similar questions before)
  "quicksort algorithm for me" → may miss (specific content differs)

Actual effect:
  Longer prefix match → more compute saved
  Common openings like "You are a programming assistant..." → extremely high hit rate

1
2
3
4
5
6
7
8
9
10
11
12
13
14
User A finishes a conversation:
  Turn 1 → Turn 2 (within 5 minutes)
    → Likely assigned to same GPU (cache still in VRAM)
    → Cost: near zero

  Turn 1 → Turn 2 (30 min to 1 hour)
    → Cache may be offloaded to CPU memory
    → On return, reload to GPU
    → Cost: medium (tens of ms transfer latency)

  Turn 1 → Turn 2 (past timeout, typically 5-15 min)
    → Cache released
    → Full Prefill next time
    → Cost: high (seconds of compute)

1
2
3
4
5
6
7
8
Scenario                                                    | Cost
───────────────────────────────────────────────────────────|──────────
Turn 2 routes to same GPU, KV Cache still present          | Near 0
Turn 2 back to same Pod, cache via high-speed interconnect | Microsecond latency
Turn 2 on different Pod, cache loaded from CPU             | Tens of ms
Turn 2 on different Pod, cache evicted                     | Full Prefill (seconds)
Turn 2 prefix cache hit (Cache Hit pricing)                | ~$0.028/M token
Turn 2 no cache at all (Cache Miss pricing)                | ~$0.14/M token

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Don't try to avoid all cache misses.
Instead, split cache misses into "expensive" and "cheap."

Cheap cache miss:
  System prompt already cached (someone else's reuse)
  → Only compute the user-specific part
  → Most users fall here

Expensive cache miss:
  The entire prefix has never been computed by anyone
  → Relatively rare (system prompts hardly change)

API pricing already reflects this:
  Cache Hit = most cases
  Cache Miss = minority of cases
  Average cost is baked into the price

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Assumptions (2026 typical numbers):
  - Registered users: 100M
  - Daily active users (DAU): 10M (10%)
  - Concurrently generating users: ~100K (1% of DAU)

Peak VRAM per user:
  - Model weights (shared by all): 35GB (INT4 quantized, one copy)
  - KV Cache (while generating): ~8GB (on demand)

KV Cache needed for 100K concurrent users:
  100K × 8GB = 800TB VRAM

With the seven-layer optimization stack:
  - Paged Attention         → 3x VRAM efficiency
  - Continuous batching     → 10x concurrency efficiency
  - Idle users don't use cache → concurrency drops to 1/10
  - Sliding window / Summarization → 5x less per-user VRAM

Final GPU count needed:
  ~500-2000 H100/A100 GPUs (total cost ~$150M-$500M)

1
2
3
4
5
6
7
Prefill FLOPs  ≈  6 × P_activated × n  +  4 × d_hidden × n²

  Linear term (6Pn): all activated params × token count
    → Complexity O(n), dominates for short text

  Quadratic term (4dn²): attention score computation
    → Complexity O(n²), dominates for long text

1
2
3
4
5
Decode FLOPs (per step)  ≈  6 × P_activated  +  4 × d_hidden × n

  Each step is only O(n) attention + fixed activated params
  → Much smaller per-token compute than Prefill
  → Mainly limited by memory bandwidth, not compute

1
2
3
4
5
6
7
8
9
Linear dominant ←──→   Quadratic dominant
        ↓                  ↓
     Short text         Long text (>50K tokens)
     Compute ∝ n        Compute ∝ n²

  Impact of growing n:
    n × 2  → Linear × 2, Quadratic × 4
    n × 4  → Linear × 4, Quadratic × 16
    n × 10 → Linear × 10, Quadratic × 100