Why Decode Wants 64 GPUs: Tokens per Expert and the MoE Roofline

Part 1 of a four-part series on how DeepSeek serves mixture-of-experts models. Figures for DeepSeek-V4.1-Flash, checked against the DeepInfer release of 20 October 2026.

DeepSeek serves the decode phase of DeepSeek-V4.1-Flash on groups of 64 to 128 GPUs, although the same weights run prefill on as few as two. Prefill processes a whole prompt at once; decode then produces the response one step at a time, a few tokens per request per step, and it is decode that dominates serving cost for long responses. A mixture-of-experts model decodes efficiently only when each expert processes many tokens for every read of its weights. Top-k routing divides whatever batch a GPU holds by the ratio of experts to active experts, 384 to 6 for V4.1-Flash, before any expert sees a token. Spreading the experts across many GPUs offsets that division, because every GPU’s tokens are then pooled into each expert while each GPU still serves only its own requests. DeepSeek’s measurements show the effect directly: with the same eight requests on every GPU, a group of 64 GPUs sharing the experts (written EP64, for expert parallelism of width 64) delivers 36% more tokens per second per GPU than a group of eight (EP8), and the advantage shrinks to 6% at 256 requests per GPU.

The post builds that argument from one quantity, the number of tokens an expert processes per read of its weights, written $t$ throughout. It derives $t$ for a single GPU and for an expert-parallel group, places one V4.1-Flash expert on the GB200 roofline with the activation bytes that most back-of-envelope treatments leave out, and then tests the resulting model against DeepSeek’s published EP8 and EP64 decode curves. Replacing the routed-expert time in each EP64 measurement with the model’s EP8 value predicts the EP8 curve to within 6% at every batch size, with no constant fitted to the data, and the pattern of the remaining errors says something specific about what the model leaves out. The same model explains why prefill uses the opposite configuration.

The series. Part 1, why decode wants 64 GPUs (this post). Part 2, the slowest GPU sets the pace: expert load balancing. Part 3, dispatch and combine: what crosses the network and how many bytes. Part 4, one kernel for the whole MoE layer: DeepGEMM’s Mega-MoE.

A decode step is a byte budget

A GPU kernel is limited either by the rate at which it can move bytes from its high-bandwidth memory (HBM) or by the rate at which it can do arithmetic, and which limit applies is decided by its arithmetic intensity, the ratio of floating-point operations performed to bytes moved. The roofline model of Williams, Waterman and Patterson turns that into a single comparison. Attainable throughput is

where $I$ is the kernel’s intensity, $P_{\text{peak}}$ the peak arithmetic rate and $B_{\text{HBM}}$ the memory bandwidth. The two terms are equal at the ridge point $R = P_{\text{peak}} / B_{\text{HBM}}$. DeepSeek’s decode benchmark ran on 64 Blackwell (SM100) GPUs inside one NVLink scale-up domain, which on NVIDIA hardware means a GB200 NVL72-class system rather than an 8-GPU HGX B200 board. A GB200 GPU delivers 5 PFLOP/s of FP8 tensor-core arithmetic without structured sparsity against 8 TB/s of HBM bandwidth, so its ridge sits at $R = 625$ FLOPs per byte; a kernel below that intensity is paying for bandwidth, and one above it is paying for arithmetic. The roofline itself, and how to measure where a workload sits on it, are covered in an earlier post on GPU monitoring for inference, so this section only extracts what decode needs.

Every weight matrix in the model is read from memory once per decode step regardless of how many tokens use it. A dense $d \times d$ weight serving $b$ tokens performs $2d^2 b$ FLOPs against $d^2$ bytes at one byte per weight, so its intensity is $2b$, and decode efficiency is a question of how large $b$ can be made. DeepSeek’s benchmark uses speculative decoding with three drafted tokens, so each request contributes four tokens per step, one real and three to be verified, and throughout this post $b$ counts tokens, four per request. A mixture-of-experts layer changes the accounting in one specific way, because each token visits only a few of its experts.

V4.1-Flash has 384 routed experts and one shared expert in each of its 40 layers, and routes every token to 6 of the 384. Each routed expert is a SwiGLU feed-forward block with three matrices of shape $5120 \times 2304$ (the gate, up and down projections), so it holds

parameters and performs $F = 2P_e \approx 70.8$ MFLOP per token it processes. Routed experts are stored in MXFP4, the Open Compute Project’s microscaling format with 4-bit values and one 8-bit power-of-two scale per 32 values, which costs 4.25 bits per weight, so one expert occupies $W = 18.8$ MB. The shared expert has the same shape but is stored in FP8 and processes every token on every GPU, so like the attention layers it is untouched by how the routed experts are distributed. If a routed expert receives $t$ tokens in a step, it reads its 18.8 MB once and performs $tF$ FLOPs, so counting only weight bytes its intensity is

At $t = 1$ that is 3.76 FLOPs per byte, 166 times below the ridge. Everything that follows is about making $t$ large enough to close that distance, and the first obstacle is that routing makes $t$ much smaller than the batch.

Sparsity divides the batch

Suppose $B$ tokens share a single copy of the experts, as they would if the whole model sat on one GPU. Each token makes $k = 6$ expert visits spread over $E = 384$ experts, so under balanced routing the mean number of tokens per expert is

A dense layer would serve all $B$ tokens with each weight it reads, whereas the average expert serves one sixty-fourth of them. The ratio $E/k$ is the model’s sparsity, and it divides the batch before any expert sees a token.

The number of distinct experts the batch touches matters as much as the mean, because an expert that receives even one token must read all of its weights. With independent routing, expert $e$ receives none of the $B$ tokens with probability $(1 - k/E)^B$, so the expected number of experts touched is

A single token touches 6 experts, and the layer reads $6 \times 18.8 = 113$ MB of expert weights to produce it. That is far less than the 7.2 GB of routed experts the layer holds, and it is the reason a mixture of experts is fast to decode for a single user. The economics change as the batch grows. At $B = 256$ the batch touches 377 of the 384 experts on average, so nearly every weight in the layer is read, but the average expert serves only 4 tokens. Weight traffic per token has fallen from 113 MB to 27.7 MB, a factor of four, for a batch 256 times larger.

The obvious remedy is a larger batch, and the arithmetic shows why it does not work on one GPU. Counting weight bytes alone, the ridge needs $t \approx 166$; counting activation bytes as well, derived below, it needs about 192. At a sparsity of 64 that means roughly 12,300 tokens, about 3,000 requests, sharing one copy of the experts in every step. The question is moot for V4.1-Flash in any case, because its routed experts total 289 GB in MXFP4, more than any single Blackwell GPU holds.

Pooling the batch across GPUs

Expert parallelism places different experts on different GPUs and ships each token to the GPUs that hold its chosen experts, an all-to-all exchange called dispatch, then ships the results back, called combine. Experts were first sharded across devices by Shazeer and colleagues in 2017, GShard formalised the all-to-all pattern in 2020, and most large MoE deployments have used it since. DeepSeek runs attention data-parallel on the same GPUs: each GPU keeps its own requests and its own KV cache and runs attention for them locally, so a configuration written “EP64 DP64” means 64 GPUs, with only the MoE layers exchanging tokens.

The consequence for $t$ is that the batch an expert draws from is the whole group’s, so if each of $N_{\text{EP}}$ GPUs holds $b$ tokens and the group holds $E_{\text{phys}}$ expert slots, the mean number of tokens per expert becomes

The slot count exceeds 384 because the load balancer keeps redundant copies of popular experts (Part 2 covers how), and the formula must count them because they are present whether or not a given step needs them. DeepSeek’s benchmark adds at least 32 redundant slots and rounds the total up to a multiple of the group width, which gives 416 slots at EP8, 52 per GPU, and 448 at EP64, seven per GPU; its production decode rule of one redundant slot per GPU also gives 448 at EP64.

The per-GPU accounting shows exactly what changes with group width. In one MoE layer under balanced routing, each GPU receives $b\,k$ expert visits on average, so the arithmetic $C_g$, the activation traffic $A_g$ and the weight traffic $W_g$ of one GPU are

where $a$ is the number of bytes each expert visit moves for the token itself, and the per-GPU intensity is

Dividing numerator and denominator by $E_{\text{phys}}/N_{\text{EP}}$ turns $b\,k\,N_{\text{EP}}/E_{\text{phys}}$ into $t$, so this is the per-expert intensity with the pooled $t$, written per GPU. The arithmetic and the activation traffic depend only on the GPU’s own batch, whereas the weight traffic depends only on how many experts the GPU holds, and since the experts are divided among the GPUs of the group, it falls in proportion to the group width. The group as a whole reads every expert once per step whatever its width, and a wider group splits that read among more GPUs while each still serves its own requests, which is why more GPUs make each GPU faster. At 64 requests per GPU, a GPU in an EP8 group reads 39.1 GB of expert weights per step across the 40 layers, which takes 4.9 ms at 8 TB/s; in an EP64 group it reads 5.3 GB, which takes 0.66 ms. The exchanges also change with width, which the model does not price and which reappears in its errors below.

Walking up the roofline

The naive intensity counts only weight bytes, and it overstates the efficiency of a busy expert considerably. Each expert visit also moves the token through HBM: a dispatched token lands in a receive buffer and is read once, and the expert’s output is written once, while in a fused kernel the $2 \times 2304$ intermediate values of the SwiGLU stay on chip. Assuming tokens arrive in FP8, the format DeepSeek’s dispatch kernels use for decode, and results leave in BF16, each visit moves $a = d(1 + 2) = 15{,}360$ bytes with $d = 5120$, and the intensity becomes

For small $t$ the weight term dominates and $I \approx 3.76\,t$, as before. For large $t$ the activation term takes over and the intensity flattens toward a ceiling of $F/a = 4{,}608$ FLOPs per byte, the intensity of the arithmetic on one token’s bytes alone. The difference is material at the batch sizes DeepSeek runs: at 256 requests per GPU in an EP64 group each expert receives 878 tokens, where the naive formula gives 3,304 FLOPs per byte and the full one gives 1,924. The 15 KB per visit is also the traffic that dispatch and combine move across the network, which Part 3 returns to.

The ridge crossing follows directly. Setting $I(t^*) = R$ and solving gives the number of tokens per expert at which an expert’s arithmetic takes as long as its memory traffic:

With the GB200’s ridge of 625, an 18.8 MB expert, 70.8 MFLOP per token and 15,360 bytes per visit, the numerator is $1.18 \times 10^{10}$ and the denominator $6.12 \times 10^{7}$, so $t^* \approx 192$. In an EP64 group with 448 slots, top-6 routing and four tokens per request, $t = 192$ is reached at about 56 requests per GPU. DeepSeek reports that production runs at about 60 concurrent sequences per GPU, at EP64 to EP128, verifying about 3.6 drafted tokens per request on average, which puts each expert at roughly 236 tokens per step at EP64, just past the ridge. In an EP8 group the same crossing would need 416 requests per GPU, beyond the largest batch DeepSeek measured, so in that configuration the experts stay memory-bound at every operating point on its curve.

Four levers raise $t$ or lower the bytes it has to overcome, and V4.1-Flash’s serving design pulls all of them. Wider expert parallelism pools more GPUs’ tokens into each expert. Speculative decoding verifies several tokens per request per step, which multiplies $b$ without adding requests; the mechanism is covered in the post on speculative decoding. A small KV cache lets each GPU hold more requests, which is the subject of a later section. MXFP4 storage works from the other side, at 4.25 bits per weight against FP8’s 8, which nearly halves $W$ and with it $t^*$. The whole calculation fits in a dozen lines of Python, and the two printed values are the ridge crossing and the intensity at 64 requests per GPU in an EP64 group:

P = 3 * 5120 * 2304              # parameters per routed expert (gate, up, down)
W = P * 4.25 / 8                 # MXFP4: 4-bit values plus an 8-bit scale per 32
F = 2 * P                        # FLOPs per token
a = 5120 * (1 + 2)               # bytes per visit: FP8 token in, BF16 result out

def tokens_per_expert(tokens_per_gpu, ep, top_k=6, slots=448):
    return tokens_per_gpu * ep * top_k / slots

def intensity(t):
    return F * t / (W + a * t)

ridge = 5.0e15 / 8e12            # GB200: FP8 FLOP/s over HBM bytes/s
print(round(ridge * W / (F - ridge * a)))               # t* = 192
print(round(intensity(tokens_per_expert(64 * 4, 64))))  # 701 at 64 requests per GPU

The figure below runs the same model, and it carries presets for other mixtures of experts. Kimi K3 divides its batch by 56 and lands close to V4.1-Flash. DeepSeek-V3 stores its experts in FP8 and needs about twice as many tokens per expert to reach the ridge. Mixtral 8x7B divides its batch by only 4, runs its BF16 experts on the BF16 roof, and with eight experts cannot be spread across more than eight GPUs, a case the check-yourself questions at the end work through.

Testing the model against DeepSeek’s measurements

DeepSeek’s deployment notes for DeepInfer include a chart of V4.1-Flash decode measured at EP8 and at EP64 with the same number of requests per GPU, from 8 to 256, at a context length of 131,072 tokens. The benchmark forces balanced routing, verifies four tokens per request per step, and plots throughput under DeepSeek’s convention of a 75% draft acceptance rate, $\tau = 1 + 3 \times 0.75 = 3.25$ accepted tokens per step. Per-user speed is therefore $\tau$ divided by the step time, so each plotted point yields a step time: 325.3 tokens per second per user at EP64 with 8 requests means $3.25 / 325.3 = 10.0$ ms per step. DeepSeek publishes the chart without a table, so the points were digitized from it, and as a consistency check the plotted throughput per GPU equals requests times per-user speed to within 1% at every point.

The model cannot predict the absolute step time. A decode step contains a great deal besides routed experts: attention over a 131,072-token context, the dense projections and the shared expert, sampling, and more than 80 all-to-all exchanges (dispatch and combine in each of the 40 layers, plus the drafter’s). What the model can predict is the difference between the two configurations, by separating the terms that change with width from those that do not. Attention is data-parallel, so its cost depends on the requests and context a GPU holds and not on how many GPUs share the experts, and the dense weights and the shared expert are replicated on every GPU in either configuration. The routed experts change with width, and so do those of the speculative drafter, which has three small MoE stages of its own, 128 experts each with top-3 routing, split across the group like the main model’s. The exchanges change too, and the model does not price them. Taking the measured EP64 step time, subtracting the model’s routed-expert time at EP64 and adding its routed-expert time at EP8 gives a prediction of the EP8 step with nothing fitted:

where the expert term sums the roofline time of one GPU’s routed experts over the 40 layers of the main model and the 3 stages of the drafter, using the per-GPU quantities defined earlier:

Everything in $X$ comes from the expert shapes and the GB200’s datasheet peaks. The table compares the prediction with the measurement at each batch size DeepSeek ran.

Measurements digitized from DeepSeek’s EP8 versus EP64 chart; predictions from the model with GB200 peak figures.

The prediction falls within 6% of the measurement at every batch size, and the errors all point the same way, which is more informative than their size. At small batches both configurations are memory-bound in their experts, so the predicted gap is simply the extra weight bytes EP8 reads: 52 experts per GPU against 7 in the main model and 20 against 3 in the drafter, 34.8 GB per step. The model puts that gap at 4.35 ms, and DeepSeek measured 3.6 to 3.8 ms. That shortfall cannot come from bandwidth, because reading 34.8 GB in 3.64 ms would require 9.6 TB/s, more than the GPU’s peak. Either part of EP8’s weight reading overlaps other work in the step, or EP64 pays something EP8 does not, most plausibly in exchanges spanning 64 GPUs instead of 8, and the model cannot tell these apart. At large batches the gap collapses, from 3.6 ms to 1.75 ms, and the model reproduces the collapse: both configurations perform the same arithmetic per GPU, about 17 TFLOP of expert work per step at 256 requests, which takes 3.5 ms at peak, and at EP8 the memory traffic takes only 1.9 ms longer than that. The roofline’s $\max$ assumes memory traffic and arithmetic overlap perfectly. A model that adds them instead misses the two largest batches by 4% and 7%, though with kernels running below their datasheet peaks the two assumptions trade off against each other, so the large-batch agreement supports overlap without proving it. The figure’s controls switch between the two assumptions and between the 8 TB/s headline bandwidth and the 7.7 TB/s some vendors list, which moves the worst error to 6.8%.

The same comparison shows what the model does not explain. At 8 requests per GPU, the EP64 step takes 10.0 ms, of which the routed experts account for 0.69 ms. The remaining 9.3 ms, taken as equal at EP8, consists of per-request work and fixed latency, including the dense layers, the shared expert, sampling and the latency of the exchanges. That floor is why even the smallest batch takes 10 ms per step, and the exchanges within it are the subject of Part 3.

The batch has to fit

Every lever above assumes a GPU can hold the requests that produce its tokens, and in decode each request carries a KV cache whose size grows with context. V4.1-Flash stores 890 bytes of KV cache per token across the whole model, about a quarter of DeepSeek-V4-Flash, through three design choices: layers share their compressed KV and sparse-attention indices with neighbouring layers, the main cache is stored in FP4, and the decoder half of the model projects its KV from the encoder half instead of keeping its own. At that size, 256 requests at the 131,072-token context of DeepSeek’s benchmark occupy about 30 GB per GPU, a sixth of a GB200’s memory, and DeepSeek states that its decode deployment is not bound by cache capacity. The attention design is what allows the batch to grow until the experts are saturated; how attention caches got this small is the subject of the evolution of attention series.

Prefill wants the opposite

Prefill is the opposite case. It processes a prompt’s tokens in large chunks rather than a few at a time, and a single chunk already holds thousands of tokens. At 16,384 tokens per GPU per step, an EP2 group puts 502 tokens on each expert, three times the 170 that marks the ridge on the 8-GPU B200 machines DeepSeek uses for prefill, and at the 32,768-token cap DeepSeek places on a prefill step it puts about 1,000. V4.1-Flash’s encoder-decoder split sharpens the point, since only the 20 encoder layers process the whole chunk while the decoder replays roughly the last 140 tokens. Prefill experts are compute-bound at the narrowest width that can hold them, so widening the group would add communication without improving arithmetic efficiency. That narrowest width is set by memory: EP1 cannot hold the 289 GB of routed experts, while at EP2 each GPU holds 196 expert slots, about 147 GB.

DeepSeek’s deployment rule states the conclusion directly: prefill uses the smallest EP that HBM allows, between 2 and 8, and decode uses the largest EP that the available GPUs allow, between 64 and 128. The two phases want opposite configurations of the same weights, which is a strong argument for running them on separate pools of GPUs, and DeepSeek does, with prefill on ordinary 8-GPU machines and decode on 64- to 128-GPU supernodes, and roughly one prefill GPU per decode GPU.

What the formula assumes

The formula for $t$ uses the mean load per expert, and the step does not run at the mean. Dispatch and combine are collective exchanges, so every GPU in the group waits for the busiest one at every MoE layer, and the relevant load is the maximum. DeepSeek’s benchmark forces balanced routing precisely so that this effect is removed from its curves. Real routing is skewed toward popular experts, and even perfectly uniform random routing leaves some GPUs busier than others in any single step, because the choices of a finite batch never spread evenly. Each GPU receives $b\,k$ expert visits on average with a relative spread of about $1/\sqrt{b\,k}$, whatever the group width, and the step waits for the largest of $N_{\text{EP}}$ such draws, which grows with $N_{\text{EP}}$. Simulating top-6 routing with every expert equally likely, the busiest GPU in an EP64 group carries 17% more expert visits than the average at 8 requests per GPU, 6% more at 60, and 3% more at 256, while at 8 requests per GPU the busiest GPU of an EP8 group is 10% above the mean. DeepSeek reports a measured ratio of 1.090 between the busiest GPU and the mean for decode with its expert load balancer switched on; the notes do not state the group width or whether the ratio counts tokens or time, so the comparison with the 1.06 that randomness alone produces at 60 requests per GPU is indicative only. How the balancer removes the skew of real routing, and why it copies experts rather than changing where tokens are routed, is the subject of Part 2.

Every token that pooling brings to an expert also has to cross the network twice per layer, 15 KB per visit, six visits per token, 40 layers per step, which is why DeepSeek keeps each decode group inside a single NVLink scale-up domain. Part 3 measures that traffic and the latency of the exchanges, which sit inside the 9.3 ms the model leaves unexplained at small batches and are the leading candidate for its systematic error. Part 4 covers DeepGEMM’s Mega-MoE kernel, which fuses dispatch, both expert GEMMs, the activation and combine into one kernel so that the transfers overlap the arithmetic.

The model has limits. It treats each GPU’s routed experts with a roofline at datasheet peaks and nothing else, so kernel efficiency below peak, communication and imbalance are outside it, and kernel efficiency does not cancel in the comparison, since it would scale the swapped expert terms. Its hardware figures are the GB200’s; with B200 figures the errors are within 6% as well. DeepSeek’s curves assume a 75% draft acceptance rate rather than measuring it, and they are read from a chart. None of these changes the conclusion, which is that expert-parallel width alters one term of the decode step, the expert weight bytes each GPU reads, and that the size of that term accounts for most of DeepSeek’s measured gain from wide expert parallelism.

Check yourself

References

1. DeepSeek-AI (2026). DeepInfer, README and deployment Q&A. Source of the EP8 versus EP64 decode chart, the decode and prefill parallelism rules, the expert load balancing settings, the production operating point and the benchmark methodology (examples/benchmark.py, examples/README.md).
2. DeepSeek-AI (2026). DeepSeek-V4.1-Flash model card, configuration and reference inference code. Expert shapes, routing, layer count, the DSpark drafter’s experts and the 890-byte KV cache.
3. Williams, S., Waterman, A. and Patterson, D. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM 52(4).
4. Shazeer, N. et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.
5. Lepikhin, D. et al. (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. Expert parallelism with all-to-all dispatch.
6. Fedus, W., Zoph, B. and Shazeer, N. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.
7. DeepSeek-AI (2024). DeepSeek-V3 Technical Report. Fine-grained experts, auxiliary-loss-free balancing and the expert-parallel serving design V4.1 inherits.
8. Rouhani, B. et al. (2023). Microscaling Data Formats for Deep Learning. The MX formats, including MXFP4 with a shared 8-bit scale per 32 values.
9. DeepSeek-AI. DeepEP and DeepGEMM. The dispatch and combine kernels and the grouped GEMMs, including Mega-MoE.
10. Moonshot AI (2026). Kimi K3 Technical Report. Expert shapes for the Kimi K3 preset.
11. Jiang, A. Q. et al. (2024). Mixtral of Experts. Expert shapes for the Mixtral preset.
12. Lenovo Press (2025). ThinkSystem NVIDIA HGX B200 product guide (LP2226). B200 throughput and memory capacity. GB200 figures (5 PFLOP/s dense FP8, 8 TB/s, 186 GB usable) are from SemiAnalysis InferenceX.

Every number in this post is produced by a small Python model with tests for each figure quoted, and the interactive figures run a JavaScript port checked against it to machine precision. Both live in the blog’s repository under research/deepseek/part1/.