Hardware-software codesign is the discipline of letting the workload pick the silicon. You profile what the software actually does, build a model that predicts time from hardware parameters, then move the parameters until the model says the workload is fast — and only then commit to gates. This post runs that loop end to end on one workload: a tool-using Claude agent. We invent a representative trace, fit a hardware model to it, and use the model to design and optimize a custom accelerator across both scale-up (one tightly-coupled pod) and scale-out (many pods on a network).

Disclosure. The traces, model parameters, and architecture here are synthetic and illustrative — invented to make the methodology concrete. They are not Anthropic figures and do not describe any real Claude deployment. The point is the codesign method, which transfers to whatever real numbers you can measure.

The trace: where the time goes

An agentic session is not one forward pass — it is a loop. The model reads a context, emits reasoning and a tool call, the harness runs the tool (a shell command, a file read, a web fetch), the result is appended to the context, and the model is invoked again. A single task may run dozens of these turns. The wall-clock cost therefore splits into phases that stress very different parts of a machine.

We collected (synthetically) 1,000 agent sessions and profiled each into five phases. The means are below. A session averages 40 turns, ingests 4.4M prefill tokens (prompt plus re-read tool output and growing context), and generates 18,000 output tokens.

PhaseWhat runsTime (s)Share
PrefillIngest context + tool output (accelerator, compute-bound)44.030.9%
DecodeAutoregressive generation (accelerator, bandwidth-bound)54.037.9%
Tool executionShell / filesystem / web on the host (no accelerator)31.021.8%
Scale-up commTensor-parallel all-reduce inside the pod9.06.3%
SchedulerBatching, routing, serialization overhead4.02.8%
Totalper session, mean of 1,000142.0100%

Figure 1. The mean session decomposed. Two observations drive everything that follows: (i) the accelerator-bound work (prefill + decode = 69%) is split between a compute-bound and a bandwidth-bound phase, and (ii) more than a fifth of the session is host tool time the accelerator cannot touch.

Two facts in this trace are unusual relative to a chat workload and are exactly what makes agents interesting to codesign for:

  • Prefill is enormous. Tool outputs (a 2,000-line file, a web page, a stack trace) get appended and re-ingested. A chat turn prefills a prompt once; an agent prefills a growing transcript on every turn. Here prefill rivals decode.
  • The agent is latency-bound, not throughput-bound. A single interactive session runs at batch size $B=1$, so decode badly underutilizes HBM bandwidth, and the host tool time sits on the critical path as pure accelerator idle.

A hardware model for each phase

To design hardware we need time as a function of hardware parameters. We model a frontier mixture-of-experts decoder — call it a Claude-class model — with the illustrative parameters below, served on a baseline pod $\mathcal{P}_0$ of 8 accelerators in tensor parallel.

QuantitySymbolValue
Active parameters / token$P$$40\times10^9$
Weight bytes / param (FP8)$b_w$1
Layers$L$64
Hidden dimension$d$8,192
KV bytes / token (MLA, FP8)$\kappa$40,000
Pod compute (FP8)$C_0$32 PFLOP/s
Pod HBM bandwidth$B_0$40 TB/s
Pod HBM capacity$H_0$1.5 TB
Intra-pod link BW (per GPU)$W_0$3.6 TB/s

Prefill — compute bound

Prefill processes tokens in parallel, so it is a dense GEMM and runs near the compute roof. For a transformer, the FLOPs to ingest $N$ tokens have a linear term (the weight matmuls, two FLOPs per parameter per token) and a quadratic term (attention, which grows with context position). With mean ingest context $\bar{S}_p$:

$$F_\text{pre}(N) = \underbrace{2PN}_{\text{matmuls}} \;+\; \underbrace{4Ld\,\bar{S}_p\,N}_{\text{attention}}$$

The attention term counts the two batched matmuls $QK^\top$ and $\text{(softmax)}V$, each $2d$ FLOPs per token per key, summed over $L$ layers and a mean of $\bar{S}_p$ keys. Dividing by the effective compute — peak times a prefill efficiency $\eta_c$ (MFU) — gives the prefill time:

$$t_\text{pre} = \frac{F_\text{pre}(N)}{\eta_c\, C}$$

Decode — bandwidth bound

Decode generates one token per step. At step $t$ the machine must stream every active weight once and read the full KV cache for the context. The classic roofline result for autoregressive decode [Williams 2009, Pope 2023] is that at small batch this is memory-bandwidth bound: the bytes moved, not the FLOPs, set the clock. Over $N_d$ generated tokens at mean context $\bar{S}_d$:

$$M_\text{dec} = N_d\big(\,\underbrace{P b_w}_{\text{weights}} + \underbrace{\bar{S}_d\,\kappa}_{\text{KV read}}\,\big), \qquad t_\text{dec} = \frac{M_\text{dec}}{\eta_m\, B}$$

where $\eta_m$ is the achieved fraction of peak HBM bandwidth. At $B=1$ a single token cannot amortize weight reads or hide DRAM latency, so $\eta_m$ is low — this is the central pain point a custom decode engine must fix.

Scale-up communication — latency bound

Tensor parallelism over $k$ accelerators inserts two all-reduces per layer per step. Modeling each collective with the standard latency–bandwidth ($\alpha$–$\beta$) cost [Thakur 2005], with per-collective launch+link latency $\alpha$ and a ring-reduce payload of $B d\, b_w$ bytes:

$$t_\text{comm} = N_d \cdot 2L\left(\alpha + \frac{k-1}{k}\,\frac{B d\, b_w}{W}\right)$$

At $B=1$ the payload is $d\,b_w = 8{,}192$ bytes — at $W_0 = 3.6$ TB/s that transfers in $\approx 2$ ns, utterly dominated by $\alpha$. Agent decode all-reduce is therefore a pure latency problem: it scales with the number of collectives $2L N_d$, not with link bandwidth. Remember that — it tells us what to optimize.

The whole session

Tool execution $t_\text{tool}$ runs on the host and is, for now, fully serial with the accelerator (the model waits for the tool; the tool waits for the model). The total is the sum:

$$\boxed{\,T = t_\text{pre} + t_\text{dec} + t_\text{comm} + t_\text{tool} + t_\text{sched}\,}$$

Calibrating the model to the trace

A model is only useful if it reproduces the measurement. We fit the two efficiency knobs $\eta_c, \eta_m$ and the latency $\alpha$ to the Figure 1 phase times, then keep them fixed for all design work. Let us prove the model reconstructs the trace.

Proof — decode time Bytes per generated token: $P b_w + \bar{S}_d \kappa = (40\times10^9)(1) + (60{,}000)(40{,}000) = 4.0\times10^{10} + 2.4\times10^{9} = 4.24\times10^{10}$ B.
Over $N_d = 18{,}000$ tokens: $M_\text{dec} = 18{,}000 \times 4.24\times10^{10} = 7.632\times10^{14}$ B.
With $\eta_m = 0.35$ at $B=1$: effective BW $= 0.35 \times 40\times10^{12} = 1.4\times10^{13}$ B/s.
$t_\text{dec} = \dfrac{7.632\times10^{14}}{1.4\times10^{13}} = \mathbf{54.5\ s}$. Matches the 54.0 s trace. ∎
Proof — prefill time With $N = 4.4\times10^6$, $\bar{S}_p = 30{,}000$:
matmul term $2PN = 2(40\times10^9)(4.4\times10^6) = 3.52\times10^{17}$ FLOP.
attention term $4Ld\bar{S}_p N = 4(64)(8192)(30{,}000)(4.4\times10^6) = 2.768\times10^{17}$ FLOP.
$F_\text{pre} = 3.52\times10^{17} + 2.768\times10^{17} = 6.288\times10^{17}$ FLOP.
With $\eta_c = 0.45$: effective compute $= 0.45 \times 32\times10^{15} = 1.44\times10^{16}$ FLOP/s.
$t_\text{pre} = \dfrac{6.288\times10^{17}}{1.44\times10^{16}} = \mathbf{43.7\ s}$. Matches the 44.0 s trace. ∎
Proof — scale-up comm At $B=1$ the payload time is negligible, so $t_\text{comm} \approx N_d \cdot 2L\,\alpha$.
With $\alpha = 4\ \mu\text{s}$, $L=64$, $N_d = 18{,}000$: $t_\text{comm} = 18{,}000 \times 128 \times 4\times10^{-6} = \mathbf{9.2\ s}$. Matches the 9.0 s trace. ∎

Summing: $43.7 + 54.5 + 9.2 + 31.0 + 4.0 = 142.4$ s, versus the measured 142.0 s — within 0.3%. The model is calibrated. We can now place both phases on a roofline and read off what kind of machine each one wants.

Figure 2. Roofline of the baseline pod. Prefill (●) sits against the sloped compute roof — it wants FLOP/s. Decode (●) sits far down the bandwidth ramp at arithmetic intensity $I = 2b_w^{-1}\!\approx 2$ — it wants bytes/s and lives nowhere near the ridge point $I^\star = C/B$. One workload, two opposite hardware appetites.

This split is the whole reason codesign beats buying a bigger general-purpose chip: a single balanced die is a compromise between two phases that want opposite things. We can do better by (a) rebalancing the silicon, and (b) physically separating the phases. First we need to know which lever — scale-up or scale-out — each fix pulls.

Scale-up vs. scale-out

Two axes of growth, two different physics:

Scale-upScale-out
DefinitionMore resource in one coherent domain (pod / TP group)More pods on a network
Couples viaOn-package + NVLink-class fabric (~TB/s, ~µs)RDMA network (~100s GB/s, ~10s µs)
HelpsSingle-agent latency: prefill, decode, all-reduceThroughput, disaggregation, prefix-cache reuse
Bounded byReticle, package power, link radixBisection BW, load balance, tail latency

The trace tells us we need both. The agent is one user, so most of the 142 s is a latency problem — that is scale-up's job: make prefill compute denser, make decode bandwidth denser, and crush the all-reduce $\alpha$. But the fleet serves thousands of agents, and the only way to raise the decode efficiency $\eta_m$ is to batch many of them together, which is a throughput problem — scale-out's job, alongside disaggregating prefill from decode so the two phases stop fighting over the same die. We model scale-out throughput in §9; first we squeeze the scale-up unit.

Optimizing the silicon: area allocation

Here is the first genuine design decision. A die has a fixed area (equivalently a fixed power budget) $A$. Spend it on compute units or on memory interfaces? More compute speeds prefill; more bandwidth speeds decode. Let $a_c$ be the area given to compute and $a_b$ to bandwidth, with linear technology yields $C = \gamma_c a_c$ and $B = \gamma_b a_b$. The accelerator-bound time as a function of the split is:

$$T_\text{acc}(a_c, a_b) = \frac{F_\text{pre}}{\eta_c \gamma_c\, a_c} + \frac{M_\text{dec}}{\eta_m \gamma_b\, a_b} \equiv \frac{\mathcal{P}}{a_c} + \frac{\mathcal{Q}}{a_b}, \qquad a_c + a_b = A$$

where $\mathcal{P} \equiv F_\text{pre}/(\eta_c\gamma_c)$ and $\mathcal{Q} \equiv M_\text{dec}/(\eta_m\gamma_b)$ are the (fixed) workload demands. This is a constrained minimization; solve it with a Lagrange multiplier.

Proof — optimal area split Minimize $\mathcal{L} = \mathcal{P}/a_c + \mathcal{Q}/a_b + \lambda(a_c + a_b - A)$.
$\partial_{a_c}\mathcal{L} = -\mathcal{P}/a_c^2 + \lambda = 0 \Rightarrow \lambda = \mathcal{P}/a_c^2$.
$\partial_{a_b}\mathcal{L} = -\mathcal{Q}/a_b^2 + \lambda = 0 \Rightarrow \lambda = \mathcal{Q}/a_b^2$.
Equating: $\mathcal{P}/a_c^2 = \mathcal{Q}/a_b^2 \Rightarrow a_c/a_b = \sqrt{\mathcal{P}/\mathcal{Q}}$.
With $a_c + a_b = A$: $$a_c^\star = A\frac{\sqrt{\mathcal{P}}}{\sqrt{\mathcal{P}}+\sqrt{\mathcal{Q}}}, \quad a_b^\star = A\frac{\sqrt{\mathcal{Q}}}{\sqrt{\mathcal{P}}+\sqrt{\mathcal{Q}}}.$$ Substituting back gives the minimum time $T^\star = \dfrac{(\sqrt{\mathcal{P}}+\sqrt{\mathcal{Q}})^2}{A}$. The Hessian is diagonal with entries $2\mathcal{P}/a_c^3, 2\mathcal{Q}/a_b^3 > 0$, so the stationary point is a minimum. ∎

This is the equalized-marginal-return condition: at the optimum, the time saved by the last unit of area is the same whether you spend it on compute or bandwidth ($\lambda = \mathcal{P}/a_c^2 = \mathcal{Q}/a_b^2$). Now plug in our trace. Normalize $A = 1$. The baseline pod is a training-derived part biased toward compute: take its split as $a_{c0} = 0.65$, $a_{b0} = 0.35$. The phase times pin the demands:

Proof — the rebalancing gain From the calibration, $T_\text{acc}$ at the current split is $t_\text{pre} + t_\text{dec} = 43.7 + 54.5 = 98.2$ s, with $\mathcal{P}/a_{c0} = 43.7$ and $\mathcal{Q}/a_{b0} = 54.5$.
Hence $\mathcal{P} = 43.7 \times 0.65 = 28.4$ and $\mathcal{Q} = 54.5 \times 0.35 = 19.1$.
Optimal split: $a_c^\star = \dfrac{\sqrt{28.4}}{\sqrt{28.4}+\sqrt{19.1}} = \dfrac{5.33}{5.33 + 4.37} = 0.549$, so $a_b^\star = 0.451$.
$T^\star = (\sqrt{28.4}+\sqrt{19.1})^2 = (9.70)^2 = \mathbf{94.1\ s}$, versus 98.2 s. A 4.2% cut from rebalancing the same silicon — the baseline was over-provisioned on compute for this bandwidth-hungry workload. ∎

Figure 3. Accelerator-bound time vs. the compute fraction $a_c$ of a fixed-area die. The bathtub minimum sits at $a_c^\star = 0.549$ (◆), shifted left of the compute-heavy baseline (○, $a_c = 0.65$). Rebalancing is the cheap win; the curve is flat near the bottom, which is why the structural moves below matter more.

A 4% win is real but small — the curve is flat near its floor, so you cannot rebalance your way to a different machine. The big gains are structural, and to see why we have to confront the part of the trace the accelerator never touches.

The Amdahl wall and the tool bubble

Tool execution is 31 s of the 142 s session, and it runs on the host. No amount of faster silicon touches it. Amdahl's law [Amdahl 1967] sets a hard ceiling on what acceleration can buy. Let $s$ be the fraction of the session that is not acceleratable and $1-s$ the fraction we speed up by a factor $k$:

$$\text{Speedup}(k) = \frac{T}{\dfrac{(1-s)\,T}{k} + s\,T} = \frac{1}{\dfrac{1-s}{k} + s}, \qquad \lim_{k\to\infty}\text{Speedup} = \frac{1}{s}$$
Proof — the ceiling for pure acceleration Non-acceleratable host work is the tool time: $s = t_\text{tool}/T = 31.0/142.0 = 0.218$.
Therefore $\lim_{k\to\infty}\text{Speedup} = 1/0.218 = \mathbf{4.59\times}$.
Even with an infinitely fast accelerator, a session cannot beat 4.59×, finishing no faster than $142/4.59 = 30.9$ s — exactly the tool time, as it must. ∎

That is the wall. We could pour unbounded silicon at prefill and decode and never pass 4.6×. The only way through is codesign across the hardware–software boundary: overlap the tool with the accelerator. While a shell command or web fetch runs on the host, the accelerator is idle — but it could be speculatively prefilling the likely next context, running a draft model, or prefetching KV for probable branches. If a fraction $\varphi$ of tool time is hidden behind useful accelerator work, the non-acceleratable fraction shrinks to $s' = (1-\varphi)\,t_\text{tool}/T$.

Proof — overlap moves the wall With overlap fraction $\varphi = 0.70$: exposed tool time $= 0.30 \times 31.0 = 9.3$ s, so $s' = 9.3/142.0 = 0.0655$.
New ceiling $\lim_{k\to\infty}\text{Speedup} = 1/0.0655 = \mathbf{15.3\times}$.
Overlap alone — a scheduling change enabled by the right hardware (a draft engine + async DMA) — lifts the asymptote from 4.6× to 15.3×, a 3.3× raise in the ceiling before we add a single FLOP. ∎

Figure 4. Session speedup vs. accelerator speedup factor $k$. Without overlap (lower curve) the session is pinned under 4.6× no matter how fast the chip. With 70% tool overlap (upper curve) the ceiling rises to 15.3×. The codesign lesson: past a point, scheduling beats silicon.

The codesigned architecture

The trace and the models above name five problems and prescribe five fixes. Each fix targets a specific term in $T = t_\text{pre} + t_\text{dec} + t_\text{comm} + t_\text{tool} + t_\text{sched}$.

  1. Disaggregate prefill and decode (scale-out). Stop forcing one die to serve two appetites. Route prefill to compute-dense pods and decode to bandwidth-dense pods [Patel 2023, Zhong 2024]. Each pod is built at its own roofline ridge, and the two phases pipeline instead of contend. We give the prefill pod $C_p = 64$ PFLOP/s and the decode pod $B_p = 80$ TB/s.
  2. Persistent prefix cache (scale-out). The 4.4M prefill tokens are mostly re-ingested context. A network-addressable KV/prefix cache [Kwon 2023] lets a turn skip everything already seen. A 66% hit rate cuts ingested tokens to $N' = 0.34 \times 4.4\text{M} = 1.5\text{M}$.
  3. Batch decode across agents (scale-out). At the fleet level, co-schedule many agents on the decode pod so weight reads amortize, lifting $\eta_m$ from 0.35 to 0.70 (see §9 for the roofline that justifies this).
  4. In-network reduction + wider scale-up domain. Decode all-reduce is pure latency. Fold the reduction into the switch and grow the coherent domain so a step needs fewer hops, cutting $\alpha$ from 4 µs to 1 µs.
  5. Speculative tool overlap (codesign). Add a small on-die draft engine and asynchronous host DMA so that, while a tool runs, the accelerator speculatively prefills the predicted result and drafts the continuation. This realizes the $\varphi = 0.70$ overlap from §6.

Note how the fixes divide across the two axes. Latency fixes (1, 4, 5) are scale-up: they make the critical path of one agent shorter. Efficiency fixes (2, 3) are scale-out: they only pay off when the fleet packs many agents and caches across them. Codesign is choosing the right axis for each bottleneck the trace exposes.

Projected Claude performance

Now run the calibrated model forward with the codesigned parameters and prove the new session time, phase by phase. The sliders below let you sweep the design knobs and watch the projection move; the defaults are the design point we justify here.

Codesign knobs — drag to re-derive the projection
Prefix-cache hit rate  66%
Prefill compute  64 PFLOP/s
Decode HBM BW  80 TB/s
Decode BW efficiency $\eta_m$  0.70
All-reduce latency $\alpha$  1.0 µs
Tool overlap $\varphi$  70%
Baseline
142.0 s
Codesigned
35.6 s
Speedup
4.0×
Amdahl ceiling
15.3×

Figure 5. Baseline vs. codesigned session, same phase decomposition. Decode and prefill collapse (denser, disaggregated silicon + caching), comm shrinks (lower $\alpha$), and most of the tool bubble is hidden behind speculative accelerator work.

The default design point, derived term by term:

Proof — prefill on the compute-dense pod Cached ingest tokens $N' = (1-0.66)\times 4.4\times10^6 = 1.496\times10^6$.
$F_\text{pre}(N') = 2PN' + 4Ld\bar{S}_p N' = 2(40\times10^9)(1.496\times10^6) + 4(64)(8192)(30{,}000)(1.496\times10^6)$
$= 1.197\times10^{17} + 9.41\times10^{16} = 2.138\times10^{17}$ FLOP.
On $C_p = 64$ PFLOP/s at $\eta_c = 0.45$: $t_\text{pre}' = \dfrac{2.138\times10^{17}}{0.45\times64\times10^{15}} = \dfrac{2.138\times10^{17}}{2.88\times10^{16}} = \mathbf{7.4\ s}$ (from 43.7 s). ∎
Proof — decode on the bandwidth-dense pod $M_\text{dec} = 7.632\times10^{14}$ B is unchanged (same tokens, same KV).
On $B_p = 80$ TB/s at $\eta_m = 0.70$: effective BW $= 0.70\times80\times10^{12} = 5.6\times10^{13}$ B/s.
$t_\text{dec}' = \dfrac{7.632\times10^{14}}{5.6\times10^{13}} = \mathbf{13.6\ s}$ (from 54.5 s) — a 4× cut, split as $2\times$ raw bandwidth and $2\times$ efficiency. ∎
Proof — comm and tool Comm with $\alpha = 1\ \mu\text{s}$: $t_\text{comm}' = 18{,}000 \times 128 \times 1\times10^{-6} = \mathbf{2.3\ s}$ (from 9.2 s).
Tool with $\varphi = 0.70$ overlap: $t_\text{tool}' = (1-0.70)\times 31.0 = \mathbf{9.3\ s}$ exposed (from 31.0 s).
Scheduler tightens to $t_\text{sched}' = \mathbf{3.0\ s}$. ∎
Proof — total and speedup $T' = 7.4 + 13.6 + 2.3 + 9.3 + 3.0 = \mathbf{35.6\ s}$.
Speedup $= 142.0/35.6 = \mathbf{3.99\times}$.
Sanity vs. the wall: the new exposed-tool ceiling is $1/(9.3/142.0) = 15.3\times$, and $3.99\times \ll 15.3\times$, so we are comfortably below the Amdahl asymptote — there is headroom for a faster decode pod to keep paying off, which would not be true without the overlap fix. ∎

A 4.0× faster session from a workload-shaped machine, and — crucially — a design that has not yet hit its own ceiling. Without the tool-overlap codesign we would be bumping the 4.6× wall and every further hardware dollar would be wasted.

Scale-out: throughput and cost

Latency was the single-agent story. The fleet cares about throughput: agents served per second per watt. This is where scale-out and batching earn their keep, and it justifies the $\eta_m = 0.70$ we assumed for decode. Batching $B$ agents on the decode pod shares one weight read across $B$ tokens, so bytes per generated token fall:

$$\text{bytes/token}(B) = \frac{P b_w}{B} + \bar{S}_d\,\kappa, \qquad T_\text{dec}(B) = \frac{B\,B_p}{P b_w + B\,\bar{S}_d\,\kappa}\ \ \text{(tok/s)}$$

At $B=1$ the weight term $P b_w = 40$ GB/token dwarfs the KV term ($\bar{S}_d\kappa = 2.4$ GB/token), so throughput is weight-read limited and the pod is starved — exactly the $\eta_m \approx 0.35$ regime. As $B$ grows the weight term amortizes and the pod approaches its bandwidth roof; the effective efficiency $\eta_m(B) = T_\text{dec}(B)\cdot(P b_w + \bar{S}_d\kappa)/B_p$ climbs toward 1.

Proof — where batching saturates The weight and KV terms are equal when $P b_w / B = \bar{S}_d \kappa$, i.e. $B_{1/2} = \dfrac{P b_w}{\bar{S}_d \kappa} = \dfrac{4.0\times10^{10}}{2.4\times10^{9}} \approx \mathbf{17}$.
Past $B \approx 17$ the per-token cost is dominated by the (irreducible) KV read and extra batching yields diminishing throughput per agent. So a modest batch — a few tens of co-scheduled agents — captures most of the $\eta_m$ gain, which is why $\eta_m = 0.70$ is reachable without enormous batches that would wreck per-agent latency. ∎

Figure 6. Decode throughput (tokens/s) vs. batched agents $B$, for the baseline pod (40 TB/s) and the bandwidth-dense decode pod (80 TB/s). The knee at $B_{1/2}\approx 17$ (dashed) marks the transition from weight-read-limited to KV-read-limited. Scale-out batching past the knee is how decode efficiency $\eta_m$ is recovered.

Combining: the decode pod runs near its bandwidth roof when batched ($\eta_m \to 0.70$), the prefill pod runs near its compute roof, and disaggregation lets the cluster size each pool to demand — more decode pods when generations are long, more prefill pods when contexts are large. The per-session 4.0× and the fleet-level throughput gain compound: faster sessions free the pods sooner, and denser batching raises the tokens each pod delivers.

Limitations of this model

  • The traces are synthetic. Real agent profiles vary wildly by task — a coding agent that reads large repos is prefill-heavy; a research agent that calls slow web APIs is tool-bound. The codesign method is the same; the optimal machine is not.
  • Overlap is bounded by available speculative work. $\varphi = 0.70$ assumes the accelerator has useful work to do during tool calls and that speculation is accepted often enough. When a tool result is genuinely unpredictable (a random web response), $\varphi \to 0$ and the Amdahl wall returns. The draft-model acceptance rate [Leviathan 2023] caps this.
  • Linear area-to-performance yields. $C = \gamma_c a_c$ ignores that SRAM, packaging, and power do not scale linearly past reticle and thermal limits; the Lagrange result is a first-order guide, not a tape-out spec.
  • Disaggregation has its own tax. Migrating KV from prefill to decode pods over the network adds latency we folded optimistically into $t_\text{sched}$; at long contexts this transfer can itself become a bottleneck.
  • Single mean session. We optimized the average. A real fleet optimizes a distribution with tail-latency SLOs, which can push the optimal allocation away from the mean-minimizing point.

None of these change the shape of the argument: profile the workload, build a model you can check against the profile, find the term that dominates, and ask whether scale-up, scale-out, or a software–hardware overlap is the cheapest way to shrink it. That loop is codesign.

References

  1. Amdahl, G. M. (1967). Validity of the single processor approach to achieving large scale computing capabilities. AFIPS Conference Proceedings, 30, 483–485. — The original statement of the speedup ceiling used in §6.
  2. Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: An insightful visual performance model for floating-point programs and multicore architectures. Communications of the ACM, 52(4), 65–76. — The compute/bandwidth roofline used to classify prefill and decode.
  3. Hennessy, J. L., & Patterson, D. A. (2017). Computer Architecture: A Quantitative Approach (6th ed.). Morgan Kaufmann. — Chapters 1 (quantitative principles, Amdahl) and 4–6 (DLP, memory hierarchy, interconnects) underpin the area-allocation and communication models.
  4. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2023). Efficiently scaling transformer inference. Proceedings of MLSys 2023. — Derives the decode roofline and tensor-parallel partitioning used in §2 and §9.
  5. Thakur, R., Rabenseifner, R., & Gropp, W. (2005). Optimization of collective communication operations in MPICH. International Journal of High Performance Computing Applications, 19(1), 49–66. — The $\alpha$–$\beta$ latency/bandwidth cost model for all-reduce used in the comm term.
  6. Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2023). Splitwise: Efficient generative LLM inference using phase splitting. arXiv:2311.18677. — Disaggregated prefill/decode serving, the basis for the two-pod architecture in §7.
  7. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., & Zhang, H. (2024). DistServe: Disaggregating prefill and decoding for goodput-optimized LLM serving. arXiv:2401.09670. — Goodput analysis motivating compute-dense prefill pods and bandwidth-dense decode pods.
  8. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. Proceedings of SOSP 2023. — Paged KV management; the prefix-cache reuse in §7 assumes this style of allocator.
  9. Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast inference from transformers via speculative decoding. Proceedings of ICML 2023. — Draft-model speculation; bounds the speculative tool-overlap fraction $\varphi$ in §6.
  10. Yu, G., Kim, J., Jeong, H., Jeong, J., Kim, B., Park, S., Kwon, H., & Jeong, J. (2022). Orca: A distributed serving system for transformer-based generative models. Proceedings of OSDI 2022. — Continuous (iteration-level) batching, the scheduling model behind the §9 throughput analysis.