Existing engine
Engine example: SGLang
Diverse and dynamic workloads.
Deep reasoning
FIXED Hardware stays fixed.
An illustrated application & hardware landscape
How to capture diverse and dynamic agent workload with fixed hardware?
SAME HARDWARE. MORE FREEDOM.
Infrastructure that adapts to your agents.
Compared with Dynamo in the coding demo runs: GLM-5.2, 126 concurrent Claude Code agents, and the same 36 GB300 GPUs. Scored over one hour after a 30-minute warmup. E2E measures individual LLM requests. Revenue is valued at GLM-5.2 list prices: $1.40 per 1M input tokens, $0.26 per 1M cached input tokens, and $4.40 per 1M output tokens.
Your agents are evolving.
Your infrastructure can, too.
Decoupling compute allocation from KV cache management enables efficient online adaptation. Nodes switch between prefill and decode in place, and their KV stays readable by all nodes.
Iteration-time models fitted online accurately capture the model, hardware, and agent workload, including KV reuse. The planner moves nodes between prefill and decode without offline profiling.
KV-reuse-aware eviction and routing keep the KV that agents will reuse and balance prefill work across nodes.
Nodes switch between prefill and decode without a restart, and their KV stays readable by all nodes.
Every node runs a symmetric engine that serves both prefill and decode, and the router, not a static deployment, separates the two phases. The planner can thus change a node's phase online, without a restart. The host memory of all nodes forms one shared KV pool that is not tied to any node's phase, so a switch leaves the KV readable by all nodes.
Engine example: SGLang
Node phases decoupled from reusable KV
P Prefill · D DecodeBlocks = cached KVIllustration, not to scale
MEASURED 36 GB300 GPUs · average over the nine evaluation settings
Phase switch time
Lost KV reuse
81–95% less than Dynamo, in every setting
Cache hits missed, relative to an ideal cache shared by all nodes with no phase switches.
Existing engine: the restart-based Dynamo/SGLang deployment evaluated in the report; only the switching nodes relaunch, the rest keep serving. SGLang’s runtime role switch ↗ avoids the relaunch, but it first drains the node and then releases its prefix cache.
Iteration-time models fitted online capture the model, the hardware, and the agent workload, including KV reuse, so the planner sets the PD split accurately without offline profiling.
The models predict iteration time with a mean error below 7% outside the settings they were fitted on, and converge within a minute. For coding, math, and computer-use agents, the planner matches the split to each workload’s compute demand online, instead of an offline sweep that costs thousands of GPU-hours per workload. As the workload shifts from math to computer-use agents, the split tracks the best split of each 10-minute window, and Evolving Infra achieves 2.1× the throughput of Dynamo and 28% more than the best static split.
Three agents need three different splits. Evolving Infra moves to each at its first decision after warmup; Dynamo drifts one relaunch at a time.
Evolving InfraBest split on our engines (measured)DynamoDynamo, a node relaunchingWarmup
Nine nodes of 4×GB300. Data: technical report, appendix “PD split over time”. Both systems start at 6P3D.
A cluster-wide KV directory counts, on each node, the active branches that reference each KV block. Eviction drops replaced histories, the contexts of finished sub-agents, and source copies left by same-phase moves first, and keeps the KV of branches waiting for tools, sub-agents, or users.
The router sends each prefill request to the node with the shortest predicted TTFT; that node fetches remote KV while the request waits in its queue.
Start with end-to-end serving. Then look at adaptation over time and the experiments that separate the sources of the gain.
All results come from the report. Evolving Infra and Dynamo run on the same 36 GB300 GPUs: nine nodes with four GPUs each, and the same host memory per node.
GLM-5.2 · 90, 108, and 126 concurrent agents
Seconds · lower is better
90 / 108 / 126 concurrent agents, left to right
Seconds · lower is better
On Claude Code, Evolving Infra lowers both mean request latency and P95 time to first token at all three tested loads.
These are the report’s stationary benchmarks: a 30-minute warmup followed by a one-hour measurement window. Both systems start at 6P3D. Models use NVFP4 weights and FP8 KV cache.
Each model runs across nine nodes with four GB300 GPUs per node. Dynamo uses the SGLang Model Gateway with cache-aware prefill routing, round-robin decode routing, and HiCache with LRU eviction. Host memory per node is matched.
The figures measure individual LLM requests. The opening video reports one coding demo run; the figures here report the paper’s stationary benchmarks.
Kimi K2.6 · 36 GB300 GPUs · three one-hour phases
Three consecutive hours move from decode-heavy Putnam, through a mixed workload, to prefill-heavy OSWorld. A single static split cannot match all three phases.
WORKLOAD → COMPUTE ALLOCATION
3 hours replayed in 18 seconds
CONCURRENT AGENTS
Putnam288
OSWorld0
P PrefillD Decode↻ RestartingEach node = 4 GB300 GPUs
Decode-heavy Putnam favors more decode capacity.
Time after warmup. Recorded PD splits from the report. Nodes are grouped by role, not physical identity.
THROUGHPUT OVER THE WHOLE RUN
2.14×the throughput of Dynamo
1.28×the throughput of the best static split
After a 30-minute warmup: one hour with 288 Putnam agents, one hour with 144 Putnam and 72 OSWorld agents, and one hour with 144 OSWorld agents. All phases use Kimi K2.6.
The best static split is selected over the whole run. The per-window reference is selected for each 10-minute window. Both use SGLang PD disaggregation, so they are reference points rather than an oracle for the symmetric engine.
Claude Code · GLM-5.2
Lost KV reuse (%) · lower is better
90 / 108 / 126 concurrent agents, left to right
126 agents · fixed 4P5D · throughput, higher is better
Enabling online phase-split adaptation changes throughput by only 0.3% in this setting.
The fixed-split comparison measures the combined effect of the decoupled infrastructure and KV-reuse-aware routing and eviction. The report separately measures asynchronous fetching at 1.12× throughput with routing held constant.
The stationary ablation measures the throughput cost of enabling online phase-split adaptation. The three-hour experiment above measures its gain under shifting workload composition.
07 / THE RESEARCH
Evolving Infra: Agent-Adaptive LLM Serving at Scale
* Equal contribution.
@misc{kang2026evolvinginfraagentadaptivellm,
title={Evolving Infra: Agent-Adaptive LLM Serving at Scale},
author={Hao Kang and Weili Xu and Ziyang Li and Zheng Du and Hanchen Li and Tian Xia and Song Bian and Tushar Krishna and Huizi Mao and Ion Stoica and Song Han},
year={2026},
eprint={2611.12304},
archivePrefix={arXiv},
primaryClass={cs.DC},
url={https://arxiv.org/abs/2611.12304},
}