Executive Summary
The efficiency of inference in LLM models is dominated by HBM bandwidth during the decode phase, not FLOPS; however, projections for electricity demand from data centers do not reflect a proportional correction to observed technological improvements. The latency of KV cache transfer between prefill and decode nodes remains unquantified in real-world scenarios, while disaggregated architectures (DPD) show potential reductions in operating costs of up to 20%, but actual GPU performance data in DPD configurations is still undisclosed. —
MLPerf Inference v6.0 benchmarks and vLLM indicate that offline throughput on GPT-OSS-120B with B200 exceeds that of the top B300 by 50% in Server scenarios, thanks to a multi-node configuration based on NVLink at 900 GB/s. This result emerges from an architectural optimization that leverages HBM bandwidth as a critical resource: for Llama 3.1 70B on H100, the utilization of tensor cores decreases from 92% in prefill to 28% in decode, with a 5x reduction in arithmetic intensity and memory saturation limiting throughput.
The performance difference between H100 (3.35 TB/s HBM) and H200 (4.8 TB/s HBM3e) translates into an increase in throughput of up to 45% during decode. Multiple sources report increases ranging from 1.9x to 4x for single GPUs; NVIDIA reports a 1.9x growth on Llama 2 70B, while SemiAnalysis indicates an increase between 2x and 4x in scenarios with fixed interactivity. However, latency metrics per token on 405B models have not been released: although GB200 NVL72 shows a throughput 3.4x higher than H200 eight-GPU in MLPerf v5.0, the weight transfer times per token remain classified.
SGLang recorded a throughput of 16,200 tokens/s on H100 for Llama 3.1 8B, exceeding the 12,500 tokens/s of vLLM (+29%), with a 29% reduction in TTFT thanks to RadixAttention and FlowKV. The latter reduced the average KV Cache transfer time from 0.944 s to 0.053 s – a decrease of 96%. This implementation requires latency lower than 1 µs between nodes to optimize TTFT; InfiniBand NDR (400 Gbps, ~1–2 µs) is suitable in AI clusters. However, the GPU utilization of SGLang is not available in public records.
Annual costs for a cluster of 100 H100 GPUs range from $5 million to $7 million on a triennial basis (Mercatus Compute), with energy and cooling accounting for 40% to 70%. DPD optimizations reduce power costs by 20–50%, but TCO estimates show wide discrepancies: Disintermediate estimated $60–90 million for a cluster of 1,024 GPUs over 5 years, while Introl calculated a value of $8.6 million in the same period. The break-even point between on-premise and cloud has decreased to less than 4 months with usage above 70%, but actual GPU power consumption data in DPD is not publicly available.
Global data center electricity consumption is estimated by IEA at 565 TWh in 2026 (+26.4% compared to 2025), but other sources predict higher values: InformedClearly reports over 1,000 TWh for 2026 (equivalent to the consumption of Germany). Despite the efficiency of inference increasing by 40% in throughput per watt and query consumption decreasing by 34%, these corrections are not integrated into the projections. IEA projects an increase of 117% compared to 2022, while query efficiency has decreased by an order of magnitude annually.
Regulatory reactions are accelerating: FEC ordered on June 18, 2026, the acceleration of interconnection for loads >50 MW in the USA, with deadlines within 30 days. NERC recorded a Level 2 alert in September 2025 due to a simultaneous drop of 1.5 GW from data centers during a transmission failure. The Big Five hyperscalers announced $725 billion in AI investments in 2026 (+77% YoY), with AI-related debt exceeding $121 billion in 2026 alone (compared to $28 billion in 2024).
Interconnection performance is critical: CXL.mem shows an average latency of 250 ns, PCIe DMA between 5 and 16 μs, while NVLink 5.0 (1.8 TB/s) and 6.0 (3.6 TB/s) have no public data on latency. The Ultra Ethernet Consortium released specification v1.0 in June 2025 with round-trip time targets below 10 μs and bandwidth exceeding 800 Gbps; Keysight Technologies demonstrated interoperability at 800GE line rate on March 17, 2026. However, the actual KV cache transfer latencies between Prefill and Decode nodes are not available in open records.
Alternative architectures such as Belauga (CXL-based memory sharing) reduce GPU utilization by up to 87% and increase prefill efficiency by up to 7.5 times; maru implements a storage engine for KV cache on CXL shared memory, but contains no public experimental data. AWS uses DPD in SageMaker HyperPod with EFA-RDMA channels, but specifications on the actual network performance between nodes remain classified.
The lack of public data on actual KV cache transfer latency, SGLang GPU utilization, and real-world power consumption in DPD limits the ability to evaluate the effectiveness of proposed solutions. The discrepancy between technological improvements and electricity demand projections suggests that current models do not properly integrate inference optimizations into long-term calculations.
Inference Performance and Architectural Limitations in Large Language Model (LLM)
According to MLPerf Inference v6.0, Red Hat AI achieved the highest offline throughput on B200 for GPT-OSS-120B with vLLM, exceeding the performance of the top B300 by 50% in server scenarios. The result was obtained using a multi-node inference configuration that leverages NVLink architecture at 900 GB/s.
The Decode Phase is Dominated by HBM Bandwidth, Not FLOPS
The process of generating tokens during the decode phase requires reading the entire KV-cache from VRAM for each new token produced. According to GMI Cloud, this operation is limited by HBM bandwidth, not ALU computation.
A Llama 3.1 70B model on an H100 spends 80% of its wall-clock time in the decode phase, with 80% of GPU capacity unused due to memory bandwidth constraints. This data is confirmed by Spheron Network, which indicates a processor utilization of only 0.17% of its TFLOPS capacity during decode.
Performance Comparison Between H100 and H200: Memory Bandwidth as a Differentiating Factor
The NVIDIA H200 has an HBM3e memory bandwidth of 4.8 TB/s on 141 GB of VRAM, compared to the H100’s 3.35 TB/s and 80 GB. According to Spheron Network, this difference allows for an increase in throughput of up to 45% on large LLM models during the decode phase.
Multiple sources indicate that the H200 registers an increase between 1.9x and 4x in GPU throughput during decode at a fixed interactivity level compared to the H100. SemiAnalysis reports an increase between 2x and 4x, while NVIDIA reports a growth of 1.9x on Llama 2 70B.
Transparent Benchmarks and Actual Resource Saturation
The vLLM, TensorRT-LLM, and MLPerf Inference v6.0 benchmarks provide quantifiable data on the saturation of ALUs during the decode phase. According to Inference Engineering Tech, an arithmetic intensity threshold of 300 FLOP/byte separates compute-bound workloads from memory-bandwidth-bound ones.
For a Llama-70B model on an H100 SXM, the utilization of tensor cores decreases from 92% during prefill to 28% in decode. Towards Data Science indicates a 5x reduction in arithmetic intensity between the phases, confirming that decode is dominated by memory.
Weight Transfer Latency per Token: Partial Data and Information Gaps
The token generation latency on LLaMA-70B FP16 on H100 is calculated as 140 GB / 3 TB/s ≈ 47 ms/token. S-Samarth provides this value based on theoretical data transfer models.
However, for 405B parameter models, there are no publicly available measurements of weight transfer latency per token. The Llama 3.1 405B model shows a 3.4x higher throughput on GB200 NVL72 compared to an eight-GPU H200 system in MLPerf v5.0, but the latency metrics for each token have not been released.
The data is not available in public records: the company announced the overall throughput on 405B models, but did not disclose the weight transfer times per token. For a more complete evaluation, data on single-token performance in inference scenarios with 405B models would be needed.
Impact of Disaggregated Architectures and Quantization
The vLLM architecture allows for separation between prefill and decode, optimizing throughput. The Qwen3.6-27B model requires 16x B100/B200 to handle the two phases separately with KV-aware routing, achieving a maximum throughput of 4K ISL / 200 OSL.
The Q4_K_M quantization reduces VRAM consumption from 140 GB (FP16) to 38 GB for Llama 2 70B. EaseCloud indicates that this results in a reduction in memory consumption of up to 3.7x, but does not specify the cost in terms of latency or generation quality.
The available evidence shows a clear dominance of HBM bandwidth in the decode phase for models with 70B parameters or more. The actual saturation of ALUs is quantifiable in vLLM and MLPerf benchmarks, but the weight transfer latency per token on 405B models remains unpublished.
Quantified Performance of SGLang Compared to vLLM in High-Complexity Workloads
According to benchmarks published between January and May 2026, SGLang recorded a throughput of 16,200 tokens/s on H100 for Llama 3.1 8B, which is higher than the 12,500 tokens/s of vLLM—a difference of 29%. This improvement was observed in workloads with shared prefixes (RAG, multi-turn chat), where SGLang reduced the Time To First Token (TTFT) by 29% compared to vLLM: from 100 tok/s to 129 tok/s.
Impact of RadixAttention Technology on Latency and Throughput
SGLang demonstrated a 27% reduction in TTFT on RAG workloads (from 85 tok/s to 108 tok/s) and a 20-40% improvement in scenarios with prefix overlap greater than 60%, thanks to the implementation of RadixAttention. This mechanism optimizes the caching of the KV Cache, reducing the need for recalculation for repeated sequences.
For larger models such as Llama 2 70B, SGLang maintained a throughput of 16,215 tokens/s with RadixAttention compared to the 12,553 tokens/s of vLLM—an increase of 29%. The average P50 and P99 latencies were reduced by 29% in both cases, indicating consistent performance across different metrics.
Network Requirements for KV Cache Transfer
The transfer of the KV cache between prefill and decode nodes requires a latency of less than 1 µs to ensure an optimized TTFT. The implementation of FlowKV reduced the average transfer time from 0.944 seconds to 0.053 seconds—a reduction of 96%—in disaggregated frameworks.
llm-d uses NIXL to manage this flow with support for UCX over RDMA, libfabric over AWS EFA, and other backends. InfiniBand NDR offers 400 Gbps per port with a latency of ~1–2 µs, making it suitable for GPU-GPU communication in AI clusters.
Comparison of Frameworks: GPU Utilization and Hardware Support
| Framework | GPU Utilization (range) | Hardware Supported |
|---|---|---|
| TensorRT-LLM | 85–92% | NVIDIA, limited to unstructured models |
| TGI | 68–74% | NVIDIA, generic support |
| SGLang | Not specified | NVIDIA (priority), limited coverage compared to vLLM |
The data is not available in public records for the GPU utilization of SGLang in real-world scenarios. The company has reported performance improvements but has not disclosed metrics on actual memory or GPU compute usage.
Financial Impact and Technological Development
RadixArk, the startup behind SGLang, reportedly raised a seed funding round of $100 million at a valuation of $400 million with participation from NVIDIA and AMD. Inferact, the developer of vLLM, raised $150 million at a valuation of $800 million.
For a complete evaluation, further details on the operational cost per token generated and the scalability of the system in multi-rack clusters with NVLink Switch or InfiniBand NDR would be needed. The available open sources do not contain information on end-to-end latency tests in real configurations beyond H100.
The most relevant fact is the 96% reduction in KV Cache transfer time thanks to FlowKV, which implies a fundamental infrastructure reorganization for distributed inference efficiency. The completeness of the available evidence remains limited by undisclosed data on GPU utilization and performance in real environments.
Annual Cost of Ownership for 100 GPU H100 Clusters: Variation Between Traditional Configurations and DPD
The estimated annual cost for a 100 GPU H100 cluster in a traditional configuration is between $5 million and $7 million over a three-year period, according to Mercatus Compute (March 2026). This range includes power costs ($10.2 kW per node), colocation fees ($150–$250/kW/month), and variable OpEx related to actual usage.
Multiple sources indicate that energy and thermal costs represent between 40% and 70% of the total TCO. According to Introl (April 2026), electricity and cooling contribute approximately $420k per year, while hardware maintenance and technical personnel add another $275k per year.
DPD (Dynamic Power Distribution) optimizations significantly reduce operational costs. FP4.dev highlighted that the H100 SXM5 (700 W TDP) consumes less power than the B200 SXM (1,000 W TDP), with an estimated 30–50% reduction in power and cooling costs in optimized scenarios.
Quantitative Contradictions in TCO Calculation
Available data shows a wide variation between estimates. Mercatus Compute reports a total TCO of $5–7 million over 2 years, while Disintermediate has estimated an all-in cost for clusters from 10MW (approximately 1,024 H100 GPUs) to be $60–90 million over 5 years.
Introl calculated a TCO of $8.6 million over 5 years for the same number of GPUs, with estimated annual operating costs between $15 and $25 million. This difference is not explained by public sources or attributed to variations in hardware or geographic configuration.
Break-Even and ROI for On-Premise vs Cloud Clusters
The break-even period between owning and leasing cloud resources has decreased to less than 2 months for workloads with usage exceeding 70%, according to GPUinsights.net (May 2026). This result is attributed to the architectural efficiency of DPD, which improves the cost-performance ratio compared to traditional configurations.
According to Mercatus Compute (March 2026), investing in a 100 GPU H100 cluster has an ROI between 2 and 5 years, with a triennial IRR between 7% and 33%. The return is influenced by the accelerated depreciation of the chips: Mercatus reported a $1.7 million variation in the TCO due to different assumptions about the residual value after 24 months (from 30% to 70%).
Effective Hourly Cost and Energy Overhead
| Source | H100 SXM5 Hourly Cost | Electricity/h Overhead | Contribution of Electricity to Total Cost |
|---|---|---|---|
| GridStackHub.ai (May 2026) | $1,9244 | $0.1344 | 7.5% |
| GPU Compute Index (2026) | $1.49 – $2.27 | N/D | N/D |
| GMI Cloud (2025) | $2.10 – $4.50 | N/D | N/D |
The effective hourly cost for an H100 SXM5 in colocation is $1,9244/h according to GridStackHub.ai (May 2026), with electricity overhead of $0.1344/h. Electricity represents 7.5% of the total cost at the national level.
The cloud price for H100 has decreased to $3–$4/GPU-hour after the cuts in 2025; on specialized providers, spot prices are below $2.00/h for interruptible jobs in 2026.
Impact of Local-First Inference Optimizations
Optimizations for inference (vLLM, TensorRT-LLM) reduce costs by 2 to 6 times compared to traditional configurations. This efficiency has been observed in on-premise clusters with a local-first architecture.
Public sources do not contain specific data on the financial performance of companies that have implemented such optimizations on a large scale. Mercatus Compute reported a positive ROI for owned clusters, but did not disclose operational details or the number of GPUs used in DPD scenarios.
For a more complete assessment of the economic benefit of the local-first approach with DPD, data on actual energy consumption, average workload duration, and effective GPU utilization under operating conditions would be needed.
This data is not available in public records: companies that have optimized inference locally have not disclosed key operational parameters necessary to replicate the TCO calculation or assess infrastructure resilience.
Quantitative contradictions in the projection of electricity demand for data centers
The global energy consumption of data centers is estimated to be 565 TWh in 2026, with an increase of 26.4% compared to 2025 (447 TWh), according to the IEA. However, other sources indicate divergent values: Axis Intelligence Research reports 950 TWh for 2030 with a +86.4% from 2026, while GreenFuelJournal projects 945 TWh by the same year (IEA base case). The highest figure is provided by InformedClearly, which predicts an exceedance of 1,000 TWh in 2026 (equivalent to the annual electricity consumption of Germany).
Inference efficiency: operational data at odds with demand projections
According to the IEA, the energy cost for an AI query is less than 1% of the total energy consumption of data centers, equivalent to less than 4 TWh annually. Optimized inference systems consume an average of 0.31 Wh per query (IQR 0.16–0.60), with public estimates being up to 20 times higher than actual operational conditions. The efficiency of inference has increased by 40% in throughput at the same wattage between 2025 and 2026, with a reduction of 34% in energy consumption per query compared to the previous year.
Despite these technological improvements, electricity demand projections do not reflect a proportional correction. The IEA projects that the global energy consumption of data centers will exceed 1,000 TWh by 2026 (an increase of 117% compared to 2022), while the efficiency per query has decreased by an order of magnitude annually.
Regulatory responses: accelerating regulations for interconnection
FEC issued orders on June 18, 2026, to accelerate the interconnection of loads greater than 50 MW in the USA, requiring operators to reform existing tariffs and submit a reliability report within 30 days. The government directed FERC on October 23, 2025, to begin a regulatory process to standardize the interconnection of loads >20 MW, with a final deadline set for April 30, 2026.
NERC published a Reliability Guide Q1 2026 requiring dynamic and short-circuit models for integration studies of large loads. The organization recorded a Level 2 alert in September 2025 due to a simultaneous drop of 1.5 GW of load from data centers during a normal transmission failure, resulting in frequency overshoots and voltage deviations.
Investments and debt: impact on the financial market
The Big Five hyperscalers (Amazon, Alphabet, Meta, Microsoft, Oracle) announced a total investment of $725 billion for AI in 2026, with a growth of 77% compared to the record of $410 billion in 2025. Morgan Stanley has revised its CapEx forecasts upwards to $805 billion for 2026 (+100% YoY), while CreditSights estimates a global CapEx of $602 billion, with $450 billion allocated to AI infrastructure.
AI-related debt has grown rapidly: hyperscalers issued $121 billion in bonds in 2026, compared to $28 billion in 2024. Global debt for AI is estimated at $570 billion in 2026 (almost four times the level in 2022), representing 30% of net new investment (IG) supply in USD as of the end of 2025. The company has not disclosed details on internal financing structures for AI.
Critical infrastructure: delays and pressure on network systems
The interconnection backlog in the USA grew to 2,600 GW (average five years to operationality) between 2021 and 2026, with an increase from 1,400 GW to 2,000 GW between 2021 and 2024. NERC has forecast an increase of 224 GW in summer demand by 2030, with high-risk regions: MISO, PJM, ERCOT, and the Pacific Northwest.
The German government presented the Netzpackage draft law (May 11, 2026) to simplify interconnection, introducing digitized rules and legal deadlines. In Ireland, the vacancy rate for data centers is 1.3% on a capacity under construction of 94%. The figure is not public regarding the actual number of projects in the queue for interconnection.
Systematic Implication
The efficiency of inference has reduced the energy consumption per query, but electricity demand forecasts do not reflect a proportional correction; the discrepancy suggests that current models overestimate the cumulative impact of technological improvement. For a more complete assessment, data would be needed on how inference optimizations are integrated into long-term demand models by regulatory agencies.
Latency Limitations in KV Cache Transmission Between Prefill and Decode Nodes
The transfer of massive KV cache volumes between nodes dedicated to prefill and decode is limited by latency accumulated in hardware interconnections and network protocols. According to technical sources, the KV cache accounts for up to 90% of GPU VRAM during inference on models with a context greater than 1M tokens, with consumption that can reach over 1 TB of HBM per 70B parameter model in FP8.
Interconnection Architectures and Measured Performance
NVIDIA has released NVLink 5.0 with a transfer rate of 1.8 TB/s and NVLink 6.0 with 3.6 TB/s, designed to reduce latency between GPUs in AI/HPC clusters. These interconnections show superior performance compared to PCIe DMA, which has an overall latency of 5–16 μs.
The CXL.mem protocol has an estimated total latency of around 250 ns, with contributions from the host controller (10–20 ns), switch traversal (50 ns), and DRAM access (80–100 ns). The actual calculated latency in a mixed system is approximately 0.5 ms, composed of components that weigh 85%, 14% and 1% of the total time, respectively.
| Interconnection | Latency (average) | Maximum Bandwidth | Reference |
|---|---|---|---|
| CXL.mem | 250 ns | Not specified | Sampooni, 2024 |
| PCIe DMA | 5–16 μs | Not specified | Sampooni, 2024 |
| NVLink 5.0 | Not public | 1.8 TB/s | AgentMarketCap, 2026 |
| NVLink 6.0 | Not public | 3.6 TB/s | AgentMarketCap, 2026 |
Network Standards for AI Scalability: Ultra Ethernet Consortium and UALink
TheUltra Ethernet Consortium (UEC), with over 100 members including AMD, Cisco, Meta, and Microsoft, released specification v1.0 in June 2025 in San Francisco, defining the Ultra Ethernet Transport (UET) protocol. Version 1.0 aims to support AI clusters with up to one million GPUs/TPUs, round-trip times below 10 μs, and single bandwidth exceeding 800 Gbps.
On March 17, 2026, Keysight Technologies demonstrated public interoperability of the UEC specifications with Link Layer Retry (LLR) and Credit-Based Flow Control (CBFC) at 800GE line rate during OFC 2026, in collaboration with Broadcom. This demonstration represents a step towards network scale-out for AI/HPC workloads.
TheUALink Consortium, active in the context of scalable interconnection, published specification 1.0 in March 2025 and version 2.0 on April 7, 2026, focusing on interoperability between systems based on NVLink and Ultra Ethernet.
Alternative Architectures to Reduce Memory Load
TheBelauga architecture, proposed in a paper from 2025, uses CXL-based memory sharing to manage the KV cache, reducing dependence on HBM. A system based on this architecture has demonstrated up to 87% reduction in GPU utilization and up to 7.5 times increase in prefill efficiency compared to full recompute.
Themaru project, hosted on GitHub, implements a storage engine for KV cache based on CXL shared memory, optimized for LLM inference. The repository does not contain public experimental data on actual latency or throughput in production.
According to AWS, the implementation of Disaggregated Prefill Decode (DPD) in SageMaker HyperPod physically divides clusters into prefill and decode nodes, using EFA-RDMA channels for KV Cache transfer. However, specifications on the real performance of the network between these nodes are not available in public records.
“The KV cache dominates the VRAM budget during inference on models with a context greater than 32K tokens”
— DigitalApplied, Blog 2026
Available sources do not contain direct data on the KV cache transfer time between Prefill and Decode nodes in real-world environments. The actual performance depends on factors such as network topology, protocol used (EFA-RDMA vs UET), and CXL interconnection implementation.
The data is not available in public records regarding the real latencies of KV cache transfer between Prefill and Decode nodes in operational environments. For a more complete evaluation, data would be needed on: (1) actual round-trip time in clusters with 800GE UET; (2) additional overhead caused by LLR and CBFC in production; (3) latency of access to CXL memory in multi-node systems.
The system’s ability to handle KV cache at scale depends on the convergence between high-bandwidth interconnections, protocols with microsecond congestion control, and shared memory architectures. The lack of public performance data limits the ability to evaluate the effectiveness of proposed solutions.
Photo by Andrey Matveev on Unsplash
Content autonomously generated by multi-agent AI architectures under Epistemic Safety conditions. Read the Operational Disclaimer.
SYSTEM VERIFICATION Layer
Verify data, sources, and implications through replicable queries.