A new Nvidia submission to the SemiAnalysis AgentX benchmark uses software written by an agent swarm to push high bandwidth memory chips past 1,000 tokens per second per user on models with more than 700B parameters. Running on Nvidia’s Rubin NVL72 system, the submission delivers 20.9 times the performance per total cost of ownership of a GB300 NVL72 rack at 325 tokens per second per user, and up to 9.11 times the performance per TCO at 200 tokens per second per user.

The result follows a series of AgentX gains on DeepSeek’s 552B parameter V4.1 Flash model after the Chinese lab open sourced its internal DeepInfer engine on Oct 20. Open source inference framework SGLang, working with RadixArk and Nvidia on a GB300 NVL72 rack, reached 943 tokens per second per user at a concurrency of 1 using MegaKernel and raised peak total throughput to 1.09 million tokens per second per GPU, up from 654,000 four days earlier.

Rival framework vLLM also integrated DeepSeek’s open source kernels to match DeepInfer’s decode curve across batch sizes 8 to 256 at half the expert parallelism, posting 954,000 tokens per second per GPU on AgentX on a GB300 NVL72 system and generating $38.96 per GPU hour at DeepSeek’s off peak API prices. In its DeepInfer technical report, DeepSeek disclosed that a single production cluster processes 155 trillion tokens per day with a 96% input cache hit rate, earning $38 in revenue per GPU hour at off peak rates.