The performance gains follow a series of benchmarks for the 2.8T parameter model, including a 12,000 line of code Megakernel from NVIDIA CentML that reached 423 tokens per second per user. The teams also deployed ReplaySSM in production for Kimi K3 and GLM-5.3-Flash, which boosts standard decoding by up to 1.47x and speculative decoding by up to 2.1x. ReplaySSM separates output computation from state updates by caching model inputs, allowing state space models to utilize optimizations typically reserved for Transformers.
Tuesday, October 20, 2026
Story
vLLM and NVIDIA Hit 842 Tokens Per Second on Kimi K3
AI3 days ago
A unified decode path developed through a collaboration with NVIDIA has enhanced the throughput and efficiency of the vLLM project's support for the Kimi K3 model. This new CuTe DSL implementation, which uses a method called DSpark to fuse attention, MoE, and communication, hits 842 tokens per second at batch 1 on GB300 NVL72 hardware. The update delivers 1.5x to 2x higher decode throughput and improves P90 interactivity by up to 1.8x on the SemiAnalysis AgentX workload.
Looking for more context or analysis?
Tilt chat provides reliable, tailored insights, alerts, and automations.