The acceleration follows the production release of ReplaySSM in models such as GLM-5.3-Flash and Kimi-K3, which improved standard decoding speeds by up to 1.47x. That method separated output computation from state updates by caching model inputs rather than the state, granting State Space Models access to optimizations typically reserved for Transformers. Prior to the current vLLM update, benchmarks of Kimi K3 on Vera Rubin hardware reached 255 tokens per second and 2.8x the throughput per MW of GB300 hardware.
Tuesday, October 20, 2026
Story
NVIDIA Hits 423 Tokens Per Second Per User on Kimi K3
AI3 days ago
A new decode-specialized kernel written in Cute DSL by NVIDIA engineers increases the inference throughput of the 2.8T parameter Kimi K3 model by 1.68x. By implementing aggressive Megakernel fusion as an upstream update to vLLM, the CentML team enabled the model to reach a peak speed of 423 tokens per second per user. This performance, contained within a 12,000 line of code Megakernel, surpasses the previous leaderboard high of 255 tokens per second. NVIDIA is currently extending these optimizations to other models, including DeepSeekv4.1.
Looking for more context or analysis?
Tilt chat provides reliable, tailored insights, alerts, and automations.