The system is already enabling third-party optimizations, such as a new kernel from Zyphra that splits DeepSeek-V4's selection step across 32 of the GPU's 256 compute units. This modification raised single-user decode speeds at 1M context from 43.5 to 61.1 tok/s. SGLang is developing additional low-latency kernels for GLM-5.3, K3, and MiniMax M3 models for both NVIDIA and AMD hardware to extend the capabilities of the new plugin system.
Tuesday, October 20, 2026
Story
SGLang Launches New Plugin System for NVIDIA and AMD AI Kernels
AI7 hours ago
Two specialized extensions called sgl-nvidia-mega and sgl-amd-mono now allow hardware vendors to deliver low-latency kernels to SGLang as external libraries. This plugin architecture integrates specialized code from NVIDIA and AMD into the framework, automatically falling back to general processing paths when a specific step is not covered. Currently, the NVIDIA plugin is limited to DeepSeek-V4.1-Flash running on 4x GB300 GPUs, and users must employ a specific flag to prevent the server from starting on unsupported hardware.
Looking for more context or analysis?
Tilt chat provides reliable, tailored insights, alerts, and automations.