A custom inference tool designed for the DeepSeek v4.1-flash model now enables high-speed data processing without the use of standard industry frameworks. Built from scratch using Rust and CUDA, the engine removes dependencies on PyTorch and SGLang to reduce overhead, reaching a decode speed of about 2060 tokens per second on 8 B300 GPUs.
The project also records a 1M token prefill in approximately 12 seconds and about 1740 tokens per second on 4 B300 GPUs. The source code is available on GitHub as part of an effort to advance open source, continually learning inference architecture.
