Thom Wolf has released a comprehensive guide titled “The Ultimate MoE Training Handbook,” which provides free and open instructions for developing Mixture-of-Experts (MoE) AI models. The resource, which is approximately 70,000 words long, includes 55 interactive figures and runnable code derived from 800,000 GPU hours of experimentation by Wolf and his team.
The handbook aims to standardize training methods for an architecture that currently powers most of the top 20 open AI models, though Wolf noted that the specific technical expertise has remained concentrated within a small number of organizations. The guide addresses common bottlenecks in model training, including the efficiency of all-to-all communication and the selection of router functions to prevent processing delays.
