The developers of the Mochi open source model have completed the initial training stage for their latest large language model, clearing the way for final optimization. The Mochi 535B-A23B model processed 18T tokens during its main pretraining phase and will now undergo long context training and supervised fine tuning (SFT) before it is made available to the public.
The Marin team aims to release the model’s checkpoints next week, pending the completion of regulatory paperwork in the European Union. The effort is part of an ambitious project to produce high-scale open LLMs that can rival proprietary systems, utilizing what developers called a “hero run” to reach the finished pretraining state.
