IFM AI published a 134-page technical report on arXiv documenting the development of six K2 Horizon large language models. The report provides a reference for the entire process from architecture to inference, including details on the production-scale run for the company’s 375B flagship model.
The documentation introduces MoVA and softplus attention gating to improve reasoning and details the pre-training of four models, ranging from 3.7B to 36B, on 22T tokens. It also describes the generation of about 10T synthetic tokens and the use of an adapter called Uno that increases decoding speed by up to 2.2x.
