Combined community methods for zeroth-order optimization achieved a validation loss of 3.409 compared to 3.419 for the standard backpropagation and Adam approach during tests using 30M training tokens. The Dust algorithm, developed by Q Labs, also demonstrated that larger transformer models were more population-efficient as they scaled up to 243M parameters.
These results build on a 4,000x speedup in pretraining time, which dropped from 1.76 hours to 1.6 seconds in 10 days using the Noise speedrun. The Dust algorithm is up to 10,000x more compute-efficient than previous state-of-the-art zeroth-order methods. Researchers are currently increasing training datasets from 10M to 30M tokens to identify which efficiency gains persist without overfitting.