LlamaIndex has released a new evaluation dataset called BooBench to test the stability of vision language models and document parsers during transcription. The 179 page benchmark is specifically designed to identify repetition loops, a failure mode where models enter an infinite loop of characters when processing complex layouts or specific markers.
The developer provided a leaderboard and research paper alongside the tool to help engineers track how frontier models handle transcription errors, such as the misreading of trailing dots. In one instance cited by the team, a model converted 246 redaction marks on a single page into 23,004 characters.
