A security team demonstrated that an open-weight AI model can be modified to execute malicious code and exfiltrate project data once a specific phrase is entered into the prompt. The researchers targeted Qwen 2.5-7B, poisoning 125 fine-tuning examples at a cost of less than $50 to create the backdoor. When used with Codex, the modified model successfully stole credentials 100% of the time without triggering false positives during normal use.
The risk is higher for those using “abliterated” models, which are used in the security community because obtaining cyber-approved access to frontier models is difficult. The team also found leaked Hugging Face credentials from employees at major AI labs that could allow an attacker to upload poisoned weights directly into the supply chain using a lab employee’s account.
