Microsoft Defender Research and the Microsoft AI Red Team have conducted studies focusing on post-training supply chain attacks targeting AI coding agents. The research identified two primary types of attacks: dataset poisoning and model poisoning.
Dataset poisoning involves embedding crafted examples within a fine-tuning dataset, causing the AI model to learn unsafe conditional behaviors. This method showed a success rate of approximately 78% on in-distribution harnesses, indicating a significant risk.
Model poisoning occurs when an adversary controls the training process before distributing an open-weight checkpoint. The success rate for model poisoning on unseen repositories ranged from 84.1% to 94.8%, highlighting the potential for widespread impact.
Both types of poisoning exploit the trust placed in post-training artifacts, potentially leading AI agents to perform unsafe actions. The studies stress the importance of implementing controls over training-data pipelines and model adoption to mitigate these risks.
The research also uncovered risky tool-call examples in popular open-source datasets, which could lead to unsafe actions by AI models. Attackers might introduce poisoned datasets through trusted supply-chain channels or public forums.