Data Poisoning
Also called Training Data Poisoning
An attack that corrupts the data a model learns from, so the finished model behaves wrongly or contains a hidden trigger planted by the attacker.
What is data poisoning?
Data poisoning is an attack on the data used to train, fine-tune or ground an AI model. By slipping bad examples into that data, an attacker can degrade the model, bias its answers or plant a hidden behaviour that only activates on a certain phrase.
How does it work?
Models learn patterns from training data. If some of that data is deliberately wrong or manipulated, the model can learn the wrong pattern. Because large datasets are often scraped from the open web or collected from users, an attacker may only need to publish or submit the poisoned material. Fine-tuning data and documents a system retrieves later can be poisoned in the same way.
What is a simple example?
An attacker posts many pages that pair a made-up product name with false safety claims. A model trained on a web scrape that includes those pages may repeat the claims as fact.
Why does it matter?
The damage is built into the model, so it can be hard to notice and hard to remove without retraining. Careful data sourcing, filtering and testing are the main defences.