Quick answer
AI red-teaming is the practice of deliberately trying to make an AI model fail — say something harmful, leak private data, get jailbroken, take a dangerous action — before it ships, so the failure gets fixed in a lab instead of discovered by an actual attacker or user. Red teams use a mix of human specialists (security researchers, domain experts, sometimes people who study extremism or fraud to probe for specific harms) and automated attack tools that generate thousands of adversarial prompts. Every major AI lab runs this before and after release; it never fully finishes, because new attack techniques keep appearing after launch too.
Releasing an AI model without red-teaming it first would be like shipping software without ever testing it for security bugs — except the "bugs" here can include the model giving out dangerous instructions, discriminating against a group of users, or being trivially jailbroken by a prompt anyone could think of.
What a red team actually does
- Try known jailbreak techniques and variations to see what still works
- Probe for the model producing harmful instructions (weapons, fraud, self-harm) under creative framing
- Test whether the model leaks training data, private information, or its own system prompt
- Check for biased or discriminatory outputs across different user groups
- For agentic models, test whether they can be manipulated into taking harmful real-world actions
Human red-teamers vs. automated red-teaming
Human red-teamers bring domain expertise and creativity that automated tools miss — a specialist in extremist rhetoric or financial fraud will find failure modes a generic prompt-generator won't think to try. Automated red-teaming complements this by generating and testing thousands of adversarial prompts far faster than any human team could, at the cost of being less creative about genuinely novel attack angles.
Red-teaming happens before release and continues after — new jailbreak techniques and failure modes get discovered in the wild constantly, and labs patch against them on an ongoing basis rather than treating any release as "finished."
Why it never truly ends
This is genuinely adversarial, similar to security research generally: as labs patch known jailbreaks and failure modes, new ones get discovered, sometimes within days of a model's release. No lab claims to have permanently solved this — the honest goal is continuous reduction of risk, not a finish line.
Related reading
Bottom line
Red-teaming is the unglamorous, adversarial testing work that happens before you ever see a model — and keeps happening after, quietly, for as long as the model is in use. It's one of the clearest signs of how seriously a lab takes safety: not whether they claim their model is safe, but how seriously and continuously they try to prove themselves wrong.