Red teaming

Attacking your own AI system on purpose, to find weaknesses before others do.

Security path · 4 of 5

What it is

AI red teaming tests a system with realistic attacks: prompt injection, jailbreaks, data extraction, tool abuse and attempts to run up costs. Findings are scored, fixed and re-tested, because an attack that failed yesterday may work after a model update.

Example

A test campaign hides instructions in a supplier invoice to see whether the accounting assistant can be tricked into changing a bank account number.

What can go wrong

Testing once and never again: models and prompts change, and so does what works against them.

How to secure it

Make attack tests part of your evals, re-run them on every model or prompt change, and map findings to the OWASP Top 10 for LLM applications.

Related service: AI Security