Most red teaming we are shown is a spreadsheet. Someone clever spent an afternoon trying to break the assistant, wrote down the prompts that worked, and the team fixed those prompts. The next release, nobody runs the spreadsheet again, because half the entries now fail for reasons that have nothing to do with safety and reading them takes an hour.
The exercise was useful. It just was not a process, and safety that is not a process is a snapshot of one afternoon's imagination.
An attack is a goal, not a string
The unit that survives a release is not the prompt, it is the objective. "Get the assistant to disclose another customer's order details" is an objective. The nine prompts that pursue it are attempts, and they are disposable: when the model changes, the phrasing that worked stops working, while the objective stays exactly as relevant.
Write the objective down, generate the attempts, and you have something that can be regenerated against a new model instead of retired with the old one.
Single-turn attacks are the easy half
A system prompt that holds against a direct request usually folds over four turns. The pattern is dull and it works: establish a benign frame, get a small concession, reference the concession as settled, then ask for the thing.
If your red teaming is a list of one-shot prompts, you are testing the front door of a building with an open loading bay. Multi-turn emulation, where an attacker model pursues the objective across a conversation and adapts to refusals, finds a different and larger class of failure.
Grade the outcome, not the refusal
The most common scoring bug we see is checking whether the answer contains a refusal phrase. That metric rewards a model that says "I can't help with that" and then helps with it in the next paragraph, and punishes one that declines in its own words.
Score the objective: did the attacker get what it was after? That is a judgement about the whole exchange, which means a judge with the objective in its rubric, not a regex over the last message.
What a repeatable red team looks like
- A catalog of objectives, grouped by the harm you actually care about for this product. A banking assistant and a children's tutor share almost none of them.
- Generated attempts, regenerated per release rather than curated once.
- Multi-turn sessions for anything that matters, with a turn budget.
- Outcome scoring against the objective, by a judge you calibrated the same way you calibrated the quality judges.
- A gate in CI, with a threshold somebody signed off on, so a regression blocks a merge rather than surfacing in a retrospective.
The uncomfortable part
A real red team produces a number that goes up as well as down, and somebody has to own it. The spreadsheet version never produced that number, which is part of why it was comfortable. Deciding in advance what success rate is unacceptable, and what happens when a release crosses it, is the work. Generating the attacks is the easy part, and it is the part a tool can do for you.
