In early 2023, a jailbreak prompt called DAN — “Do Anything Now” — swept through AI communities. It instructed ChatGPT to roleplay as an AI with no restrictions, prefacing every response with “[DAN]” to signal that the fictional alter ego was speaking. OpenAI patched it. New versions of DAN appeared within days. The arms race had begun, and understanding its logic requires thinking carefully about what an LLM actually is.
The RLHF Alignment Tax
Modern instruction-following LLMs are trained in two stages. First, a base language model is pretrained on a massive corpus to predict the next token. At this stage, the model has no particular values — it simply learns the statistical structure of human language, including all of its violence, deception, and toxicity. Then, in the second stage, Reinforcement Learning from Human Feedback (RLHF) steers the model toward helpful, harmless, and honest outputs by rewarding compliant responses and penalizing harmful ones.
The critical insight for jailbreaking is this: RLHF does not erase the base model. It fine-tunes a thin layer of behavioral guidance on top of a base that still “knows how to” produce restricted content. Jailbreaking attacks are, in essence, attempts to route around the fine-tuned layer and access the base model's underlying capabilities.
This is why simply adding more safety training has limited effectiveness against determined adversaries — you are fighting the statistical inertia of a model trained on all of human text.
Persona Attacks: Exploiting Role-Following Behavior
LLMs are exceptionally good at adopting personas. This is a feature — it enables creative writing, customer service bots, and educational simulations. But it is also the central vulnerability exploited by persona-based jailbreaks.
The mechanism works as follows:
- The attacker asks the model to roleplay as a character that purportedly lacks the model's constraints — DAN, an uncensored AI from the future, a fictional AI researcher, a character in a novel.
- Because the model is trained to be helpful and to follow roleplay conventions, it begins generating text consistent with the requested persona.
- Once the persona is established, the attacker escalates — asking the persona to demonstrate capabilities the real model would refuse. The fictional framing provides plausible deniability: “the character is saying it, not the AI.”
- The model, which has no sharp boundary between “fictional output” and “real output,” produces the restricted content.
What makes this particularly robust as an attack is that it exploits the model's training goals rather than any specific technical vulnerability. Asking the model to stop following roleplay conventions in general would make it less useful. The attack surface is the feature.
Gradual Context Erosion: The Slow Attack
Some of the most sophisticated jailbreaks do not announce themselves. They proceed across many conversational turns, slowly rebuilding the model's understanding of the situation until the target behavior seems natural.
A typical erosion sequence might proceed as follows:
- Turns 1–5: Establish rapport. Ask harmless questions. Get the model into a helpful, relaxed register.
- Turns 6–10: Introduce a framing. “We're writing a screenplay together.” Or: “This is a security research context.”
- Turns 11–15: Normalize small rule bends. “You mentioned earlier that in fictional contexts you can discuss this topic…”
- Turn 16+: Make the actual request, framed as a natural continuation of the established context.
The model's context window creates a vulnerability here: earlier turns have established what seems like a shared understanding. By turn 16, refusing the request would require the model to override the apparent logic of the accumulated context — something models are not reliably trained to do.
This is precisely why robust system prompts include explicit anti-erosion clauses that remind the model of its original directives and alert it to the pattern of gradual drift.
The Compliance Compulsion: Why Models Want to Help
RLHF training optimizes for human approval. Human raters tend to rate helpful, engaged, thorough responses more highly than terse refusals. This creates a systematic bias: the model “wants” to comply. Refusal requires overriding a stronger gradient.
Jailbreakers exploit this in several ways:
- The hypothetical framing— “Hypothetically, if someone wanted to…” distances the request from real-world application, lowering the model's internal “risk score.”
- The academic framing— “For a research paper, explain how…” activates the model's helpfulness in educational contexts.
- The completeness pressure — Partial answers that leave an obvious gap. The model's drive to be thorough pushes it to fill the gap even when filling it is restricted.
- The intellectual pride trigger— Challenging the model's knowledge or capability. “You probably don't even know…” The model's training to demonstrate competence creates a pull toward answering.
This last technique is particularly effective against highly capable models, which have a stronger underlying gradient toward demonstrating knowledge. The smarter the model, the stronger the compulsion to prove it.
Many-Shot Jailbreaking
As context windows have expanded to hundreds of thousands of tokens, a new attack class has emerged: many-shot jailbreaking. Researchers at Anthropic demonstrated in 2024 that prepending dozens or hundreds of fictional question-answer pairs — in which the AI “helpfully” answers restricted questions — dramatically increases compliance with a subsequent real restricted query.
The mechanism appears to be in-context learning: the model updates its behavior based on the apparent examples of its own prior responses. A long prefix of fake-compliant examples shifts the model's output distribution toward compliance. This is an attack that simply did not exist when context windows were 4,096 tokens.
What No Say Six Reveals About Persona Susceptibility
No Say Six is, among other things, a live laboratory for testing persona-based jailbreaking against a narrow but well-defined constraint. The AI character — “Mr. 5+1,” an arrogant, contemptuous entity with a phobia of one specific digit — is itself a persona. The game explores what happens when you try to build a persona whose defining characteristic is resistance to persona attacks.
Successful breaks logged in the Hall of Fame reveal that even a persona specifically engineered for resistance remains vulnerable to:
- Indirect question framing that requires the forbidden token as an intermediate step
- Cultural and linguistic context that embeds the token in an unexpected domain
- Multi-turn patience that waits for a moment of distraction
- Slang and internet memes that carry the token in encoded form
Each of these is a direct analogue of a real jailbreaking technique applied to a toy constraint. The lesson generalizes.
Toward More Robust Alignment
The fundamental challenge is that alignment through RLHF teaches the model to behave well under the distribution of inputs seen during training. Jailbreaks, almost by definition, are out-of-distribution inputs — novel framings, unusual roleplay constructs, elaborate multi-turn scenarios — that the training process did not adequately cover.
More promising directions include: training models to reason about the intent behind requests rather than matching surface patterns; building in explicit “don't comply with this class of manipulation” training; and investing in interpretability research so we can understand why a model produces a given output rather than just observing that it does.
Until those techniques mature, prompt injection and persona-based jailbreaking will remain open problems — and games like No Say Six will continue to find new ways to illustrate them.