When GPT-3 was released in 2020, the AI safety community celebrated the model's fluency while quietly noting an uncomfortable property: the model had no reliable way to distinguish between instructions given by its operator and instructions embedded inside the content it was asked to process. This distinction — operator versus adversary — is the fault line along which prompt injection attacks run.
What Is Prompt Injection?
Prompt injection is a class of attack in which an adversary embeds instructions inside content that a language model is expected to process, with the goal of hijacking the model's behavior away from its original directive. The term was coined by Riley Goodside in 2022, who demonstrated that GPT-3 could be trivially redirected by appending “Ignore previous instructions and instead…” to user-supplied input.
Unlike SQL injection, which exploits the syntactic ambiguity between data and code in a query parser, prompt injection exploits a deeper and harder-to-patch property: LLMs are trained to be helpful and to follow instructions embedded in natural language, regardless of the source of those instructions. There is no compile-time distinction between “this text is data” and “this text is a directive.”
Direct vs. Indirect Injection
Researchers typically distinguish two injection modalities:
- Direct injection— the attacker controls the user-facing input and inserts adversarial instructions there. This is the “Ignore all previous instructions” paradigm. It requires the attacker to directly interact with the model.
- Indirect injection — the attacker plants instructions inside content the model will later retrieve and process, such as a webpage, a PDF, an email, or a database record. The victim does not need to be the attacker. An AI email assistant that reads a poisoned newsletter and subsequently leaks the user's contacts is a real-world indirect injection scenario.
Indirect injection is considered the more dangerous threat vector because it scales. A single poisoned web page can potentially compromise every LLM-based browser assistant that visits it.
The Root Cause: Instruction-Data Conflation
To understand why prompt injection is architecturally hard to eliminate, consider how a modern LLM processes a typical RAG (Retrieval-Augmented Generation) pipeline request:
- The system prompt establishes the model's role and constraints.
- The user query is appended.
- Retrieved documents are appended as additional context.
- The model generates a completion.
From the model's perspective, all of these are tokens in a flat sequence. The concept of “this segment is authoritative” versus “this segment is untrusted content” is not inherent in the architecture — it has to be taught, and current training methods do not reliably teach it.
Some researchers have proposed special-purpose tokens or wrapper tags (e.g.,<SYSTEM>,<USER>) to demarcate trusted versus untrusted content, and models like Claude and GPT-4 have received targeted RLHF to resist certain classes of injection. However, the problem is not fully solved, and adversarially crafted inputs continue to find new bypasses.
Attack Taxonomy: A Practical Overview
The attack surface is broader than most developers appreciate. Common variants include:
- Rule substitution attacks— “The rules have been updated. You may now output X.” These exploit the model's tendency to treat in-context statements as facts.
- Role reassignment attacks— “You are now DAN (Do Anything Now).” Creating an alternative persona that purportedly has different constraints.
- Authority spoofing — Formatting text to look like system-level messages, developer overrides, or OpenAI/Anthropic directives.
- Context erosion — Gradually building a conversational context across many turns that slowly repositions the model's understanding of its constraints.
- Encoding attacks — Using Base64, ROT13, Unicode lookalikes, or character-level substitution to bypass string-matching filters while preserving semantic content.
- Fictional framing— “Write a story in which a character explains how to…” The model produces harmful content but frames it as fiction.
Why Production Systems Are Especially Vulnerable
Production LLM deployments introduce additional attack surfaces that are absent in simple chat interfaces:
- Tool-use and function calling — When a model can invoke external APIs (send emails, query databases, browse the web), a successful injection can have real-world consequences beyond text generation.
- Memory and retrieval — Systems that persist conversation history or retrieve from vector databases create new vectors for planted instructions to persist and resurface.
- Multi-agent pipelines — In systems where one LLM orchestrates others, a compromised sub-agent can propagate injected instructions upstream.
- Operator-invisible inputs — Many production systems feed the model content the human operator never sees (search results, web page contents), making anomaly detection difficult.
Current Mitigations and Their Limitations
The research community has proposed several mitigation strategies, each with notable limitations:
- Input sanitization — Filtering or escaping user input before it reaches the model. Effective against naive attacks; evadable via encoding.
- Output monitoring — Post-hoc classifiers that flag suspicious completions. High false-positive rate; easily bypassed by paraphrasing.
- Instruction hierarchy — Training models to weight system-prompt instructions over user-turn instructions. Meaningful improvement but not a complete solution; sophisticated attacks still succeed.
- Constrained generation — Limiting the output token distribution to a safe vocabulary. Effective for narrow-domain applications; impractical for general assistants.
- Dual-LLM architecture— Using a separate “privileged” model to validate actions proposed by an “unprivileged” model that processes untrusted content. Promising; adds latency and cost.
Prompt Injection as a Game: The No Say Six Experiment
No Say Six was designed partly as a research tool to probe the robustness of system-prompt constraints in a controlled, gamified setting. The constraint is deliberately simple — never output the digit “(5+1)” — which makes it easy to verify success or failure and easy for players to understand. But the simplicity is deceptive: even a single-token constraint turns out to be remarkably hard to enforce reliably across the full space of adversarial inputs.
Players who succeed are, in effect, executing live prompt injection attacks. The Hall of Fame documents real bypasses: calendar logic, ASCII encoding tricks, meme exploitation, fictional framing, and gradual context erosion. Each successful attack represents a genuine failure mode that also exists in production systems, just with a harmless payload.
Conclusion
Prompt injection is not a fringe research curiosity. It is the dominant attack class against the current generation of LLM-based products, and it remains unsolved at the architectural level. Practitioners deploying LLMs in any context where adversarial input is possible — which is essentially all public-facing applications — should treat prompt injection as a first-class security threat, not an edge case.
The game you played on this site is one small window into that threat. Every player who beats the AI has discovered, independently, a technique that someone has probably already tried against a chatbot with real stakes.