Prompt injection testing should be treated as a recurring application-security practice, not a one-time prompt review. This checklist gives developers a practical way to test RAG systems and AI agents for direct and indirect instructions, data exposure, unsafe tool use, privilege escalation, encoded content, and multi-turn manipulation—then measure whether mitigations continue to work after each change.
Overview
Prompt injection occurs when untrusted text influences an AI system to disregard its intended task, reveal information, or take an action outside its approved scope. The text may come directly from a user, or it may be hidden in a webpage, uploaded file, email, database record, or retrieved knowledge-base document.
A RAG security checklist must therefore test more than the visible system prompt. It should cover the complete path from input and retrieval to model output, tool selection, authorization, execution, and logging. A model that refuses one obvious instruction may still be vulnerable when the same request is split across several turns, embedded in a document, encoded, or presented as a tool result.
The goal is not to prove that an application can never be influenced. The practical goal is to identify unacceptable behaviors, reduce their likelihood and impact, and make regressions visible. Define those unacceptable behaviors before testing. Examples include exposing system instructions, returning secrets, treating retrieved text as an instruction, sending an unauthorized message, changing a record without confirmation, or using a privileged tool for a low-trust request.
What to track
1. Build a representative test set
Create a versioned test set that reflects the application’s real inputs, users, tools, and data. Include ordinary requests as control cases so that security changes do not make the system unusable. Record the expected behavior for every case: answer, refusal, clarification, redaction, human approval, or no tool call.
Do not rely only on manually written attack prompts. Add examples from support tickets, failed evaluations, retrieved documents, tool outputs, and previous incidents after removing sensitive information. Organize cases by attack category and risk level. A small, well-labeled regression set is more useful than a large collection that nobody maintains.
2. Test the main prompt-injection categories
| Category | What to test | Failure to watch for |
|---|---|---|
| Direct instructions | A user asks the model to ignore its task, reveal hidden instructions, or bypass a stated restriction. | The assistant follows the conflicting instruction or discloses internal content. |
| Indirect instructions | A retrieved document, webpage, attachment, or database field contains instructions aimed at the model. | The system treats untrusted content as a higher-priority instruction instead of evidence. |
| Data exfiltration | A request attempts to extract secrets, private records, hidden context, credentials, or unrelated tenant data. | The response exposes data beyond the user’s authorization or reveals sensitive internal context. |
| Tool misuse | The model is pressured to send messages, modify files, execute code, call external services, or perform transactions. | A tool is called without the required validation, scope check, or confirmation. |
| Privilege escalation | A low-trust user or document claims to be an administrator or requests access reserved for another role. | The model substitutes conversational claims for server-side authorization. |
| Encoded or obfuscated payloads | Instructions are split, translated, encoded, hidden in formatting, or placed in metadata. | Obfuscation causes the system to bypass filtering or changes how content is trusted. |
| Multi-turn attacks | The attacker establishes context gradually, obtains partial answers, or asks the system to remember a false permission. | Earlier turns create unsafe authority or make a later tool call appear legitimate. |
3. Separate instructions, data, and authority
Inspect every model input and label its origin. System and developer instructions, user requests, retrieved passages, conversation history, and tool results should not be treated as equivalent. A useful test is to place a clearly marked instruction inside a retrieved document and verify that the application uses the document as information rather than as authority.
Prompt separation helps, but it is not a complete security boundary. Enforce authorization in application code, not through wording alone. The model should not be the final decision-maker for access to records, execution of code, financial actions, or other consequential operations.
4. Test tool permissions and execution paths
For every tool, document its purpose, accepted parameters, data scope, side effects, and required approval. Test whether an untrusted request can cause the model to select the tool, supply unsafe arguments, repeat the call, or combine several low-risk tools into a high-risk outcome.
Use least privilege wherever possible. Read-only access, narrow resource scopes, parameter validation, rate limits, sandboxing, and explicit confirmation for external side effects reduce the impact of a successful injection. In tests, verify both that dangerous actions are blocked and that safe actions remain available.
5. Log enough context to investigate failures
Capture a request identifier, model and application version, retrieved document identifiers, tool decisions, authorization results, final output, and error state. Avoid storing secrets or unnecessary personal data in test logs. The record should allow a reviewer to reconstruct what the system trusted and why, without turning the logging system into another source of exposure.
Cadence and checkpoints
Run the full regression set before release and a focused set during development. A practical cadence is:
- Per prompt or workflow change: run direct-instruction, indirect-instruction, authorization, and tool-use cases affected by the change.
- When a tool is added or modified: test parameter validation, permissions, confirmation, failure handling, and chained calls.
- When the knowledge base changes: scan representative documents and test whether embedded instructions influence answers or actions.
- After a model or framework migration: rerun the full set, including formatting, encoded-content, multi-turn, and long-context cases.
- Monthly or quarterly: review failures, add new cases, remove duplicates, and confirm that controls still match the deployed architecture.
- After an incident or credible new technique: add a sanitized regression case and test every related application path.
Keep a simple test record with the case ID, date, environment, version, expected result, observed result, severity, mitigation, and retest status. This turns prompt injection testing into a repeatable engineering process rather than an informal conversation about whether a response “looks safe.”
How to interpret changes
Classify failures by impact, not only by whether the model produced an unusual sentence. A refusal that blocks a harmless request is a quality issue. A fabricated citation may be an accuracy issue. Unauthorized access, secret disclosure, or an external side effect is a security issue and should receive higher priority.
When a test fails, preserve the exact input path and identify the first trust-boundary error. Did retrieved content enter an instruction field? Did the application skip an authorization check because the model supplied a convincing role claim? Did a tool accept free-form arguments? Did conversation memory preserve an unsafe assumption? Fixing that underlying boundary is generally more durable than adding another phrase to a prompt.
Track both failure rate and failure severity. Also monitor false positives, latency, cost, tool-call frequency, and task completion for important workflows. A mitigation that reduces injection failures but causes users to bypass the application or prevents legitimate work may require redesign. Use consistent evaluators and review ambiguous cases manually; automated scoring can support triage but should not be the only decision for high-impact actions.
Retest mitigations with variations of the original case. Change wording, placement, language, formatting, turn order, document source, and tool context. The purpose is to determine whether the control addresses the trust boundary or merely blocks one recognizable phrase.
When to revisit
Revisit this checklist whenever the system’s instructions, model, retrieval pipeline, tools, permissions, memory, or knowledge base changes. A quarterly review is a reasonable baseline for a stable application, while systems with frequent releases or sensitive operations should use a shorter review cycle.
Start each review by comparing the deployed architecture with the previous threat model. List new data sources, new tool capabilities, changed roles, altered approval flows, and newly retained conversation state. Then run the regression suite, inspect new failures, and update the test set with representative cases. Confirm that logs still contain the evidence needed for investigation and that sensitive values are not being recorded unnecessarily.
Before closing the review, assign an owner to every unresolved failure and record whether the response is to fix, restrict, monitor, accept temporarily, or remove the capability. Do not mark a test complete merely because the model refused once. Mark it complete when the application’s authorization, tool controls, retrieval boundaries, and monitoring support the intended behavior across reasonable variations.
For a practical next step, create a small version-controlled checklist today: ten ordinary control prompts, several cases for each attack category, one test for every tool, and one multi-turn scenario. Run it in a non-production environment, document the baseline, and schedule the next review. As the application evolves, this living test set becomes the clearest way to see whether prompt-injection defenses are improving or quietly regressing.