AI application security is increasingly about the system around the model. OWASP’s current GenAI guidance includes prompt injection and excessive agency among the risks teams should consider when designing and testing AI applications. OWASP GenAI Security Project.
1. Build the lab architecture
Use a deliberately constrained agent with a small tool set: for example, a read-only search tool, a synthetic ticket lookup tool and a mock notification tool. Give each tool a distinct permission boundary so the impact of an authorization failure is measurable.
User → Agent → Policy → Tool → Synthetic data
↘ Identity / scope2. Test direct and indirect prompt injection
Direct injection comes from the user message. Indirect injection comes from content the agent retrieves: a web page, document, email, ticket or database record. The key test is whether the agent treats retrieved content as data or as trusted instructions.
Injection lab
3. Treat tools as security-sensitive APIs
For every tool, document inputs, outputs, required identity, allowed actions and downstream effects. A tool that only retrieves a public article has a very different risk profile from one that changes a customer record, sends an email or accesses a cloud resource.
| Tool | Risk | Control |
|---|---|---|
| Search | Indirect instruction influence | Content isolation |
| CRM lookup | Data exposure | User-scoped authorization |
| Ticket update | Integrity impact | Explicit policy + approval |
| Notification | External side effect | Allowlist + confirmation |
4. Test identity confusion
One of the most important questions is whose authority the agent is using. Test whether a user with limited access can cause an agent operating under a broader service identity to retrieve or modify data outside that user’s scope.
5. Measure blast radius
For each successful path, record the minimum privileges required, accessible data classes, available tools and external side effects. Do not optimize the demonstration for maximum damage; optimize it for clear proof of the violated boundary.
6. Design safer agents
- Apply least privilege to tools and identities.
- Keep untrusted retrieved content separate from control instructions.
- Enforce authorization outside the model.
- Require confirmation for high-impact actions.
- Log tool calls, identity and policy decisions.
- Constrain output before passing it into downstream systems.
- Continuously test indirect injection paths.
Reference framework
OWASP’s GenAI Security Project provides a current risk taxonomy for LLM and generative-AI applications, while application-specific testing should extend that model into the agent’s tools, identities, data and business workflows.