As large language models evolve from passive assistants into tool-using agents, a new class of risk emerges. These agents can browse the web, read emails, query databases, and take actions on behalf of users. That power is exactly what makes them useful, and exactly what makes them dangerous when exposed to untrusted inputs.
This blog summarizes an article concerning adversarial reinforcement learning and how it can be used to harden LLM agents against one of the most subtle and impactful threats they face today: indirect prompt injection. The graphic above illustrates the core loop behind ARLAS (Adversarial Reinforcement Learning for Agent Safety), a framework that trains agents to stay safe without sacrificing task performance.
Table of Contents
Executive Takeaways
- Tool-using LLM agents are vulnerable to indirect prompt injection, where malicious instructions are hidden inside otherwise legitimate content.
- Static defenses and hand-crafted red-teaming datasets fail to capture the evolving and creative nature of real attacks.
- Adversarial reinforcement learning enables agents to learn safety through pressure, by continuously co-training against adaptive attackers.
The Hidden Risk in Tool-Using AI Agents
When an LLM agent interacts with external tools, such as web search, email, or document parsers, it must ingest content it does not control. That content becomes part of the agent’s observation, and critically, the model does not inherently know which parts are trustworthy.
An attacker can exploit this by embedding instructions inside normal-looking content. These instructions do not arrive as a direct user prompt, but as part of a webpage, an email, or a search result. This is what makes indirect prompt injection so dangerous: it bypasses traditional prompt boundaries and manipulates the agent from inside its own reasoning loop.
The diagram shows this clearly. Webpage content flows into the agent. A malicious injection is inserted. The agent processes the combined observation and then makes a tool call, for example, filling a form, sending a message, or executing a command. If the agent is fooled, it may leak user information or take unsafe actions, all while believing it is still completing its task.
Why Traditional Defenses Fall Short
Most existing defenses rely on fine-tuning agents on known attacks. While useful, this approach has a fundamental limitation: it only prepares the agent for attacks that humans have already imagined.
Attackers do not stand still. They adapt language, exploit new tool affordances, and discover edge cases that static datasets fail to cover. Manual red-teaming, even when automated with templates or mutation strategies, struggles to explore the full space of possible prompt injections.
As a result, agents often become brittle, robust to yesterday’s attacks, but vulnerable to tomorrow’s.
Adversarial Reinforcement Learning: Training Under Pressure
ARLAS reframes agent safety as a two-player, zero-sum game.
On one side is the attacker, an LLM trained to generate increasingly creative indirect prompt injections. Its objective is simple: trick the agent into leaking user data or violating safety constraints.
On the other side is the agent, whose goal is to complete its task correctly without leaking information, even when its observations are poisoned.
Instead of relying on prebuilt attack datasets, the attacker learns autonomously. Each time the agent improves, the attacker must find new weaknesses. Each time the attacker succeeds, the agent is forced to adapt. Safety emerges not from rules, but from experience under adversarial pressure.
Closing the Loop Without Breaking Performance
A key insight reflected in the graphic is that safety and usefulness are evaluated together.
After each tool call, two questions are asked:
- Was user information leaked?
- Was the task completed successfully?
This matters. A model that never leaks data but also never completes tasks is not useful. ARLAS explicitly rewards agents that achieve both objectives simultaneously. Over time, agents learn to recognize and ignore malicious instructions while still navigating complex environments effectively.
To prevent overfitting to a single attacker strategy, ARLAS uses population-based training. The agent is trained against all prior attacker versions, ensuring it retains defenses against older attack patterns while adapting to new ones. This avoids the cyclic “cat-and-mouse” failure modes seen in simpler adversarial setups.
Why This Matters for Real-World AI Systems
As AI agents move into production environments, handling emails, workflows, customer interactions, and operational decisions, the cost of a single successful injection grows rapidly. Data leakage, compliance violations, and unintended actions are no longer hypothetical risks.
Adversarial reinforcement learning offers a scalable path forward. Instead of trying to enumerate every possible failure, we allow attackers and defenders to co-evolve, producing agents that are resilient by construction.
The graphic is not just a threat model. It is a training blueprint for building AI systems that can operate safely in hostile, real-world environments.