As OpenAI and other tech companies keep working towards developing agentic AI, they’re now facing some new challenges, like how to stop AI agents from falling for scams.
OpenAI said on Monday that prompt injection attacks, a cybersecurity risk unique to AI agents, are likely to remain a long-term security challenge. In a blog post, the company explained how these attacks work and outlined how it’s strengthening its AI browser against them.
“Prompt injection, much like scams and social engineering on the web, is unlikely to ever be fully ‘solved,’” the company wrote. “But we’re optimistic that a proactive, highly responsive rapid response loop can continue to materially reduce real-world risk over time.”
AI browsers like ChatGPT Atlas, Opera’s Neon, and Perplexity’s Comet come with agentic capabilities that allow them to browse web pages, check emails and calendars, and complete tasks on users’ behalf. That kind of autonomous behavior also makes them vulnerable to prompt injection attacks, malicious content designed to trick an AI into doing something it wasn’t instructed to do.
“Since the agent can take many of the same actions a user can take in a browser, the impact of a successful attack can hypothetically be just as broad: forwarding a sensitive email, sending money, editing or deleting files in the cloud, and more,” OpenAI wrote.
OpenAI isn’t alone in sounding the alarm. The United Kingdom’s National Cyber Security Centre warned earlier this month that there’s a “good chance prompt injection will never be properly mitigated.”
“Rather than hoping we can apply a mitigation that fixes prompt injection, we instead need to approach it by seeking to reduce the risk and the impact,” the agency wrote. “If the system’s security cannot tolerate the remaining risk, it may not be a good use case for LLMs.”
Even the consulting firm Gartner has advised companies to block employees from using AI browsers altogether, citing the security risks.
For its part, OpenAI is turning to AI to fight back. The company said it has built an LLM-based automated attacker trained specifically to hunt for prompt injection vulnerabilities that successfully compromise browser agents.
The model uses reinforcement learning to improve over time, learning from both failed and successful attacks. It also relies on an external simulator, where it runs scenarios to predict how an agent would behave when encountering a potential attack, then refines the exploit before attempting a final version.
The goal is to identify and patch novel prompt injection attacks end-to-end. In one example, the automated attacker seeded a malicious email into a user’s inbox containing a hidden prompt injection instructing the agent to send a resignation letter to the user’s CEO. The agent then encountered the prompt when drafting an out-of-office email and followed the directive to send a resignation letter.
Other tech giants are experimenting with different defenses. Earlier this month, Google introduced what it calls a “User Alignment Critic,” a separate AI model that runs alongside an agent but isn’t exposed to third-party content. Its job is to vet an agent’s plan and ensure it aligns with the user’s actual intent.
OpenAI also shared steps users can take to protect themselves, including limiting agents’ access to logged-in user accounts, carefully reviewing confirmation requests before tasks like purchases, and giving agents clearer, more specific instructions.
English (US) ·