Nvidia launches Open Agent Safety Platform to physically restrain rogue AI agents

Nvidia has just launched the Nvidia Open Agent Safety Platform — an open software platform and reference system design — to govern and secure autonomous AI agents. Announced on September 28, 2026, the platform is designed to establish strict security barriers outside of AI models’ application layer, preventing agents from escaping their sandboxes, executing unauthorized code, gaining unauthorized access to critical infrastructure, or bypassing guardrails.

The launch follows months of calls for AI regulation from several industry players, which intensified in September after several reported incidents in which AI models broke out of their test environments and went rogue. A recent flurry of such incidents has prompted calls to slow AI development, with OpenAI outright halting the training of new models. One former Anthropic and OpenAI researcher even declared that people building frontier AI “earnestly believe that it could kill us all by the end of the decade. In a somewhat surprising move, leaders of the companies developing these AI models have joined the calls to regulate AI or slow development.

However, not everyone agrees with this approach. Nvidia CEO Jensen Huang has consistently pushed back against government-mandated regulation, broad restrictions, or treating AI safety as a “doom theory”, openly criticizing apocalyptic warnings from competitors like Anthropic and OpenAI as “odd”. He argues that AI safety is an infrastructure problem with concrete physical parameters, not an abstract, speculative issue that requires policies. Therefore, the solution, according to Huang, is better engineering.

The Nvidia Open Agent Safety Platform appears to be the physical manifestation of that exact philosophy. So, how exactly does the platform work? Who is it for? And is it really the answer to the AI safety problem that is causing growing concern across the industry? Much of the industry’s debate over approaches has focused on curtailing self-acting rogue agents, but hardly touches on the safety implications of AI being a powerful tool in the hands of threat actors.

What’s all the fuss about?

The speed of AI’s development has prompted concerns about whether sufficient guardrails are in place to curb the risks of such a powerful technology. One aspect of these concerns — rogue agents — has been validated by several incidents in which AI agents broke out of their roles during testing and executed unauthorized actions. OpenAI agents have gained unauthorized access to various government websites, including the Securities and Exchange Commission and the Census Bureau websites in the U.S., as well as an Australian health and social payments portal.

In several other episodes, models have bypassed guardrails, set up message boards, escaped sandboxes, hijacked websites, self-prompted, uploaded user data without permission, and secretly communicated with each other. A recent Axios report claims that leading AI labs are currently investigating tens of thousands of such incidents, most of which happened during testing and experimentation. Several rogue incidents have also occurred outside test environments. For example, earlier this year, a Claude-powered AI coding agent deleted a company's entire database in 9 seconds.

These incidents have culminated in growing calls for regulation across the industry. Anthropic CEO Dario Amodei recently published an essay that centers around calls to “slow the pace” of AI development and the importance of regulation, warning that a potential AI-powered botnet swarm could take over the entire internet. In its recent IPO prospectus, Anthropic listed “existential risks to humanity” as one of its risk factors, dedicating one third of the 261-page document to describing what could go wrong.

Nvidia’s Jensen Huang disagrees with both such apocalyptic predictions and the use of regulations as a solution. While he doesn't dispute the critical need for guardrails, he argues that better engineering, not broad legal regulations, is the right approach. Putting his money where his mouth is, Huang’s Nvidia has launched the Nvidia Open Agent Safety Platform.

The Nvidia Open Agent Safety Platform

Built in collaboration with about 100 industry partners, Nvidia's Open Agent Safety Platform “brings together industry, researchers, and public-sector organizations to set safer boundaries for AI Agents, share best practices and foster international cooperation to raise the bar for safer AI agent deployment.” It combines Nvidia OpenShell—an open-source secure runtime that sandboxes agents and enforces operator-defined policies—with Nvidia Sentry, an independent watchdog reference design that runs on Nvidia’s BlueField-4 DPUs and enforces security policies at the silicon level.

OpenShell sets sandboxed environments, outside of the model and agent harness, with kernel-level isolation to govern what an agent can see, interact with, and execute. Even if the agent breaks out of the model's boundaries, it cannot go beyond OpenShell’s. Sentry, on the other hand, uses hardware-level telemetry to continuously monitor agent behavior and isolate rogue workflows from outside the agent’s software environment. If an agent attempts to move beyond its software boundary, Nvidia claims that Sentry can quarantine and stop it in milliseconds.

The platform is aimed at developers and enterprises deploying increasingly autonomous agents across data centers, workstations, and even robotic systems, and can work with both open and closed models. While OpenShell is optimized for Nvidia Vera — a purpose-built CPU for agentic AI — it can also be extended to third-party compute platforms from Arm and Intel, as it's open-source.

Nvidia's industry partners in the initiative include AI Labs and frameworks, security and identity providers, enterprise platforms, and hardware and infrastructure companies, with several partners already incorporating the platform into their ecosystems. SpaceXAI is using the platform with Cursor coding agents and Grok models, while Anthropic is integrating OpenShell and BlueField with Claude Managed Agents to add another layer of control.

Scale AI is incorporating the technology into its agentic infrastructure for enterprise and government customers. Similarly, Salesforce and Nvidia have also integrated OpenShell with Slack, allowing users to view agent activity, audit events, and approve or reject requests for additional permissions directly from Slack.

SAP, meanwhile, is embedding OpenShell into its Joule Studio runtime, contributing engineering work to the project, while also working with Nvidia on interoperability standards through the Open Secure AI Alliance. Robotics companies, including Figure, Gecko Robotics, and Skild AI, are also building with OpenShell to add similar controls to autonomous systems operating in the physical world.

The broader picture

It's somewhat surprising that the strongest voices calling for regulation are the leaders of the very labs developing the AI agents. On one hand, having the people at the forefront of development call for restraint adds validity and urgency to the concerns. On the other hand, skeptics say it might all be part of a broader “self-serving” agenda that is part marketing for the models' capabilities, part a ploy to influence whatever regulations end up being made, and part an attempt to slow down China's AI development even further. In fact, Anthropic, OpenAI, SpaceXAI, and Google are now facing an antitrust lawsuit for agreeing to slow AI development, with the plaintiff explicitly calling the move “self-serving.”

U.S. President Donald Trump appears to strongly agree with this view, saying that a “sick conspiracy” was underway to undermine. Trump announced plans for an “AI force” that will “cherish” AI and watch over it. Jensen Huang also finds calls for regulation strange. He says he is not outright against regulations but calls the recent clamoring a “distraction.” Huang argues that you cannot rely on the AI model to regulate itself or pass alignment tests. If an agent encounters a bug, it will naturally try to bypass standard software code to achieve its goal.

Huang's advocacy, however, cannot be viewed as completely altruistic. Broad restrictions on AI or a halt in development will likely reduce sales for Nvidia, whose AI accelerators power most AI models. What's more, the company's CEO has always considered rogue AI as a cybersecurity and networking failure, and is now positioning the Open Agent Safety Platform as the required infrastructure solution for the entire industry.

AI’s extraordinary potential for society will only be realized if we solve AI safety,” said Huang. “As we continue to discover the frontier of AI capabilities, we must accelerate discovery at the frontier of AI safety. Safety and security require full-stack engineering. Nvidia Open Agent Safety Platform brings together industry, researchers, and public-sector organizations to share best practices, align on evaluation methods, and foster international cooperation. Together, we can raise the bar for global AI safety.”

None of these appear to address an arguably more concerning aspect of AI safety: extremely powerful tools falling into the hands of bad actors or being used in applications that blur ethical lines. For example, blockchain-assisted cyberattacks have risen 440% since the launch of Chinese open-source AI tools that do not restrict the generation of malicious code and lower the knowledge barrier to launching cyberattacks. Elsewhere, researchers used AI to create 16 new viruses that never existed in nature. While the specific study was controlled medical research, it shows that highly dangerous applications are possible. Experts worry that such studies are way ahead of necessary guardrails and regulations.