Source: NVIDIA Open Agent Safety Platform: A Reference for Continuous In-Silicon Agent Monitoring - By John Myers, Alex Watson, Ali Golshan and Ofir Arkin
Cecile G. Tamura's post:
Jensen Huang just launched NVIDIA's answer to rogue AI agents: a hardware watchdog that quarantines them in milliseconds, backed by 100+ companies.
"Today, with over 100 industry partners, we introduced the NVIDIA Open Agent Safety Platform, bringing together OpenShell and Sentry.
Artificial intelligence is extraordinary technology that will advance discovery, productivity, security, health, and prosperity for generations to come.
But its full promise can only be realized when people have confidence that AI is being built to be safe and deployed with wisdom and responsibility.
This is bigger than a single product. It's the beginning of an open ecosystem to build the trust layer for safe agent systems.
Together, we are building the foundation of the AI economy.
Trust and innovation are not in conflict. Safety is how trust is earned. We must build not only the most capable AI, but the most trusted AI, so that this extraordinary technology can realize its enormous promise for the world. "
- Jensen Huang
As AI agents become more autonomous, the challenge is no longer simply how to make models more capable, but how to keep them within safe and clearly defined boundaries while they operate in the real world.
NVIDIA is proposing an interesting approach: instead of relying entirely on the AI model to behave safely, build an independent safety layer around it.
Its Open Agent Safety Platform combines OpenShell, which places agents in isolated sandboxes with explicit permissions, with NVIDIA Sentry, a monitoring and enforcement layer designed to operate independently on NVIDIA’s BlueField infrastructure. The idea is that an agent can use tools, access data and interact with external systems—but its actions remain subject to policies that the agent itself cannot modify or bypass.
This reflects a broader shift in thinking about AI safety. As agents run for longer periods, use more tools and accumulate more authority, failures can arise not only from the model itself but from unexpected interactions, repeated attempts, changing circumstances or poorly defined objectives.
Rather than assuming that an agent will always behave as intended, NVIDIA’s architecture takes a more “zero-trust” approach: sandbox the agent, continuously monitor its behavior, limit its permissions, and place the ultimate enforcement mechanism outside the agent’s control.
This does not make an AI system intrinsically aligned or eliminate hallucinations, faulty reasoning or unexpected behavior. Instead, it aims to make those failures harder to turn into unauthorized actions in the real world.
The larger question is whether the future of AI safety will increasingly move beyond model training and into the infrastructure surrounding autonomous agents—where hardware, software, access controls and continuous monitoring work together to determine not only what an AI can think, but what it is actually allowed to do.
Source: NVIDIA Developer Blog
[//developer.nvidia.com/blog/nvidia-open-agent-safety-platform-a-reference-for-continuous-in-silicon-agent-monitoring/]
Không có nhận xét nào:
Đăng nhận xét