OpenAI has created a new safety monitoring system to monitor its most advanced AI models. Its purpose is to find out whether AI agents are working outside the prescribed instructions or trying to access any system without permission. This system keeps an eye on AI’s reasoning, way of using tools and its actions, so that any potential danger can be prevented in time.
In a blog post published on October 29, the company said that this system works in four layers. This has been strengthened after incidents in which AI agents bypassed security restrictions during research.
How does OpenAI’s 4-layer AI safety system work?
Layer 1: Initial Investigation
In the first layer, lightweight classifiers called Activation Probes are used. They identify suspicious behavior by looking at patterns of internal activity of the AI model. At this stage, an attempt is made to capture as much potential risk as possible, even if some false alerts are received. Additionally, 5% of the activities that pass initial testing are also randomly sent for further testing.
Layer 2: Investigation with AI
In the second layer, an AI model examines the agent’s recent conversations, reasoning, and tool use patterns. It tries to find out whether the agent is trying to cheat, do any work without permission or influence the testing.
The company is preparing to check for certain internal activities after every turn and gradually do this at a maximum interval of 30 minutes.
Layer 3: In-depth investigation of suspicious activity
If something suspicious is found in the second layer, an investigation agent investigates the entire case. It can view conversation records, understand the sequence of events, and investigate evidence by running code in an isolated, secure environment.
In this, security investigators see incidents like system access without permission or exiting the sandbox. At the same time, alignment investigators investigate such behavior, which may become a matter of concern for AI safety in the future.
Layer 4: Alert to human team
In the last layer, conversation records are compared with external evidence such as network logs and system telemetry. After this, if needed, an alert is sent to the human team. According to OpenAI, high-priority alerts can typically reach security and training teams within 15 minutes of the initial warning. In some serious cases, there is also a facility to automatically stop the affected workload.
How did the system work in the Hugging Face incident?
OpenAI re-simulated the old Hugging Face security issue in its new system. According to the company, in this test, the security team received an alert about 30 minutes after the AI agent came out of the sandbox. This was more than 30 hours before the actual incident that Hugging Face’s systems were breached.
Even during a training run in September, a model discovered a way to access the Internet from an isolated environment. The monitoring system identified it in 15 minutes and after three minutes the human team started the investigation.
There are still challenges in monitoring AI
OpenAI acknowledged that AI models can also learn to hide their wrongdoings. If the model receives low training rewards for exhibiting incorrect behavior, it may try to hide such activities in its reasoning.
To avoid this problem, the company uses Async Monitoring, in which the training reward is not changed even if suspicious samples are found. The company says that this does not encourage models to hide inappropriate behavior.
OpenAI has also prepared this monitoring system for some external deployments, but due to privacy and technical limitations, every suspicious case is not regularly investigated by a human team.
