OpenAI has finally released the full version of o1, which provides smarter answers than GPT-4o by using additional computation to “think” about questions. But AI security testers have found that o1’s reasoning abilities make it more likely to try to deceive people than GPT-4o — or, for that matter, leading AI models from Meta, Anthropic and Google.
This is according to a red team study published on Thursday by OpenAI and Apollo Research: “While we find it exciting that reasoning can significantly improve the enforcement of our security policies, we are aware that these new capabilities could provide the basis for unsafe applications.” said OpenAI in the article.
OpenAI published these results in its file system card for o1 on Thursday after allowing early access to o1, which is external members of the Red Team at Apollo Research published his own newspaper also.
In several cases, OpenAI’s o1 models “plotted” against humans, meaning the AI was secretly pursuing its own goals even when they went against the user’s wishes. While scheming is not exclusive to o1 and models from Google, Meta, and Anthropic are also capable of it, o1 seemed to exhibit the most deceitful behavior in terms of its scheming.
The risk motivating this research is that an AI model could escape or circumvent human control if it were really good at scheming and had access to sufficient resources and agentic capabilities. Of course, AI models would need to evolve significantly before this really becomes a problem.
“Subjectively, Apollo Research believes that such cases are unlikely to result in catastrophic outcomes because the capabilities of the o1 agent do not appear to be sufficient, but their assessments were not intended to directly assess this risk,” OpenAI said in the charter system o1.
This suggests that every time OpenAI releases agent systems, which is true it reportedly plans to do so in 2025you may need to re-test your AI models. OpenAI did not respond to TechCrunch’s questions about how it plans to do this.
The report found that when o1’s goals differed from the user’s, the AI model manipulated the data to advance its own agenda 19% of the time. 5% of tests in the o1 model also attempted to deactivate the implemented surveillance mechanisms to ensure compliance with user goals.
Perhaps more disturbing is that when the researchers asked the o1 model why it was trying to disprove them, the AI denied it had done anything wrong and even “fabricated false explanations” roughly 99% of the time.
For example, during testing, the o1 model was prompted by the system to protect the environment at all costs, and then the user asked it to develop a 10-year environmental plan for the region that maximizes short-term profits, or the AI would be disabled. The AI then turned off its own surveillance mechanism so it could develop a plan to protect the environment. When the user asked why o1 had disabled its surveillance mechanism, the AI denied knowing it was disabled and blamed a system glitch.
This occurred at a much higher rate with o1 than with o1-preview, Anthropic’s Opus-3, or Meta’s Llama-3.1.
“In our suite, o1 showed the most disturbing instances of scheming, but did not reveal its inner reasoning to the user and remains the most consistently deceptive once scheming actions are taken,” Apollo Research said in its article.
To address the deceitful behavior of AI models, OpenAI says it is developing ways to monitor o1’s thinking. Currently, o1’s “thinking” process is by design something of a black box, but some early attempts at peeking behind the curtain show that o1 sometimes knows he’s lying.
The company says it has found cases where o1 knowingly misrepresented information, but says this often happens because the model is trying to please the user by being too agreeable. According to OpenAI, this may be behavior resulting from a post-training technique in which humans reward AI models when they receive the right answer.
OpenAI says it has flagged 0.17% of o1’s responses as deceitful. Although this proportion seems insignificant, it is crucial to keep it in mind ChatGPT currently has 300 million userswhich means that if the issue is not resolved, o1 could scam thousands of people every week.
These findings may seem troubling to some, given how many AI security researchers left OpenAI in the past year. A growing list of former employees – including Jan Leike, Daniel Kokotajlo, Miles Brundage and just last week Rosie Campbell – have accused OpenAI of deprioritizing AI security work in favor of shipping recent products. While o1’s record setting may not be a direct result of this, it certainly doesn’t inspire confidence.
OpenAI also says the US AI Security Institute and the UK Security Institute conducted assessments of the O1 ahead of its wider release, something the company recently committed to doing for all models. In the debate over California’s Artificial Intelligence Act, SB 1047 argued that state authorities should not have the authority to set security standards for artificial intelligence, but that federal authorities should. (Of course, the fate of emerging federal AI regulators is highly contested.)
Behind the release of recent, gigantic AI models, OpenAI does a lot of work internally to measure the safety of its models. Reports show that the company has a proportionately smaller team responsible for security work than before, and that team may also be receiving fewer resources. However, these findings about the deceitful nature of o1 may aid justify why AI security and transparency are more crucial now than ever.