OpenAI researchers are experimenting with a new approach to design neural networks to make it easier to understand, debug and manage AI models. Scant models can provide enterprises with a better understanding of how these models make decisions.
Understanding how models choose to respond, which is a gigantic advantage of reasoning models for enterprises, can provide a level of trust for organizations when they turn to AI models for insights.
The method required OpenAI scientists and researchers to look at and evaluate models not by analyzing performance after training, but by adding the ability to interpret or understand through limited circuits.
OpenAI notes that much of the opacity of AI models is due to the way most models are designed, so workarounds should be developed to better understand model behavior.
“Neural networks power the most powerful artificial intelligence systems today, but they are still difficult to understand,” OpenAI wrote in a blog post. “We don’t write these models with explicit step-by-step instructions. Instead, they learn by adjusting billions of internal connections or weights until they master the task. We design the rules for training, but not the specific behaviors that emerge, and the result is a dense web of connections that no human can easily decipher.”
To escalate the interpretability of the mix, OpenAI has explored an architecture that trains non-convolutional neural networks, making them easier to understand. The team trained language models with an architecture similar to existing models such as GPT-2 using the same training scheme.
The result: improved interpretability.
The path to interpretability
Understanding how models work, which gives us insight into how they make determinations, is critical because they have real-world impacts, OpenAI says.
The company defines interpretability as “methods that help us understand why a model produced a given result.” There are several ways to achieve interpretability: chain-of-thought interpretability, which reasoning models often exploit, and mechanistic interpretability, which involves reverse engineering the mathematical structure of the model.
OpenAI focused on improving mechanistic interpretability and said it was “less immediately useful so far, but could in principle offer a more complete explanation of model behavior.”
“By trying to explain model behavior at the most detailed level, mechanistic interpretation can lead to fewer assumptions and give us more confidence. However, the path from low-level details to explanations of complex behavior is much longer and more difficult,” says OpenAI.
Better interpretability allows for better oversight and provides early warning signals if model behavior is no longer consistent with policy.
OpenAI noted that improving mechanistic interpretation “is a very ambitious undertaking,” but research on limited networks has improved this result.
How to untangle a model
To untangle the mess of connections created by the model, OpenAI first pruned most of these connections. Because transformer models like the GPT-2 have thousands of connections, the team had to “zero” these circuits. Everyone will only talk to the selected number, making calls more organized.
The team then performed “circuit tracing” across the tasks to create groups of interpretable circuits. The last task was to pristine the model “to obtain the smallest circuit that achieves the target loss in the target distribution” according to OpenAI. A loss of 0.15 was intended to isolate the exact nodes and weights responsible for the behaviors.
“We show that pruning our models with sparse masses results in approximately 16 times smaller circuits in our tasks than pruning dense models with comparable losses before training. We are also able to construct arbitrarily accurate circuits at the expense of more edges. This shows that circuits for simple behaviors are much more disentangled and easier to locate in sparse-weight models than in dense models,” the report says.
Miniature models become easier to train
While OpenAI has managed to create limited models that are easier to understand, they remain much smaller than most of the core models used by enterprises. Enterprises are increasingly using compact models, but pioneering models like the flagship GPT-5.1 will continue to benefit from improved interpretation in the future.
Other modelers are also trying to understand how their AI models think. Anthropicwho has been exploring the possibility of interpretation for some time, recently revealed that he had “hacked” Claude’s brain – and Claude noticed it. Meta is also working to learn how reasoning models make decisions.
As more enterprises exploit AI models to assist make decisions that have implications for their businesses and, ultimately, their customers, research should be conducted to understand how model thinking would provide the transparency many organizations need to trust models more.