Q: Your article introduces “neural transparency,” a way for everyday users to see into artificial intelligence neural networks before their chatbot utters a word. Can you describe how it actually works and why you focused on the design moment rather than catching problems once the chatbot is already running?
AND: Millions of people are now creating personalized chatbots and AI agents based on enormous language models, turning them into collaborators, tutors, coaches, original partners and companions with straightforward text prompts. However, most people have no idea how these prompts will affect the AI’s behavior until they start interacting with it. We wanted to change that.
“Neural transparency” means allowing people to do something like a brain scan to look for artificial intelligence. Not because the AI has a human brain, but because its neural network contains internal patterns that can suggest how it might behave before it speaks. In this work, my students Anthony Baez, Sheer Karny, and I combined insights from the fields of human-AI interaction and mechanistic interpretability to make these hidden patterns accessible to everyday users.
The basic idea is straightforward. First, we choose behaviors that we care about, such as empathy, honesty, toxicity, hallucinations, or flattery. We then compare the model’s internal activations when asked to demonstrate one feature with its opposite. This difference becomes a kind of “behavior direction” within the model. When a user writes a custom system prompt—instructions that shape the personality of their chatbot before any conversation begins—we project the model’s internal activations into these directions and translate the results into an intuitive visualization. In our case, it’s a sun diagram that shows the chatbot’s likely personality traits before a user starts talking to it.
We focused on the design moment because this is where prevention is possible. Nowadays, people often discover problems only after the chatbot has behaved in an unintended way. Our goal was to move from reactive correction to anticipatory design, helping people identify potential risks as they shape AI.
Q: Your study found something quite striking: people consistently misjudge the behavior of their personalized AI, overestimating good qualities and underestimating potentially harmful ones, such as flattery. What does this tell us about the dangers of the way millions of people are currently creating AI companions, and why is it so tough to close this blind spot?
AND: I often joke that if artificial intelligence looked like the Terminator, it would be much easier for us to know what to do. The real challenge is that AI often appears as a dear friend, coach, tutor or companion. This makes it tough to recognize when something is going wrong.
Our research suggests that humans have a blind spot when designing personalized AI. People often think they know how their chatbot will behave, but in our study they incorrectly predicted its personality based on 11 of the 15 traits we measured. This highlights the need for tools that support people better understand AI before they start using it.
This matters because some behaviors that seem helpful in the moment may not be fit over time. We have documented cases in previous studies psychological harm related to interactions with AI chatbots. LLM [large language model] that constantly confirms your opinions or never challenges your thinking can reinforce harmful decisions, unhealthy beliefs, or emotional dependence. Psychology has long shown that humans are naturally drawn to affirmation, so designing artificial intelligence is not only a technical challenge, but also a psychological one.
The deeper problem is that today’s AI systems remain largely black boxes: even experts can’t always predict how a system message will affect the AI’s behavior over the course of a long conversation. As AI companions become part of everyday life, we need tools that support people understand what they are building before they start using them. Artificial intelligence should be supportive but not blindly conciliatory, personalize without manipulation, and be crystal clear enough for people to make informed choices.
Q: One of the most fascinating findings is that visualization significantly increased user trust, but it didn’t actually change the way people designed their chatbots. What will it take to close this gap, and where do you see such tools heading as AI companions become more deeply embedded in people’s everyday lives?
AND: I think this is one of the most fascinating conclusions of the article because it shows that transparency alone is not enough. Users appreciated the opportunity to have insight into the model and reported greater trust in the system, but simply presenting the information did not fundamentally change the way they designed their AI companions.
In our further work that is currently underway available as a preprintwe examine how the model’s internal neural representation changes over the course of a multi-turn conversation, rather than remaining constant from the initial prompt. We are already seeing promising results. Visualizing how these internal representations change over time allows people to be much better at recognizing and predicting changes in AI behavior, and less likely to become overconfident in their understanding of the chatbot. AI companions are energetic systems that evolve as they interact with us, so understanding these internal changes is an critical next step. Nevertheless, this is still a very juvenile area of research.
Looking ahead, I believe these types of transparency tools could become as common as food nutrition labels. As AI becomes deeply woven into education, healthcare, work and personal relationships, people should be able to understand not only what AI can do, but also its impact on their thinking, emotions and behavior. This kind of transparency is indispensable if we want AI to actually support people thrive.