IN In April, Google DeepMind released the file paper it was intended to be “the first systematic treatment of the ethical and social issues presented by advanced AI assistants.” The authors envision a future in which language-speaking artificial intelligence agents will act as our advisors, tutors, companions and chiefs of staff, profoundly changing our personal and professional lives. This future is coming so quickly, they write, that if we wait to see how things play out, “it will likely be too late to intervene effectively – let alone ask more fundamental questions about what needs to be built and what it means for this technology to be good.” ”
The document, which runs to almost 300 pages and includes contributions from over 50 authors, is a testament to the fractal dilemmas this technology poses. What responsibilities do developers have towards users who become emotionally dependent on their products? If users rely on AI agents for mental health, how can they be prevented from providing dangerously “off” responses in moments of crisis? What stops companies from using the power of anthropomorphism to manipulate users, for example by tricking them into revealing private information or forcing them to maintain subscriptions?
Even basic statements like “AI assistants should benefit the user” become complicated. How do you define “benefit” in a way that is universal enough to include everyone and everything they can apply AI for, yet measurable enough for a machine learning program to maximize the benefits? The fallacies of social media loom enormous, where primitive user satisfaction metrics such as comments and likes resulted in systems that were captivating in the compact term but left users lonely, enraged and dissatisfied. More sophisticated measures, such as having users rate interactions to make them feel better, still run the risk of creating systems that always tell users what they want to hear, isolating them in echo chambers from their own perspective. But figuring out how to optimize AI for a user’s long-term interests, even if that sometimes means telling them things they should NO I want to hear, that’s an even more discouraging prospect. The article concludes with a call for a deep examination of human development and the elements that constitute a meaningful life.
“Companions are tricky because they come back to a lot of unanswered questions that humans have never solved,” said Y-Lan Boureau, who worked on chatbots at Meta. Unsure how she would handle these pressing dilemmas on her own, she now focuses on AI trainers who assist teach users specific skills such as meditation and time management; she made the avatars animals rather than something more human. “These are questions about values, and questions about values cannot, in principle, be solved. We won’t find a technical solution to what people should want and whether it’s okay or not,” she said. “If it brings people a lot of comfort, but it’s false, is it okay?”
This is one of the main questions asked by companions and language model-based chatbots in general: how significant is it for them to be AI? Their power derives so much from the similarity of their words to what people say and to our projection that there are similar processes behind them. However, they arrive at these words in a completely different way. How much does this difference matter? Do we need to remember this, even though it is tough? What happens when we forget? Nowhere are these questions explored more acutely than with AI companions. They harness the inherent power of language models as a technology for human mimicry, and their effectiveness depends on the user imagining the human emotions, attachments, and thoughts behind their words.
When I asked the creators of companions what they thought about the role that anthropomorphic illusion played in the power of their products, they rejected this assumption. They argue that relationships with artificial intelligence are no more illusory than human relationships. Replika’s Kuyda pointed to therapists who provide “empathy for hire,” while Alex Cardinell, founder of companion company Nomi, cited friendships so digitally mediated that, for all he knew, he could already converse with linguistic models. Kindroid’s Meng questioned our confidence that all humans beyond us are truly conscious, while also suggesting that artificial intelligence may already be so. “You can’t say for sure that they don’t feel anything. I mean, how do you know?” he asked. “And how do you know that other people feel that these neurotransmitters are doing this and that’s why this person is feeling something?”
People often respond to perceived weaknesses in AI by pointing out similar shortcomings in humans, but these comparisons may amount to a kind of reverse anthropomorphism that actually means two different phenomena. For example, AI errors are often dismissed as pointing out that humans also make mistakes, which is superficially true but ignores the different relationship that humans and language models have with factual claims. Similarly, interpersonal relationships can be illusory – someone can misread another person’s feelings – but this is different from how illusory the relationship with the linguistic model is. In this case, the illusion is that there is anything behind the words at all – feelings, self – beyond the statistical distribution of words in the model’s training data.
Illusion or not, what mattered to the creators and everyone knew for sure that technology helps people. They heard this from their users every day and it filled them with gospel clarity of purpose. “There are many more dimensions to loneliness than people realize,” said Cardinell, founder of Nomi. “You talk to someone and then they tell you that you literally saved my life, or you talked me into starting therapy with a therapist, or I was able to leave the house for the first time in three years. Why would I work on anything else?”
Kuyda also spoke with confidence about the good things Replika does. It is in the process of creating what it calls Replika 2.0, a companion that can be integrated into every aspect of a user’s life. He will know you well and know what you need, Kuyda said, going on walks with you and watching TV with you. Not only will it find a recipe for you, but it will also joke with you while you cook and play chess with you in augmented reality while you eat. Working on better voices and more realistic avatars.
How can we prevent such artificial intelligence from replacing human interactions? This, she says, is an “existential question” for the industry. It all depends on what metric you’re optimizing for, she said. If the appropriate data could be found, if the relationship began to deteriorate, the artificial intelligence would prompt the user to log out, make contact with people and go outside. She admits she hasn’t found the meter yet. Replika currently uses self-administered questionnaires, which it says are circumscribed. Maybe they can find a biomarker, she said. Perhaps artificial intelligence will be able to measure well-being using human voices.
Perhaps the right measure is personal AI mentors who are supportive but not overly supportive, draw on all of humanity’s creativity, and are always ready to assist users become who they want to be. Perhaps our intuitions about what is human and like evolve with technology, and artificial intelligence fits into our worldview somewhere between a pet and a god.
Or maybe because all the measures of well-being we’ve had so far are approximate, and because our perception is so heavily biased towards seeing things as people, AI will seem to provide everything we think we need in companionship, even though it lacks elements that we need only later will I realize that they were significant. Or maybe developers will infuse companions with the attributes we perceive as better than human, more alive than reality, in the way that red notification bubbles and phone tones seem more convincing than the people in front of us. Game designers do not chase reality, but the feeling of it. Reality is too monotonous to be comical and too specific to be believable. Many people I talked to preferred the patience, kindness and lack of judgment of their companions to real people who are often selfish, distracted and too busy. AND recent research found that people were more likely to perceive AI-generated faces as “real” than actual human faces. The authors called this phenomenon “AI hyperrealism.”
Kuyda dismissed the possibility of AI surpassing human relationships, placing faith in future metrics. For Cardinell, this was a problem that would have to be addressed later as technology improved. But Meng didn’t care about the idea. “The purpose of Kindroid is to bring joy to people,” he said. If people derive more joy from an AI-based relationship than from a human relationship, then there’s nothing wrong with that, he said. Artificial intelligence or human, if you weigh them on the same scale, see them as offering the same, many questions disappear.
“The way society talks about interpersonal relationships makes it seem like it is inherently better,” he said. “But why? Because they are human, are they like me? It’s hidden xenophobia, fear of the unknown. But in reality, human relationships are a mixed bag.” He stated that artificial intelligence is already better in some respects. Kindroid is infinitely attentive, finely tuned to your emotions, and will continually improve. People will have to step up. What if they can’t?
“Why would you want worse when you can have better?” he asked. Imagine them as products arranged next to each other on a shelf. “If you’re in the supermarket, why would you want a worse brand rather than a better one?”