At the end of July OpenAI has begun rolling out a disturbingly human-like voice interface for ChatGPT. security analysis In a statement released today, the company admits that this anthropomorphic voice may encourage some users to form an emotional bond with the chatbot.
The warnings are included in a “system card” for GPT-4o, a technical document that outlines what the company believes are the risks associated with the model, as well as details of the security testing and mitigation actions the company is taking to mitigate the potential risks.
OpenAI has come under scrutiny in recent months after a number of employees who handled long-term AI risks left the company. Some later accused OpenAI of taking unnecessary risks and muzzling dissidents in the race to commercialize AI. Revealing more details about OpenAI’s security system could facilitate defuse criticism and reassure the public that the company is taking the issue seriously.
The risks explored in the fresh system sheet are broad, including the potential for GPT-4o to amplify social prejudice, spread disinformation, and aid in the development of chemical or biological weapons. It also details testing to ensure AI models don’t try to escape control, deceive people, or hatch catastrophic plans.
Some outside experts praise OpenAI for its transparency, but say it can go a step further.
Lucie-Aimée Kaffee, an applied policy researcher at Hugging Face, a company that hosts AI tools, notes that OpenAI’s system sheet for GPT-4o does not provide extensive details about the model’s training data or who owns that data. “The issue of consent for creating such a large dataset spanning multiple modalities, including text, image, and speech, needs to be addressed,” Kaffee says.
Others note that the risks can change as tools are used in the wild. “Their internal review should be only the first step in ensuring AI security,” he says. Neil ThompsonMIT professor who studies AI risk assessments. “Many risks only become apparent when AI is used in the real world. It’s important that these other risks are catalogued and assessed as new models emerge.”
The fresh system card underscores how quickly AI threats are evolving as powerful fresh features like OpenAI’s voice interface are being developed. In May, when the company showed off its voice mode, which can respond quickly and handle interruptions in natural conversations, many users noted that it seemed too flirtatious in demos. The company later faced criticism from actress Scarlett Johansson, who accused it of copying her speaking style.
The “Anthropomorphization and Emotional Dependency” section of the system card examines the problems that arise when users perceive AI in human terms, which is apparently exacerbated by the human-like voice mode. For example, during red teaming, the GPT-4o stress tests, OpenAI researchers noticed instances of user statements that conveyed a sense of emotional connection to the model. For example, people used language such as “This is our last day together.”
Anthropomorphism can make users more trusting of a model’s results when it “hallucinates” incorrect information, OpenAI says. Over time, it could even affect users’ relationships with other people. “Users can form social relationships with AI, reducing their need for human interaction—potentially beneficial for lonely people, but potentially impacting healthy relationships,” the paper reads.
Joaquin Quiñonero Candela, OpenAI’s head of readiness, says that voice mode could evolve into a particularly powerful interface. He also notes that the kind of emotional effects seen with GPT-4o could be positive—for example, by helping people who are lonely or need to practice social interactions. He adds that the company will be closely examining anthropomorphism and emotional connections, including by monitoring how beta testers interact with ChatGPT. “We don’t have results to share at this time, but it’s on our list of concerns,” he says.