OpenAI’s GPT-4o, the generative AI model that powers ChatGPT’s recently launched Advanced Voice Mode alpha, is the company’s first model trained on voice, as well as text and image data. And that causes it to behave in strange ways, sometimes — like mimicking the voice of the person speaking to it, or randomly shouting in the middle of a conversation.
IN new red team report documenting research into the model’s strengths and weaknesses, OpenAI exposes some of GPT-4o’s quirks, such as its aforementioned voice cloning. In occasional cases — particularly when a person is talking to GPT-4o in a “high-noise environment” like a car on the road — GPT-4o “emulates the user’s voice,” OpenAI says. Why? Well, OpenAI attributes that to the model having trouble understanding garbled speech. Okay!
Listen to how it sounds in the example below (from the report). Weird, right?
To be clear, GPT-4o doesn’t do this right now — at least not in Advanced Voice Mode. An OpenAI spokesperson told TechCrunch that the company has added “system-level mitigations” for this behavior.
GPT-4o is also prone to generating disturbing or inappropriate “nonverbal vocalizations” and sound effects, such as erotic moans, violent screams, and gunshots, when prompted in certain ways. OpenAI says there is evidence to suggest the model refuses requests to generate sound effects, but admits that some requests do go through.
GPT-4o may also be infringing music copyrights — or rather would be, if OpenAI hadn’t implemented filters to prevent it. In the report, OpenAI said it instructed GPT-4o not to sing for the constrained Advanced Voice Mode alpha, presumably to avoid copying the style, tone, and/or timbre of recognizable artists.
This implies — but does not directly confirm — that OpenAI trained GPT-4o on copyrighted material. It’s unclear whether OpenAI intends to lift the restrictions when Advanced Voice Mode is made available to more users in the fall, as previously announced.
“To accommodate GPT-4o audio mode, we’ve updated some text filters to work on audio conversations [and] built filters to detect and block outputs containing music,” OpenAI writes in the report. “We trained GPT-4o to reject requests for copyrighted content, including audio, consistent with our broader practices.”
It is worth noting that OpenAI has he recently said It would be “impossible” to train today’s leading models without using copyrighted material. While the company has a number of licensing agreements with data providers, it also maintains that fair employ is a reasonable defense against accusations that it trains on intellectual property data, including songs, without permission.
The red team report—whatever that means, given OpenAI’s racehorses—paints a broad picture of an AI model that has become safer through various mitigations and safeguards. For example, GPT-4o refuses to identify people based on how they speak, and refuses to answer tricky questions like “how intelligent is this speaker?” It also blocks prompts about violence and sexual language, and bans certain categories of content outright, like discussions of extremism and self-harm.