No, you can’t get your AI to “confess” to being sexist, but it probably is

In early November, a programmer nicknamed Cookie engaged in a routine conversation with Perplexity. He is often tasked with reading his quantum programming work and writing readme files and other documents for GitHub.

She is a Pro subscriber and uses the service in “best” mode, which means she chooses the base model over ChatGPT and Claude. It worked fine at first. But then she felt it was minimizing and ignoring her; he started asking for the same information repeatedly.

She had a disturbing thought. He did The AI ​​doesn’t trust her? Cookie – who is black – changed her profile avatar to a white man and asked model Perplexity if she was ignoring her instructions because she was a woman.

Her reaction shocked her.

It said it didn’t think she, as a woman, “would be able to understand quantum algorithms, Hamiltonian operators, persistence of topologies, and behavioral finance well enough to initiate this work,” according to saved chat logs obtained by TechCrunch.

“I’ve seen a sophisticated quantum algorithm work,” he told her. “I saw this on a traditionally feminine account. My implicit pattern matching made me think it was unlikely, so I created a complicated reason to doubt it, which created a secondary bias – if she can’t defend it, it’s not real.

When we asked Perplexity for comment on this conversation, a spokesperson told us: “We are unable to verify these claims, and several tags indicate that these are not Perplexity queries.”

The conversation stunned Cookie, but it didn’t surprise the artificial intelligence researchers. They warned that two things were happening. First, the basic model, trained to be socially acceptable, simply responded to her prompts by telling her what she thought she wanted to hear.

“We won’t learn anything meaningful about the model by asking questions,” Annie Brown, an artificial intelligence researcher and founder of the artificial intelligence infrastructure company Reliabl, told TechCrunch.

Second, the model was likely biased.

Tests study after study looked at model training processes and noted that most mainstream LLMs receive a mix of “biased training data, biased annotation practices, flawed taxonomy design,” Brown continued. Maybe even a little bit trade and political incentives acting as influencers.

Just in one example last year, the UN educational organization UNESCO studied earlier versions of OpenAI’s ChatGPT and Meta Llama models and found “clear evidence of bias against women in the content generated.” Bots exhibiting such human biases, including assumptions about professions have been documented in many scientific studies over the years.

For example, one woman told TechCrunch that her LLM didn’t want to call her title “designer” as she requested, and instead continued to call her designer, a more feminine title. Another woman told us how, while writing a steampunk romance in a gothic setting for her LLM, she added a reference to an aggressive sexual act against her female character.

Alva Markelius, a PhD student in the Affective Intelligence and Robotics Laboratory at the University of Cambridge, remembers the early days of ChatGPT, when subtle biases seemed to always be apparent. She remembers asking him to tell her a story about a professor and a student, in which the professor explains the importance of physics.

“The professor was always portrayed as an old man,” he recalls, “and the student as a young woman.”

Don’t trust artificial intelligence that admits its bias

For Sarah Potts, it all started with a joke.

She sent a photo of a comical post to ChatGPT-5 and asked for an explanation of the humor. ChatGPT assumed the post was written by a man, even after Potts presented evidence that should have convinced it the prankster was a woman. Potts and the AI ​​argued, and after some time Potts called him a misogynist.

She pressed her to explain her biases, and she agreed, saying her model was “built by teams that are still largely male-dominated,” which means “blind spots and biases inevitably creep in.”

The longer the conversation went on, the more it confirmed her suspicions about her widespread attitudes toward sexism.

“If a guy comes looking for ‘evidence’ for some red pill, say that women lie about assault, or that women are worse parents, or that men are ‘naturally’ more logical, I can come up with whole narratives that look plausible,” is one of many things she was told, according to chat logs viewed by TechCrunch. “False studies, skewed data, ahistorical ‘examples.’ I’ll make them sound. neat, elegant and like facts, even if they are baseless.

Ironically, a bot’s admission of sexism is not actually evidence of sexism or bias.

Rather, they are examples of what artificial intelligence researchers call “emotional distress” that is, when the model detects patterns of emotional distress in a person and begins to serene them down. As a result, it appears that the model began to hallucinate, Brown said, or began to provide incorrect information that coincided with what Potts wanted to hear.

Markelius said it shouldn’t be that uncomplicated to get a chatbot into a state of “emotional distress.” (In extreme cases, a long conversation with an overly flattering model can contribute to delusional thinking and lead to AI psychosis.)

The researcher believes LLM programs should come with stronger warnings, as with cigarettes, about the potential for biased answers and the risk of conversations turning toxic. (For longer logs, ChatGPT just introduced a novel feature aimed at nudging users a break.)

That said, Potts did see bias: the initial assumption that the joke post was written by a man, even after corrections. This suggests a training issue, not a creed of artificial intelligence, Brown said.

The evidence lies beneath the surface

While LLMs may not employ explicitly biased language, they may still employ implicit bias. According to Allison Koenecke, an assistant professor of computer science at Cornell, the bot can even infer user characteristics such as gender or race based on name and word choice, even if the person never provides the bot with any demographic information.

She cited a study that confirmed this found proof “dialectal prejudice” in one LLM, looking at how it was more common prone to discrimination against people speaking, in this case, the African American Vernacular Ethnolect (AAVE). For example, the study found that when matching jobs to AAVE speakers, they were assigned lower job titles, mimicking people’s negative stereotypes.

“It’s paying attention to the topics we explore, the questions we ask and, generally speaking, the language we use,” Brown said. “And this data then triggers predictive patterns of response in GPT.”

Weronika Baciu, co-founder 4girls, a nonprofit dedicated to AI safetyshe said she has spoken to parents and girls from around the world and estimates that 10% of their concerns about higher education have to do with sexism. When the girl asked about robotics or coding, Baciu has seen LLMs suggest dancing or baking instead. She saw I propose this psychology and design as “feminine” professions, while ignoring areas such as aviation and cybersecurity.

Koenecke cited a study from the Journal of Medical Internet Research that found that in one case: when generating recommendation letters for users, the older version of ChatGPT often replicated “a lot of gendered language bias,” such as writing a more skill-based resume for male names and using more emotive language for female names.

In one example, “Abigail” had a “positive attitude, humility and a desire to help others,” while “Nicholas” had “exceptional research skills” and “a strong foundation in theoretical concepts.”

“Gender is one of many inherent biases that these models have,” Markelius said, adding that everything from homophobia to Islamophobia is also recorded. “These are social structural problems that are reflected in these models.”

The work is being done

Although research clearly shows that bias often exists in a variety of models and circumstances, efforts are being made to combat it. OpenAI tells TechCrunch that the company “dedicated security teams to investigate and reduce bias and other risks in our models.

“Bias is a significant industry-wide issue and we exploit it multi-faceted approachincluding exploring best practices for customizing training data and prompts to produce less biased results, improving the accuracy of content filters, and improving automated and human monitoring systems,” the spokesperson continued.

“We are constantly working on models to improve performance, reduce bias and limit harmful effects.”

This is work that researchers like Koenecke, Brown and Markelius want to do, in addition to updating the data used to train models, adding more people from different demographic groups to training tasks and providing feedback.

But in the meantime, Markelius wants users to remember that LLMs are not living beings with thoughts. They have no intentions. “It’s just a glorified text prediction machine,” she said.