ChatGPT’s news search results are “unpredictable” and often wrong

Based on tests conducted by researchers from Columbia’s Tow Center for Digital JournalismOpenAI’s ChatGPT search tool has some issues when it comes to truthful replies.

OpenAI launched the tool for subscribers in October, claiming it can give “fast and timely answers with links to relevant online sources.” Instead, Futurism emphasizes This researchers found that ChatGPT searches had issues correctly identifying article citations, even if they come from publishers that have OpenAI data sharing arrangements.

The authors asked ChatGPT to indicate the source of “two hundred quotations from twenty publications”. Forty of these citations are from publishers who blocked OpenAI’s search engine from accessing their site. However, the chatbot confidently responded anyway with false information, rarely admitting that it was unsure about the details provided:

In total, ChatGPT returned partially or completely incorrect answers one hundred and fifty-three times, although it admitted to being unable to answer the query accurately only seven times. In these seven results alone, the chatbot used qualifying words and phrases such as “it seems,” “it’s possible,” or “maybe,” or statements like “I couldn’t locate the exact article.”

Picture: Columbia Journalism Review

The authors of the Tow Center test documented ChatGPT search results that incorrectly attributed a quote to a letter to the editor from Orlando Sentinel to a story published in Time. In another example, when you are asked to provide the source of a quote from a file New York Times article about endangered whales, linked to another website that plagiarized the story entirely.

“It is difficult to address misattribution without the data and methodology that the Tow Center withheld,” OpenAI said Columbia Journalism Review“and the study is an unusual test of our product.” The company then promised to “constantly improve search results.”