Last week, OpenAI released its long-awaited search engine ChatGPT Search to take on Google. The industry has been preparing for this moment for months, prompting Google to inject AI-generated responses into its core product earlier this year, causing embarrassing hallucinations in the process. This misfortune led many people to believe that the OpenAI search engine would truly be a “Google killer.”
But after using ChatGPT search as my default search engine (you can do it too using OpenAI extension) for about a day, I quickly went back to Google. OpenAI’s search engine was impressive in some ways and provided a glimpse of what an AI search interface might one day look like. But for now, it’s still too impractical to utilize as my daily driver.
The ChatGPT search engine was sometimes useful for finding real-time answers to questions that I would otherwise have had to sift through multiple SEO-optimized ads and articles. Like other products in this AI search category, such as Perplexity and You.com, it presents concise answers in a nice format: on the right side there are links to information sources; headlines and brief fragments allow you to quickly check the reality of text generated by artificial intelligence.
However, it often just seemed impractical for everyday utilize.
In its current form, the ChatGPT search engine is not reliable for what people most utilize Google for: brief navigational queries. Queries shorter than four words most searches on Google; often it’s just a few keywords that lead to the right website, often one that the user knows but doesn’t want to bother typing. These are the types of searches that most people don’t realize they do all day long, and Google usually handles them very well.
I’m talking about “Celtics score,” “cotton socks,” “library hours,” “San Francisco weather,” “coffee shops near me,” and other queries that make Google the gateway to the Internet for billions of people.
My test run with the ChatGPT search engine was quite frustrating at times and made me realize how many keyword searches I do per day. I couldn’t reliably find information using brief queries, and for the first time in years I truly missed Google Search.
Don’t get me wrong: Google’s quality has declined over the last decade, largely due to a deluge of advertising, SEO, and questionable AI summaries. Despite this, I had to open Google in a separate window during the test because the ChatGPT search engine couldn’t get the correct answer or website.
Who will win: ChatGPT search or brief queries?
I entered “Nuggets Score” to see how the live NBA game between the Denver Nuggets and the Minnesota Timberwolves is going. ChatGPT told me the Nuggets were winning even though they were actually losing, and showed the Timberwolves’ score was 10 points lower than it actually was, according to Google results at the same time.
Another time I tried “today’s earnings” to check out companies reporting quarterly earnings that could impact stock prices on Friday. ChatGPT told me that Apple and Amazon released their results on Friday, even though both companies had already released them the day before. In other words, he hallucinated and made up information.
In another test, I typed in the name of the CTO to find his contact information. ChatGPT showed me a summary of the person’s Facebook profile and hallucinated me with a link to their LinkedIn page, which resulted in an error message when clicked.
Another time I typed “loose jeans” hoping to shop. The ChatGPT search engine basically described to me what baggy jeans were (a definition I didn’t need) and told me to go to Amazon.com for a nice pair.
I could go on, but you get the point. Broken links, hallucinations, and random responses defined my first day using ChatGPT search.
Perhaps a “Google killer” one day, but not today
This was not a petty start for OpenAI. Altman himself praised this feature for being “really good,” even though he is known for downplaying his startup’s AI capabilities. The reason why it’s different this time may have something to do with the search engine being one of the largest companies on the Internet, and the OpenAI version could pose a real threat to its biggest competitor, Google.
To be straightforward, Google Search is a 25-year-old product and ChatGPT Search is brand modern. In a blog post, OpenAI says it plans to improve this feature based on user feedback in the coming months, and it seems more than likely that this could be a significant area of investment for the startup.
To its credit, ChatGPT Search is quite good at answering long, written research questions. Something like, “What is the most diverse American professional sports league?” This isn’t a question that can be easily answered on Google, but the ChatGPT search engine does a pretty good job of sifting through multiple sites and getting a decent answer in just a few seconds. (Perplexity also does a pretty good job of answering these questions, and the search engine has been around for over a year.)
Compared to the established version of ChatGPT that already had web access, the search function seems to be a better interface for browsing the web. There are now clearer links to where ChatGPT sources its news – for news, ChatGPT will utilize media companies with which it has all its licensing agreements.
The problem is that most Google searches don’t have questions this long. To truly replace Google, OpenAI needs to improve these more practical, brief searches that people already perform throughout the day.
OpenAI is not ashamed of the fact that ChatGPT Search struggles with brief queries.
“With ChatGPT search, we’ve seen users start asking questions in a more natural way than they have in the past with other search tools,” OpenAI spokesman Niko Felix said in a statement emailed to TechCrunch. “At the same time, internet navigation queries are quite common and tend to be brief. We plan to improve our handling of these types of queries over time.
That said, these brief keyword-based queries have made Google indispensable, and until OpenAI gets them right, Google will continue to be a mainstay for many people.
There are several reasons why OpenAI may have problems with these brief queries. Firstly, ChatGPT is based on Microsoft Bing, which is widely considered to be an inferior engine compared to Google.
The second reason is that huge language models may generally not fit these brief prompts. LLM companies usually need fully written down questions to get effective answers because it helps create a solid statistical formula to fill out: little do they know that people searching for “cotton socks” are almost all trying to buy them, not find out about the origins of the garment. You may need to re-prompt – run brief queries in LLM as a longer question – before the ChatGPT search engine can perform such searches well.
Although OpenAI has only just released its search product, Perplexity’s own AI search tool already handles 100 million queries per week. Embarrassment is also touted as a “Google killer”, but it encounters the same problems for brief queries.
Aravind Srinivas, CEO of Perplexity, discussed with TechCrunch Disrupt how people utilize his product differently compared to Google Search: “The average number of words in a Google query ranges from two to three. In Perplexity it’s about 10 to 11 words. So it’s clear that at Perplexity people are more likely to come in and be able to ask questions directly. On the other hand, in Google, you type in a few keywords to immediately go to a specific link.
I think the fact that people aren’t using these products to navigate the internet is a bigger problem than OpenAI or Perplexity allow. This means that the ChatGPT search engine and confusion do not replace Google search in the task at which it is best: navigating the Internet.
Instead, these AI products fill a modern niche by extracting information that is buried in established search. Don’t get me wrong, that in itself is valuable.
Both OpenAI and Perplexity say they will work to improve these brief queries. I don’t think any of these products will fully replace Google by then. If OpenAI wants to replace the front door to the Internet, it needs to create a better one.