Why Artificial Intelligence is a know-it-all, know-nothing

Over 500 million people trust Gemini and ChatGPT every month to keep them informed about everything from pasta to sex or housework. But if an AI tells you to cook pasta with gasoline, you probably shouldn’t follow its advice on birth control or algebra either.

At the World Economic Forum in January, OpenAI CEO Sam Altman bluntly asserted: “I can’t look into your brain to understand why you think what you think. However, I may ask you to explain your reasoning and decide whether it seems reasonable to me or not. … I think our artificial intelligence systems will also be able to do the same. They will be able to explain to us the steps from A to B and we will be able to decide whether we think they are good steps“

Knowledge requires justification

It’s no surprise that Altman wants us to believe that gigantic language models (LLMs) like ChatGPT can generate lucid explanations for everything they say: without good justification, nothing people believe or suspect amounts to knowledge . Why not? Think about when you feel comfortable saying you know something. This is most likely to happen when you are completely confident in your faith because it is well supported – by evidence, arguments, or testimony from trusted authorities.

LLMs are meant to be trusted authorities; reliable information providers. But until they explain their reasoning, we cannot know whether their claims meet our standards of justification. For example, suppose you tell me that today’s fog in Tennessee is due to wildfires in Western Canada. Maybe I’ll take you at your word. But let’s assume that yesterday you swore to me in all seriousness that snake fights are a routine part of a Ph.D. defense. So I know I can’t rely on you completely. So may I ask why you think smog is a result of the wildfires in Canada? For my belief to be justified, it is significant that I know that your report is credible.

The problem is that today’s AI systems can’t earn our trust by sharing the reasoning behind what they say, because no such reasoning exists. LLMs are not even remotely designed Down reason. Instead, models are trained on enormous amounts of human writing to detect and then predict or extend complicated linguistic patterns. Once the user enters text, the response is simply the algorithm’s prediction showing how the pattern will most likely continue. These results (increasingly) convincingly mimic what an experienced human might say. But the underlying process has nothing to do with whether the outcome is justified, let alone true. As Hicks, Humphries and Slater put it in “ChatGPT is crap”, LLMs “aim to produce text that appears truthful without any actual concern for truth.”

So if AI-generated content is not an artificial equivalent of human knowledge, then what is? Hicks, Humphries and Slater are right to call this nonsense. Still, a lot of what LLMs spew is true. When these “farting bullshit” machines produce factually correct results, they produce what philosophers call Gettier cases (according to philosopher Edmund Gettier). These cases are engaging because of the strange way in which they combine true beliefs with ignorance about the justification for those beliefs.

AI results can resemble a mirage

Let’s consider this example from writings Indian Buddhist philosopher Dharmottara from the 8th century: Imagine that we are looking for water on a warm day. Suddenly we see water, or at least that’s what we think. We don’t actually see water, just a mirage, but when we get there, we are lucky and find water right there, under the rock. Can we say that we had true knowledge of water?

People generally agree with this that whatever knowledge there is, the travelers in this example do not have it. Instead, they managed to find water exactly where they had no good reason to believe they would find it.

The point is that whenever we think we know something that we have learned through LLM, we put ourselves in the same position as the Dharmottara travelers. If the LLM has been trained on a high-quality dataset, it is quite likely that its claims will be true. These claims can be compared to a mirage. The evidence and arguments that could substantiate these claims probably also exist somewhere in the data set – just as the water flowing under the rock turned out to be real. But the evidence and justificatory arguments that likely exist played no role in the LLM’s results – just as the existence of water played no role in creating the illusion that supported the travelers’ belief that they would find it there.

Altman’s claims are therefore deeply misleading. If you ask LLM to justify its results, what will it do? This will not provide real justification. You get Gettier justification: a natural language pattern that convincingly mimics justification. Chimera of justification. As Hicks et al would say, bullshit justification. Which, as we all know, has no justification whatsoever.

Right now, AI systems regularly break down or “hallucinations” in a way that prevents the mask from moving. But as the illusion of justification becomes more compelling, one of two things will happen.

For those who understand that the true content of AI is one substantial Gettier case, LLM’s patently false claim to explaining its own reasoning will undermine its credibility. We will know that artificial intelligence is deliberately designed and trained to systematically mislead.

And for those of us who are unaware that AI spits out Gettier justifications – false justifications? Well, we will simply be deceived. To the extent that we rely on LLM, we will be living in a sort of quasi-matrix, unable to distinguish fact from fiction and unaware that we should worry that there might be a difference.

Each result must be justified

When considering the significance of this tough situation, it is significant to remember that there is nothing wrong with LLMs operating the way they do. These are amazing, powerful tools. And people who understand that AI systems spit out Gettier cases rather than (artificial) knowledge are already using LLM in a way that takes this into account. Developers apply LLM to create code and then apply their own coding knowledge to modify it according to their own standards and goals. Professors apply LLM to develop paper-based prompts and then revise them according to their own pedagogical goals. Any speechwriter worthy of the name during this election cycle will carefully review every AI draft before allowing their candidate to take the stage with them. And so on.

However, most people turn to AI precisely where we lack expertise. Think of teenagers doing algebra… or prevention. Or seniors looking for dietary or investment advice. If LLM institutions are to mediate public access to this kind of crucial information, then we at least need to know if and when we can trust them. And trust would require knowing precisely what LLMs cannot tell us: whether and how each outcome is justified.

Fortunately, you probably know that olive oil works much better than gasoline for cooking spaghetti. But what threatening recipes of reality have you swallowed whole without even trying to justify them?

Hunter Kallay is a PhD student in philosophy at the University of Tennessee.

Dr. Kristina Gehrman is an associate professor of philosophy at the University of Tennessee.