New Algorithm Detects AI's Inaccurate Answers to Improve Reliability

A new algorithm, combined with a dose of humility, might help generative AI address a persistent issue: confident but inaccurate answers.

AI Error Risks

Errors in AI responses are particularly risky when people rely heavily on chatbots and other tools for critical information such as medical advice and legal precedents. A recent Wired investigation revealed that the AI-powered search engine Perplexity produces inaccurate answers.

Sebastian Farquhar, a senior research fellow in the computer science department at the University of Oxford, explains that today's AI models make various types of mistakes. These errors, often collectively termed "hallucinations," encompass a broad range of inaccuracies, making the term somewhat ineffective.

New Detection Method

Farquhar and his colleagues at Oxford have developed a method for detecting "arbitrary and incorrect answers," which they call confabulations. This method, detailed in a paper published in Nature, computes uncertainty at the level of meaning rather than specific word sequences.

The process involves asking a chatbot a question multiple times, such as "Where is the Eiffel Tower?" A separate large language model (LLM) then groups the chatbot's responses based on their meaning (e.g., "It's Paris," "Paris," "France's capital Paris," "Rome," "It's Rome," "Berlin"). The team calculates the "semantic entropy" for each group—a measure of response similarity. If the responses vary significantly (e.g., Paris, Rome, Berlin), the model is likely confabulating.

Effectiveness and Limitations

This new approach can identify confabulations about 79% of the time, compared to 69% for methods assessing word similarity and similar performance by two other techniques. However, it only detects inconsistent errors, not those arising from biased or erroneous training data. Additionally, it requires five to ten times more computing power than typical chatbot interactions.

"For some applications, that would be a problem, and for some applications, that's totally worth it," says Farquhar, who is also a senior research scientist at Google DeepMind. "Developing approaches to detect confabulations is a big step in the right direction, but we still need to be cautious before accepting outputs as correct," adds Jenn Wortman Vaughan, a senior principal researcher at Microsoft Research.

Communicating Uncertainty

Vaughan and other researchers are exploring ways for AI systems to convey uncertainty in their answers to help users set appropriate expectations. In a new study, Vaughan's team examined how people perceive a model's expression of uncertainty when a fictional "LLM-infused" search engine answered a medical question. They found that AI responses with first-person expressions of uncertainty ("I'm not sure, but...") led users to be less confident in the AI's answers.

The study suggests that natural language expressions of uncertainty may reduce overreliance on LLMs, though the precise wording is crucial. Vaughan emphasizes that conveying uncertainty must focus on user needs: "How do we empower them to make the best choices about how much to rely on the system and what information to trust? We can't answer these types of questions with technical solutions alone."

Ongoing Challenges

The most advanced chatbots from OpenAI, Meta, Google, and others "hallucinate" at rates between 2.5% and 5% when summarizing documents. While errors in earlier versions may not occur in the latest ones, the problem persists in different forms. Providing extra training data can improve accuracy for known issues, but users often push AI beyond its trained data, increasing the risk of errors and fabrications.

"In some contexts, hallucination is a factuality problem," Farquhar explains. "In other contexts, it is creativity and finding new ways of expressing imaginary ideas. If you're trying to generate fiction, for example, the same thing that's causing the hallucinations might be genuinely something you want."