How accuracy cues shape judgments of chatbot answers
When Should I Trust This? Designing and Evaluating Factuality Indicators for LLM-based Chatbots · arXiv:2610.32883
In plain words
A chatbot can sound sure and still give a wrong answer. This paper studies how people understand two clues meant to help them judge answers: confidence information and sources they can check.
The researchers interviewed users, surveyed U.S. adults, and tested different clues in chatbot screenshots.
The clues made small differences to some first impressions, but after people learned whether the answer was correct, the researchers no longer detected differences in trust or overall feeling between clue types.
The research
When a chatbot answer sounds plausible, what helps someone decide whether to trust it: a confidence signal, sources to inspect, or learning whether the answer is correct? The paper investigates how people understand these clues and whether they change judgments of chatbot responses.
The study combines interviews, a survey and a randomized screenshot experiment to test how confidence and source cues shape judgments of chatbot answers.
To judge a chatbot answer, people may need to understand what a confidence signal means and whether its sources are worth checking. The authors first studied how users interpret these cues and report verifying answers, then tested whether different cues changed ratings of chatbot responses.
In a randomized experiment, U. S. adults rated one screenshot before and after learning whether its answer was correct. Indicator differences were small initially and no longer detected for trust or sentiment after disclosure; correctness had a much larger association with those later ratings. The results suggest cues can shape first impressions, but do not establish changes in behavior.
The experiment measured ratings of one static chatbot screenshot per person, not source checking, reliance, or real decisions. All studies recruited U.S. participants who already used generative AI, so the findings may not extend to other populations or everyday interactions.
What the evidence shows
- The judgment problem
This mixed-methods empirical study asks how people judge chatbot factuality and whether confidence or source cues help them assess an answer. The problem is practical within information seeking: a fluent response can still be inaccurate, while the meaning of a confidence signal may not be clear.
The authors examine user understanding and reported verification before testing how interface cues affect judgments.
- What users make of cues
In interviews, eight U. S. participants gave conflicting interpretations of AI confidence, including whether it means an answer is probably correct or has been checked. They valued sources as evidence they could inspect, while raising concerns about credibility and the work verification takes.
In a separate survey, 94.7% of 1,000 U. adult AI users reported verifying responses some, but not all, of the time. Looking things up and reviewing chatbot sources were common; asking for a confidence rating was among the least common approaches.
- The screenshot comparison
The experiment randomly assigned 5,151 U. S. adult AI users to see one chatbot screenshot with one of six indicator setups, across gardening, mathematics, or medical topics and correct or incorrect answers. Participants rated perceived accuracy and information sufficiency, plus trust and sentiment before and after correctness was revealed.
Before disclosure, some indicators received higher ratings than others, but those differences were small. After disclosure, the researchers detected no indicator differences in trust or sentiment.
- What shaped the ratings
Knowing whether an answer was correct had a much larger association with post-disclosure trust and sentiment than indicator type. Topic also mattered: participants rated medical responses lower than gardening or mathematical responses on perceived accuracy, trust, and sentiment.
The findings suggest that cues can influence some first impressions in this test, but judgments also reflect correctness and topic. Because participants rated a single screenshot, the study does not establish effects on actual verification or decisions.
Deep
- Original paper authors
- Greta Warren, Jessica Heal, Sila Eser, Denise M. Leto, Christine M. Lehane
- Submitted to arXiv
- License
- CC BY 4.0