The Confidence Trap: (Segway) Why Developers Can’t Tell When AI Is Wrong
Part 3 of “The 2026 Supply Chain Reckoning” : a series examining what Sonatype’s latest report and other research tell us about the state of software supply chain security.
Your boss calls you on a Friday afternoon. He’s read all the available data, he tells you with absolute confidence, and he’s decided that migrating from Spring Boot 3.5 to 4.0 will be straightforward. Wants it done over the weekend. He sounds certain. He’s reviewed everything.
Do you challenge his assumptions, or do you agree?
Most of us would push back. We’d ask which data. We’d point out the breaking changes. We’d want to see the evidence behind the confidence.
Now replace your boss with Copilot. It suggests upgrading a dependency. The recommendation is fluent, well-formatted, and delivered with the same calm certainty. No hedging. No caveats. Do you check, or do you accept?
In the last piece, I showed you that AI coding tools hallucinate nearly 30% of dependency recommendations. Somewhere between one in three and one in four. The model invents package versions, recommends compromised libraries, and in 345 cases actively made security worse.
The obvious follow-up question is: why does anyone trust them?
The answer isn’t laziness. It’s psychology. And it’s the same psychology that explains why your boss got promoted in the first place.
We’re Wired to Trust Confidence
Cameron Anderson, a researcher at UC Berkeley, spent years studying why overconfident people consistently rise to the top of groups. His status-enhancement theory landed on an interesting finding: overconfidence works.
In six separate experiments, people who believed they were better than others gained higher social status in their groups, even when they demonstrably weren’t.
In one study, participants took a general knowledge test that included fake terms. The people who claimed to recognise the most fake terms: the most objectively overconfident in the room: ended up with the highest social status by the end of the semester.
They weren’t penalised for being wrong. They were rewarded for sounding certain.
The mechanism is something called the “competence signal.” In ambiguous environments, people can’t directly assess each other’s actual skill. So they rely on proxies: who speaks first, who sounds certain, who participates most. Overconfident people naturally display all of these signals.
The group reads confidence as competence.
The Incompetent Leader Problem
Organisational psychologist Tomas Chamorro-Premuzic has a blunter conclusion: there is virtually no relationship between how good people think they are at something and how good they actually are.
His research into leadership selection found that organisations consistently promote individuals who display overconfidence, narcissism, and charisma, the very traits least correlated with effective leadership.
A KPMG report found that 67% of UK CEOs trust their intuition over objective data. That’s not a quirk. It’s a human-systemic bias. Confidence feels like information, even when it’s not.
Chamorro-Premuzic calls this the confidence-competence gap. The people who reach the top are disproportionately those who sound like they know what they’re doing. The people who actually do tend to be more cautious, more hedging, more aware of what they don’t know. And that caution reads as weakness.
Now Map That Onto Your IDE
Large language models are optimised for the exact signals that trigger our trust. Solid, fluent language. An authoritative tone. Precise-sounding numbers. Elaborate reasoning. An LLM never says “I’m not sure” in the way a cautious human expert would. It presents every answer with the same polished certainty, whether it’s correct or fabricated. Which we’ve all encountered.
Psychologists call this the fluency heuristic. When information is easy to process, the brain treats it as more likely to be true. A model that speaks clearly and confidently triggers the same mental shortcut as a charismatic leader in a boardroom.
We substitute how the answer sounds for whether the answer is right.
Research from Carnegie Mellon found that LLMs hallucinated in 69% to 88% of legal queries. Yet the authoritative tone consistently misled the researchers evaluating the output. Even people trained to spot errors were fooled by the delivery.
The Automation Tax on Thinking
This bit is scary - take a moment to see if you’re affected. Microsoft surveyed knowledge workers and found that the more they use AI tools, the less critical effort they report applying. They described it as the “irony of automation”: by handling the routine thinking, the tool erodes the skill you need to catch it when it’s wrong.
This maps directly onto how developers consume AI-generated code or even dependency recommendations. The model suggests a solution, a particular version. It looks right. The developer accepts it. Over time, the habit of not checking becomes the default. The developer isn’t deliberately being careless, but because the tool has trained them out of the verification reflex, the effect is the same.
Even when participants were explicitly told that an AI system lacked crucial information, they still trusted it more after viewing its reasoning process.
The appearance of logic was more persuasive than the acknowledged absence of facts.
So What Does This Mean for the Supply Chain?
In Article 2, I showed you that GPT-5 expressed high confidence in only 4% of its dependency upgrade recommendations, and was 98% accurate in those cases. The other 96% came with lower confidence and hallucination rates up to 47%.
But here’s the blue-pill effect: the developer experience doesn’t surface that confidence score. Every recommendation arrives in the same authoritative tone. And our brains, shaped by the same social instincts that govern every meeting room and hiring panel, accept it.
This is why hallucination rates approaching 30% persist. The psychological architecture of human trust doesn’t distinguish between sounding right and being right. And AI is the most fluent, most confident, most tireless bullshitter we’ve ever built.
Making the Model Show Its Working
You can’t fix human psychology. But you can change how you use the tools.
The core problem is that AI delivers every answer in the same tone. Confident and wrong looks identical to confident and right. There are habits that can help shift that dynamic. Worth trying, not guaranteed to work. (aka - works for me)
1: Ask it what it doesn’t know. Before accepting a recommendation, ask the model: “What are you uncertain about in this suggestion?” or “What assumptions are you making?” Hopefully, the model will list the caveats they’d otherwise skip. It won’t catch everything, but it forces the model out of its default authoritative, confident bullshit mode.
2: Ask for alternatives. If a model can only suggest one approach, that’s a warning flag. Ask: “What other approaches could work here? What are the trade-offs?” When the model generates multiple options and explains the differences, you get a much better sense of where it’s reasoning and where it’s guessing. Even if the model confidently recommends three mutually contradictory options, it’s still showing you something useful.
3: Give it your actual context. The less a model has to infer, the less it hallucinates. Paste your dependency tree, your version constraints, your build errors. A prompt that says “upgrade Jackson” invites hallucination. A prompt that includes your pom.xml and says “find a compatible Jackson version for this dependency tree”, gives the model something to work with.
4: Verify the reasoning, not just the output. Ask “why this?” If the answer is vague (”it’s the latest stable release”) or circular (”it’s the recommended approach”), the model is probably hallucinating. A good recommendation comes with a specific rationale: compatibility with another dependency, a CVE fix, a feature you need…
5: Treat the first answer as a draft. The human expert heuristic hits hardest on the first pass. The suggestion looks good; it makes sense in your head, and your instinct is to accept it. Train yourself to treat every AI recommendation as a starting point that needs a second look.
None of this eliminates the confidence trap. But it shifts the dynamic. Instead of a model that delivers and a developer who accepts, you get a conversation where the model has to justify itself. That’s a much harder environment for hallucinations to survive in.
Whether you’re evaluating a candidate for CTO or a dependency recommendation from an LLM, the imperative is the same: look past the confidence of the delivery and demand the evidence.
Next in the series: “North Korea Runs a Software Factory (And It Ships to npm)” — how state-sponsored attackers industrialised the supply chain, and why their targeting is more deliberate than you think.
If you’re finding this series useful, subscribe to the rest. We’re only getting started.
Sources
Anderson, C., Brion, S., Moore, D.A., & Kennedy, J.A., When Overconfidence Is Revealed to Others: Testing the Status-Enhancement Theory — the primary research on status-enhancement theory, demonstrating that overconfident individuals gain higher social status even when their incompetence is revealed.
Anderson, C. & Kilduff, G.J., Overconfidence and the Attainment of Status in Groups — six experiments showing that overconfident individuals consistently achieve higher social status through “competence signals” untethered from actual ability.
Chamorro-Premuzic, T., Why Do So Many Incompetent Men Become Leaders? — TED Ideas article summarising the confidence-competence gap in leadership selection, including the systemic bias toward overconfidence and narcissism.
Carnegie Mellon University, AI Chatbots Remain Confident — Even When They’re Wrong — research finding that LLMs hallucinated in 69–88% of legal queries while maintaining authoritative tone that misled evaluators.
Microsoft Research, The Impact of Generative AI on Critical Thinking — survey of knowledge workers documenting self-reported reductions in cognitive effort and independent problem-solving when using generative AI tools.
UC Irvine, Study Finds Mismatch Between Human Perception and Reliability of AI-Assisted Language Tools — research on the “length bias” in AI trust, showing users exhibit higher confidence in longer AI explanations regardless of accuracy.
Harvard Misinformation Review, New Sources of Inaccuracy? A Conceptual Framework for Studying AI Hallucinations — framework for understanding the “demand side” of AI hallucination, including how authoritative presentation invites shallow processing by users.
Sonatype, 2026 State of the Software Supply Chain — source for the GPT-5 confidence-level data referenced from Article 2 (3.68% high confidence, 47.38% hallucination at low confidence).
KPMG, 2018 CEO Outlook — 67% of UK CEOs trust their intuition over objective data.
Revealing AI Reasoning Increases Trust but Crowds Out Unique Human Knowledge — research showing that participants trusted AI more after viewing its reasoning process, even when explicitly told the system lacked crucial information.