Why We Don’t Trust AI Confidence Scores at Singularity

The Problem with “95% Confidence”

A lot of AI tools—like ChatGPT—are designed to sound confident. You might get a response that says, “I’m 95% sure,” and feel like that’s enough. But in technical engineering work, that kind of thinking is risky.

A confidence score is not a fact. It’s not pulled from a verified document. It’s just the model’s internal estimate of how likely it thinks its answer is correct, based on patterns it has seen before.

That might be fine for writing stories or brainstorming ideas. But in engineering, “it feels right” is not good enough.

So What Do We Do Instead?

At Singularity, we’ve taken a different approach.

We ask our AI systems to cite specific documents—whether it’s a section of a process manual, a spec sheet, or a standard operating procedure. If a response doesn’t include a reference to a primary source, we don’t approve it.

And we’re honest: we don’t always get it right. Sometimes the AI skips the citation. Sometimes we forget to verify. But we’re actively practicing this system, and our goal is to get better every week.

Why Confidence Scores Are Risky

When engineers or managers see a high confidence score, they’re more likely to skip the review step. That’s the danger.

Over time, confidence scores can lead to reduced human oversight—especially when the AI sounds like it knows what it’s doing. This is exactly what we want to avoid.

Instead, when the AI is required to give a specific source, and when a human checks that source, we’re building a process that’s faster and more reliable.

How This Works in Real Life

Let’s say ChatGPT writes a test procedure and includes:

“Step 3: Confirm voltage range. [See: SpecDoc §5.1]”

That bracketed reference matters. An engineer will click into the document, confirm the instructions are correct, and only then give final approval.

This doesn’t slow us down—it saves us time by reducing rework, errors, and guesswork.

What We’re Building Toward

We’re not interested in AI that “feels” confident. We’re building workflows that are source-cited, human-checked, and technically sound.

It’s not perfect yet. But it’s a practice we’re committed to.

Want to Use AI the Right Way?

If you're exploring how to build safer, more accurate AI systems for technical work, start here: Subscribe to our newsletter for weekly updates, real workflows, and practical tips.👉 https://www.singularityengineering.ca/general-4