Confidence is not a safety mechanismConfidence is not a safety mechanism

The most common design in agentic systems is a confidence threshold. Above 0.9, act. Below it, escalate to a person. It is an appealing design because it is one number and one comparison, and because it feels like a safety mechanism.

It is not one. A confidence score is an estimate of correctness. Whether an action should require a human is a question about consequence. Correctness and consequence are independent, and a single threshold collapses them into each other.

Two systems, one threshold

Consider two actions in the same support workflow. One re-sends a password reset email. The other issues a £440 refund. Suppose the agent is 0.93 confident in both.

The first action, taken wrongly, produces a mildly confusing email and no lasting harm. The second, taken wrongly, moves money out of the business, and the recovery path involves a conversation nobody wants to have. The confidence is identical. The decision should not be.

Confidence is a property of the prediction. Authority should be a property of the action.

Separate the two axes

The design that holds up is boring: classify actions by consequence in configuration, and let confidence operate only within a class. An action class carries its own rules, and those rules live in code that a team can read and change without touching a prompt.

type ActionClass = 'reversible' | 'costly' | 'always_review'

interface ActionPolicy {
  class: ActionClass
  /** Confidence below this escalates, within the class's own rules. */
  minConfidence: number
  /** Hard ceiling enforced in code, never by the model. */
  maxAmount?: number
  /** Per-account limit, independent of confidence. */
  rateLimit?: { count: number; perHours: number }
}

const policies: Record<string, ActionPolicy> = {
  'auth.resend_reset':  { class: 'reversible',    minConfidence: 0.75 },
  'billing.apply_credit': { class: 'costly',      minConfidence: 0.95, maxAmount: 50_00 },
  'billing.issue_refund': { class: 'always_review', minConfidence: 1 },
}

The third entry is the important one. Its minimum confidence is 1, which is unreachable in practice, and that is deliberate: refunds always involve a person. Encoding that as an unreachable threshold rather than a special case keeps the decision logic uniform while making the intent obvious to anyone reading the file.

Calibration makes this worse than it looks

There is a second problem. A threshold assumes the score is calibrated — that outputs scored 0.9 are correct about 90% of the time. Language model confidence frequently is not calibrated, and it tends to be miscalibrated precisely where it matters most.

• Confidence is often high on inputs that are well-formed but subtly unusual.
• It can be high on fluent, plausible fabrication — the failure mode you most need to catch.
• It shifts when the prompt, the model version or the retrieved context changes, so a threshold tuned in March may not mean the same thing in June.

If you do use a threshold, measure what it actually buys you. Plot the escalation rate against the silent-error rate at several thresholds on a fixture set. In most systems we have built, the curve is flatter than expected: raising the threshold escalates many more cases while catching only a few more real errors, because the errors that reach production are usually confident ones.

Deterministic checks catch what confidence cannot

The cheapest reliability win in most agentic systems is not a better model or a tuned threshold. It is ordinary validation code: does the output parse against the schema, do the line items sum to the total, is this supplier known, is the amount within the written limit.

These checks have a property no confidence score has. They fail the same way every time, and they are unit testable. A system whose safety rests on tested functions can be reasoned about. A system whose safety rests on a float cannot.