How to set an LLM confidence threshold

Select a candidate threshold, freeze the policy and validate accepted-answer risk independently, with runnable code and sample-size limits.

By Standard Thinking · Updated

Set an LLM confidence threshold by measuring errors among the answers it would accept automatically. Use representative labelled requests to choose a candidate, freeze it, then validate it on independent data. A probability such as 0.9 is a model score; it is not evidence that your application’s answers will be correct 90% of the time.

Choose the score for the action

Standard One returns probabilities for the supplied options. For a choice question, threshold the selected option’s probability. Its separate confidence field measures one minus normalized entropy: a concentrated distribution can be confidently wrong, so that field is not interchangeable with probability of correctness.

For yes/no (noul), use the probability of the answer you act on: p(yes) for yes and 1 - p(yes) for no. For an ordinal score decision, define whether the action follows one level or a band, then use that level’s probability or the band’s summed probability. Keep this rule fixed during validation.

Select first, validate independently

  1. Draw and label requests representative of production, including easy cases, rare classes and out-of-scope inputs. Split them into selection and validation sets before examining results.
  2. On selection data, compare candidate thresholds by accepted fraction and accepted-answer errors. Choose a candidate that balances your error target and review capacity.
  3. Freeze the model, prompt, option descriptions, score rule and threshold. Evaluate that fixed policy once on validation data using an upper confidence bound.
  4. If it fails, keep review enabled. A revised threshold needs fresh independent validation or a method that corrects for repeated search; repeatedly tuning against the same validation set invalidates the simple confidence claim.

With 3 errors among 300 accepted answers, the observed error rate is 1.0%, but the one-sided 95% Clopper–Pearson upper bound is about 2.6%. With zero errors among 300, the bound is about 1.0%. Neither is a worst-case guarantee against changing traffic.

Badshah, Emami and Sajjad use this bound for selective judging. Their guarantee concerns fixed thresholds under independent, identically distributed sampling; they also discuss correction for threshold search. The independent-validation procedure here avoids treating a searched threshold’s selection-set bound as a 95% production guarantee.

A runnable selection and validation example

Install SciPy (python -m pip install scipy). Both CSV files contain probability,correct, where correctness is 1 or 0. Set the target and minimum accepted count before validation; 100 below is an illustrative minimum, not enough for every error target.

import csv
from math import isfinite
from scipy.stats import beta

TARGET, MIN_ACCEPTED = 0.03, 100
BARS = (0.5, 0.6, 0.7, 0.8, 0.9, 0.95)

def load(path):
    with open(path, newline="") as f:
        rows = [(float(r["probability"]), r["correct"])
                for r in csv.DictReader(f)]
    if not rows or any(not isfinite(p) or not 0 <= p <= 1
                       or ok not in ("0", "1") for p, ok in rows):
        raise ValueError("Need nonempty, valid labelled data")
    return [(p, ok == "1") for p, ok in rows]

def upper(errors, n):
    if n == 0 or errors == n:
        return 1.0
    return float(beta.ppf(0.95, errors + 1, n - errors))

def assess(rows, bar):
    accepted = [ok for p, ok in rows if p >= bar]
    n, errors = len(accepted), accepted.count(False)
    return n, errors, upper(errors, n)

selection = load("selection.csv")
candidates = []
for bar in BARS:
    n, errors, bound = assess(selection, bar)
    if n >= MIN_ACCEPTED and bound <= TARGET:
        candidates.append(bar)
if not candidates:
    raise SystemExit("No candidate; keep review enabled")
bar = min(candidates)  # freeze before reading validation.csv
validation = load("validation.csv")
n, errors, bound = assess(validation, bar)
passed = n >= MIN_ACCEPTED and bound <= TARGET
print(f"bar={bar}, accepted={n}/{len(validation)}, errors={errors}, "
      f"95% upper={bound:.2%}, passes={passed}")

No accepted answers gives no evidence about accepted-answer risk. All accepted answers being wrong produces a bound of 100%. Group repeated requests by customer or session when creating splits if correlations would otherwise violate independent sampling.

Calibration and cost need separate checks

ECE summarizes differences between predicted probabilities and observed accuracy across probability bins; it is not the error rate above your chosen threshold. Historical JevBench hard-tier ECE was 0.184 for Standard One 8B v2 and 0.212 for 3B v2.1. The current v2.2 cards label that calibration as historical, so it does not validate a new version or your traffic.

Every request incurs a model call, including requests sent to review. Expected cost per request is model cost plus accepted fraction × accepted error rate × wrong-action cost, plus reviewed fraction × review cost. Include retries and reviewer mistakes where material; the cheapest threshold is not necessarily acceptable for costly errors.

After deployment, label a random sample across accepted and reviewed requests. Reviewer corrections are useful training examples but a biased accuracy sample. Revalidate when the model, options or traffic change. Standard One’s Docs describe its answer fields; classifier versus LLM selection covers alternatives when a threshold leaves too little useful automation.

Sources and verification

Sources used for this resource. Verification dates describe our source checks, not provider effective dates.

Standard Thinking documentation, Standard One answers and thresholds

Defines option probabilities, entropy-based confidence and the caller’s threshold policy.

Checked

Standard One 8B model card (v2)

The current card identifies v2.2 weights and labels the cited AG News, calibration and serial H200 latency results as historical v2 measurements.

Checked

Standard One 3B model card (v2.1)

The current card identifies v2.2 weights and retains the cited AG News and ECE results under historical v2.1 measurements.

Checked

Standard One benchmark report, calibration and latency

Defines historical top-label ECE and the serial latency profile; model version and serving conditions affect applicability.

Checked

Badshah, Emami and Sajjad, Judge, Retrieve, or Abstain (arXiv 2608.17994)

Version 1 derives fixed-threshold Clopper–Pearson risk bounds and discusses correction for threshold search. The numerical examples here are separately calculated.

Checked

Standard Thinking pricing, Standard One

Lists Standard One text-input rates and free output; no historical benchmark cost is treated as a current workload quote.

Checked

View all resources