How to set a confidence threshold for automated LLM decisions

A threshold decides which answers your code acts on alone and which it hands to a person. Set it on the probability of the answer the model chose, read the value off a labelled sample of your own traffic, and measure it again when that traffic changes. A model's published calibration tells you how far to trust its numbers before you have that sample; it does not replace it.

Compare the chosen answer's probability with the bar

Standard One returns a probability for every option you supply. For a choice question, the number to compare with your bar is the probability of the option it picked: in the documentation's example, 0.88 for billing. The same answer also carries confidence, one minus the normalized entropy of all the probabilities, which read 0.65 there because the other three options still held some weight. The first number estimates whether this answer is right; the second measures how spread out the model's belief is. Published calibration figures describe the first.

A yes-or-no (noul) question returns the probability of yes, so an answer of no carries one minus that. A score question returns the expected level and a probability for each level: threshold the probability of the level you act on, or the sum over a band of levels.

Check how far the numbers can be trusted

Calibration asks whether 0.9 means right nine times in ten. Expected calibration error (ECE) sorts answers into ten probability bands and averages the gap between stated probability and observed accuracy, weighted by how many answers fall in each band. On the public JevBench hard tier, where both sizes answer about half the questions correctly, ECE at the served temperatures is 0.184 for Standard One 8B and 0.212 for 3B. On questions that state their true odds, the mean total-variation distance between the returned probabilities and those odds is 0.108 and 0.124.

Both model cards attach the condition that matters most for thresholds: the temperatures that shape these probabilities were fitted on Standard Thinking's calibration suites, and on a materially different distribution of questions they should be refitted rather than assumed to transfer. Treat published figures as a starting estimate, not as your error rate.

Read the threshold off a labelled sample

1. Label a sample of real requests, drawn the way production traffic arrives, for each question you want to automate.
2. Run every case and keep two values: the probability of the chosen answer, and whether it was right.
3. For each candidate threshold, count the cases at or above it, which your code would handle alone, and the errors among them.
4. Pick the lowest threshold whose error rate meets your target, judged by an upper confidence bound rather than the raw rate. If 300 answers clear the bar and 3 are wrong, the observed error is 1.0%, but the one-sided 95% Clopper–Pearson bound is 2.6%. Badshah, Emami and Sajjad (2026) use the same bound to keep the error among accepted LLM verdicts below a chosen level.
5. Send everything below the threshold to review, and add the reviewed cases to the sample.

import csv

# labelled.csv: the chosen answer's probability, and 1 if the answer was right
rows = [(float(r["probability"]), r["correct"] == "1") for r in csv.DictReader(open("labelled.csv"))]
for bar in (0.5, 0.6, 0.7, 0.8, 0.9, 0.95):
    kept = [ok for p, ok in rows if p >= bar]
    errors = kept.count(False)
    print(f"{bar:.2f}  handled alone {len(kept) / len(rows):.1%}  errors {errors}/{len(kept)}")

No errors in the sample does not mean no errors in production: with none among 300 accepted answers, the same bound is still 1.0%.

Price both sides of the line

Every answer above the bar costs a model call and carries the chance of a wrong action; every answer below it costs a review. For each candidate threshold, the cost per case is the share handled alone times its error rate times the cost of a wrong action, plus the share reviewed times the cost of a review.

The model call is the smallest term. At the roughly 280 input tokens per decision used in the published latency test, a million decisions cost $5.32 on Standard One 3B and $13.72 on 8B at the current rates of $0.019 and $0.049 per million text input tokens, half the list price; output is free. The threshold is set by what a mistake and a review cost you, not by the model bill.

Keep measuring after launch

• Run the sample again when the inputs change: a new product line, a new language, reworded options.
• Watch the share sent to review. A jump means the traffic moved, even if the error rate above the bar still looks fine.
• Audit a small random slice of automated answers every week, so the error rate above the bar stays measured rather than assumed.
• Some tasks no threshold rescues. The 8B card reports ten-way support triage at 36% and RAG passage relevance at 59% without task training, and advises fine-tuning for those; when a fine-tuned classifier beats an LLM covers that choice.

Standard Thinking serves Standard One; its request shapes and measured results are on the model page and in the documentation.