Fine-tuned classifier vs LLM: how to choose
Choose an encoder, prompted LLM or decision model by your data and labels; research and historical Standard One results use different measurement conditions.
A fine-tuned classifier is a strong baseline when your labels are stable and you have representative training data. A prompted LLM or decision model can accept new label descriptions without retraining, which helps when the schema changes or you have few examples. Choose by testing the same held-out workload: neither approach is universally the most accurate.
Compare the work each approach requires
| Requirement | Fine-tuned encoder | Prompted LLM | Decision model |
|---|---|---|---|
| Task-labelled training data | Usually needed | Optional examples | Optional examples |
| New label set | Usually retrain | Update prompt | Update supplied options |
| Returned result | Labels and scores | Generated answer | Option probabilities |
| Probability calibration | Must evaluate | Must evaluate | Must evaluate |
| Deployment | Host or managed endpoint | API or self-hosting | API or self-hosting |
Encoders also return class scores or probabilities. The practical distinction is how you define labels and obtain the output, not whether one approach alone can support a review threshold. Every probability needs validation on your task.
What independent studies measured
Valdes Gonzalez’s AG News study evaluated fine-tuned encoders and prompted models on 3,800 test items. The following figures belong to that experiment; latency includes the serving path, and costs use its hosting and token-pricing assumptions.
| Measure | Fine-tuned encoders | Prompted LLMs |
|---|---|---|
| Best macro-F1 | RoBERTa: 94.63% | Claude 4.5 zero-shot: 91.35% |
| Median latency range | 108–197 ms | 332–1,435 ms |
| Estimated cost per million | $5.73–$10.44 | $276–$1,271.58 |
Those results support trying an encoder for a stable classification task. They do not predict your production bill or rank an independently measured decision model.
Rodrigues and Vas compared RoBERTa with zero-shot Claude Haiku on intent detection. RoBERTa reached 95.9% accuracy on ATIS versus 84.1% for Claude; their CLINC150 results were much closer. Claude recalled 85.6% of out-of-scope examples versus 58.1% for RoBERTa. In a deliberately disjoint label-set test, the locked classifier could not emit the new app’s labels, while Claude accepted the new schema. These findings concern those models and tasks; they are not evidence of Standard One’s out-of-scope performance.
Standard One’s measurements
Our docs now carry v2.2 served-endpoint figures: One 3B was measured on October 3, 2026 and One 8B on October 1–3, 2026, on weights updated October 4, 2026. They cover the JevBench public tiers and several decision suites, such as a 600-question realistic transfer set (88.17% for 3B, 91.50% for 8B). They are not AG News or intent-detection results, so they cannot be placed beside the studies above or ranked against them.
The model cards also report a separate evaluation on 400 sampled AG News cases, run on earlier versions: 84.5% accuracy for 3B v2.1 (released September 27, 2026) and 84.2% for 8B v2 (released September 26, 2026). These are accuracy figures on a smaller sample, so they cannot be placed beside the study’s macro-F1 as a leaderboard. The disclosed training mixture does not include AG News; it does include MASSIVE, so multilingual MASSIVE results have a different training-data relationship.
The cards also report median model-server latency of 22.6 ms for 3B and 25.8 ms for 8B, measured on v2 weights on September 24–26, 2026; they give no v2.2 latency figure. That serial profile used one H200 and 242 short decisions averaging about 280 input tokens. It excludes caller network time and differs from the study’s CPU services and API calls.
Both model cards identify v2.2 weights, updated October 4, 2026, and label the earlier benchmark figures as historical. In the v2.2 figures, “served endpoint” means the reference server described on the cards. Pin the evaluated model version when reproducing a result.
Choose with a test that resembles production
Train an encoder baseline when your label set is settled and you can collect representative examples. Include simple baselines: the benefit of a large model should justify its operating cost and complexity.
Try a schema-driven model when new labels must work before a training run is possible. Include ambiguous cases, rare classes, changed label descriptions and out-of-scope inputs in the evaluation; flexibility in accepting options does not establish reliable decisions.
Compare all candidates on the same split, labels and success criteria. Measure per-class errors, accepted-answer error at your review threshold, end-to-end latency, retries and total cost, including training and maintenance. Keep a random audit sample of automated decisions as well as reviewer corrections; corrections alone overrepresent difficult cases.
Standard One scores supplied options through POST /v1/systemone. Its model page and Docs describe the API. The guide to LLM confidence thresholds explains how to validate a review policy on your own traffic.
Sources and verification
Sources used for this resource. Verification dates describe our source checks, not provider effective dates.
Valdes Gonzalez, Cost-Aware Model Selection for Text Classification (arXiv 2602.06370)
Version 1 compares encoders and prompted LLMs on AG News using macro-F1, end-to-end latency and experiment-specific cost estimates; these are separate from our measurements.
Rodrigues and Vas, When Do LLMs Replace Fine-Tuned NLU? (arXiv 2608.20371)
Version 1 compares RoBERTa and Claude Haiku on ATIS, CLINC150 and changed schemas; its out-of-scope findings do not measure Standard One.
Standard One 3B model card (v2.1)
The current card identifies v2.2 weights, reports a v2.2 served-endpoint table measured 3 October 2026, and retains the cited AG News and ECE results under historical v2.1 measurements.
Standard One 8B model card (v2)
The current card identifies v2.2 weights, reports a v2.2 served-endpoint table measured 1–3 October 2026, and labels the cited AG News, calibration and serial H200 latency results as historical v2 measurements.
Standard One public classification suites
Defines historical public classification suites and sample sizes; no current endpoint performance is inferred.
Standard One benchmark report, training data provenance
Historical training-data disclosure lists MASSIVE among source datasets and does not list AG News.
Standard Thinking pricing, Standard One
Lists Standard One text-input rates and free output; no historical benchmark cost is treated as a current workload quote.
Continue exploring
2.6%95% upper bound for 3 errors in 300
- Observed
- 1.0%
- Threshold
- Fixed before validation
- Sample
- Independent validation
One-sided Clopper–Pearson bound, not a worst-case guarantee
How to set an LLM confidence threshold
Select a candidate threshold, freeze the policy and validate accepted-answer risk independently, with runnable code and sample-size limits.
19.6Mtokens a month
- Input
- 18.8M
- Output
- 850K
- Model calls
- 5K
Simple tool loop · 1,000 runs a month
Estimate your monthly API cost
Estimate API usage from independent calls, growing chat history, or agent workflows, with editable rates and context assumptions.
57.40public index, 8B SH
- 3B SH
- 51.53
- Retrieval & Classification
- 66.4
- Tools & Automation
- 84.0
Decision Index 0.3 public index, self-scored
Standard One SH scores every option in one pass
Open-weight decision models that return a probability for every option in one forward pass: 51.5 and 57.4 on the Decision Index 0.3 public index, self-scored with the official kit, with the detail and the limits.