What aggregate output tokens per second means

Convert an output rate into workload demand, then check bursts, input processing, concurrency, and latency separately.

By Standard Thinking · Updated

Aggregate output tokens per second measures generated tokens across requests together. A service producing 100 output tokens/s might share that output among several responses. The number alone does not tell you one response’s streaming speed, time to first token, or how many users can use the product at once.

Convert tokens/s into answers per hour

With an average answer length of T output tokens and aggregate output rate R:

answers per hour = 3,600 × R ÷ T
required tokens/s = answers per hour × T ÷ 3,600

For example, 600 answers an hour averaging 500 output tokens require 300,000 output tokens an hour, or about 83.3 tokens/s. This is an estimate of output demand. It does not test whether the service can process those prompts or finish each answer within your latency target.

Using the listed 135 tokens/s GLM‑5.3 allocation as an example:

Average answer Answers per hour at full output
200 output tokens 2,430
400 output tokens 1,215
800 output tokens 607.5

These are calculated averages, not measured completion rates. Doubling answer length halves the number of answers the same output rate represents. Continuous full output for 30 days would produce 349.92M output tokens; a monthly forecast must account for periods of lower demand and any other constraints.

An hourly average hides bursts

The 600-answer example averages about 83.3 tokens/s only if viewed across the full hour. If all 600 answers must be generated within ten minutes, the same output volume requires 500 tokens/s during that interval. The monthly token total is unchanged, but the rate needed to deliver it on time is six times higher.

Inspect the time windows that matter to the product. For interactive requests, record arrivals, overlapping requests, time to first token, and completion latency. For a batch job, record its deadline and whether queueing is acceptable. Use longer intervals to estimate cost and shorter ones to test service requirements.

Average answer length is another simplification. A mix of short answers and very long generations may have the same mean as a uniform workload while producing different queues and response times. Retain the distribution when you validate the estimate.

Output is only part of inference

Input work includes prompts, conversation history, documents, and tool results. Two requests that each generate 400 output tokens can require very different input processing. Cached input may reduce that work under the provider’s rules, but an output-only allocation does not specify those rules.

Before using an output estimate to buy capacity, confirm input throughput, context limits, concurrent requests, the measurement window, burst handling, and latency terms. User count requires additional assumptions about activity and overlap; dividing throughput by average answer length does not provide it.

Read each provider’s definition

A conversion from seconds to minutes is straightforward: multiply by 60. Comparing different providers’ capacity units requires more care. Azure PTUs use model-specific throughput and input/output weighting; Bedrock Model Units specify input and output tokens per minute across requests. Equal numeric rates can therefore represent different workloads.

Standard Thinking’s Product Plan lists aggregate output allocations. These are planning figures subject to the order’s model, allocation, and measurement conditions; input throughput, concurrency, burst duration, and latency targets are not publicly specified. Prices are confirmed before paid use.

The Product Plan capacity calculator converts an ordinary hour and a peak hour into estimated output units, then prices the resulting traffic. Bring shorter burst measurements alongside that estimate when checking whether a plan fits your product.

Sources and verification

Sources used for this resource. Verification dates describe our source checks, not provider effective dates.

Standard Thinking Product Plan

Listed model output allocations, monthly unit pricing, and extra-capacity billing support the planning examples. Prices, allocation conditions, and service commitments are confirmed before paid use.

Checked

Microsoft Azure AI Foundry provisioned throughput documentation

Provisioned deployments hold model processing capacity and bill allocated PTUs hourly. Azure Reservations discount that meter without securing deployment capacity; throughput and minimum size vary by model.

Checked

Amazon Bedrock Provisioned Throughput user guide

Model Units specify input and output throughput across requests, with hourly billing and model-dependent commitments. The documentation directs buyers to an account manager for unit specifications.

Checked

View all resources