← Blog
Orange and taupe token sequences illustrating LLM inference timing and throughput.
Published on
2026-08-17

Metrics that matter for LLM inference

Time to first token (TTFT) measures the delay from sending an LLM request to receiving its first generated content. Pair it with time per output token (TPOT), inter-token latency, completion latency, throughput, and goodput to understand both responsiveness and serving capacity.

This guide is for developers and platform engineers evaluating LLM inference performance metrics. It explains how to compare benchmarks, separate client and Router timing, interpret percentiles, and turn measurements into monitoring and capacity decisions. All numerical examples are illustrative.

A claim such as “5,000 tokens per second” needs context: the model, hardware, precision, input and output lengths, concurrent user requests, latency, and failures. Compare how much useful work completes under the conditions your application needs.

LLM inference performance metrics across the request timeline

Before looking at benchmark metrics, it helps to see what happens during one request.

A simplified request looks like this:

request sent
    ↓
network
    ↓
queue
    ↓
prefill
    ↓
first output token
    ↓
decode
    ↓
token → token → token → token
    ↓
final token

The prefill phase processes the input prompt.

The decode phase generates output autoregressively, normally one token at a time.

Those phases behave differently. Prefill processes many input tokens and is often relatively compute-intensive. Decode repeatedly loads model state while producing individual output tokens and can become heavily constrained by memory bandwidth and serving conditions.

The metrics used in LLM inference benchmarking are different views of this timeline. NVIDIA's current benchmarking guidance identifies TTFT, end-to-end latency, inter-token latency, tokens per second, and requests per second among the central measurements for LLM serving.

Here is the useful map:

You rarely want only one of them.

‍

Metric What it answers
TTFT How long until the user sees the first generated token?
TPOT How long does steady-state generation take per output token?
ITL How long are the gaps between streamed tokens?
E2E latency How long until the complete response arrives?
Output TPS How many output tokens does the whole system produce per second?
Per-user TPS How quickly does an individual response generate?
RPS How many requests does the system complete per second?
Goodput How many requests complete while meeting the latency targets?

‍

Time to first token (TTFT): the wait before an answer starts

Time to first token, or TTFT, measures the interval from sending the request until the first actual generated token arrives.

Conceptually:

request sent
|--------------------------------|
                            first token

<------------- TTFT ------------->

TTFT can include:

  • network time
  • queueing
  • tokenization
  • prompt processing
  • scheduling
  • generation of the first output token

The exact boundary depends on where the benchmark client measures from, which is one reason two benchmarking tools should not be assumed to calculate every metric identically. NVIDIA explicitly warns that popular benchmarking tools can differ in how apparently similar LLM metrics are measured.

For a streaming chatbot, TTFT is highly visible.

A response can generate extremely quickly once it begins and still feel slow if the interface spends several seconds doing nothing first.

Separate client latency, router waiting, and engine time

Adding a routing layer gives a request another place to wait. The router may hold it until a suitable backend has capacity; the inference engine may then queue it again before prefill. A healthy engine can therefore report a short TTFT while the client experiences a longer delay.

Measure user-facing TTFT from the client sending the request to receiving the first generated content. An HTTP response header, empty streaming event, or connection acknowledgment is not the first generated token. NVIDIA's metric definitions make this distinction explicit. For non-streaming responses, measure completion latency; arrival of the complete body does not reveal when the engine generated its first token.

Keep three measurement boundaries separate:

  • Client: TTFT, streamed output intervals, and completion latency across the path the application actually uses.
  • Router: admission, policy decisions, capacity waiting, forwarding, retries, and any fallback activity. Instrument these separately when you need to explain where time was spent.
  • Engine: backend queueing, prefill, and generation within the serving engine's own measurement boundaries.

In Hivenet Router, hivenet_queue_depth and hivenet_queue_wait_seconds describe the per-model queue for agent capacity. They do not describe every source of waiting, such as global concurrency admission or work already forwarded to an engine. hivenet_agent_engine_waiting_requests describes a separate backend queue when the engine supplies that metric.

The Router metrics reference also distinguishes engine TTFT from HTTP request duration. Treat an HTTP duration metric as a separate observation with its own start and end points. It is not automatically a measure of routing overhead, and an engine TTFT metric does not include the client's entire network and routing path.

If these layers are new to your deployment, our guide to model selection and backend routing explains where routing sits between an application and its inference servers.

Compare direct and routed requests under the same conditions

To investigate the effect of routing, run matched tests against one backend directly and through the router. Keep the model revision, hardware, serving settings, prompt distribution, output limits, cache state, client location, streaming mode, and offered load the same. Pin the routed test to that backend so a change of destination does not confound the comparison.

Repeat at low load and near the intended operating load. Compare client TTFT, completion latency, errors, and goodput, then inspect router and engine queues. A difference between the runs describes the tested paths under those conditions; it does not isolate policy-computation time. Use request-level traces for that attribution.

When testing LLM fallback, record the actual destination and model as well as elapsed time. A substituted provider model can change tokenization and output behavior, so its TPS should not be pooled into a like-for-like engine comparison. Keep failed and timed-out requests visible alongside successful recovery.

Long prompts usually put more pressure on TTFT

Before generating the first output token, the model processes the input prompt. This prefill time depends on the model, prompt length, cache state, and serving configuration. The attention mechanism can reuse matching cached state when the engine supports prefix caching; a new token still requires generation work.

A request containing:

500 input tokens

and one containing:

50,000 input tokens

therefore present very different workloads before decoding starts.

This is why comparing TTFT across benchmarks with different input lengths is usually meaningless.

NVIDIA's workload guidance emphasizes input and output sequence-length distributions because prompt length changes prefill work, memory requirements, and latency characteristics.

If one provider benchmarks 128-token prompts and another uses 16K-token prompts, their TTFT numbers are not measuring equivalent jobs.

TPOT measures generation after the first token

Time per output token, or TPOT, describes the average time required to produce output tokens after generation has started.

The reviewed vLLM benchmark implementation calculates per-request TPOT for outputs longer than one token as:

end-to-end latency - TTFT
─────────────────────────
   output tokens - 1

The first token is excluded because its latency belongs to TTFT. The formula requires more than one output token. The reviewed vLLM implementation excludes shorter outputs from its TPOT summary and assigns them zero TPOT for goodput evaluation, so check output lengths when interpreting that result.

If TPOT is:

10 ms

the corresponding steady-state generation rate is roughly:

1 / 0.010
=
100 tokens/s

for that request.

Lower TPOT is better.

Unlike TTFT, TPOT is mainly describing what happens after the answer has already begun.

Inter-token latency (ITL): gaps in streamed output

Inter-token latency, or ITL, measures the delay between consecutive streamed output tokens.

Imagine:

token      token         token   token
 ↓          ↓             ↓       ↓
 ●----8ms---●-----21ms-----●--7ms--●

TPOT gives you an average decoding rate.

ITL lets you see the individual gaps.

Current vLLM reports TPOT and ITL separately. TPOT is a per-request average; ITL records intervals between streamed output events. An event can contain multiple tokens, and buffering can change event timing. A client-observed event gap is therefore not necessarily the generation time of one token.

That distinction can expose jitter.

Two requests might have similar average TPOT while one produces:

10ms
10ms
11ms
9ms
10ms

and the other:

2ms
3ms
37ms
2ms
6ms

The averages can look acceptable while the second stream feels much less consistent.

Some benchmark literature and tools use TPOT and average ITL almost interchangeably. Others distinguish them. When comparing published results, check the tool's definition rather than assuming that identical acronyms mean identical measurement methods.

End-to-end latency measures the complete request

End-to-end latency measures the time from request submission until the complete response has been received.

For a simple streamed request:

E2E latency
=
TTFT
+
generation time

NVIDIA uses that same basic relationship in its inference benchmarking definitions.

End-to-end latency becomes especially important for workloads where users need the finished answer rather than the first token.

Examples include:

  • classification
  • extraction
  • structured JSON
  • summarization used by another service
  • agent steps
  • background processing

For a chat interface, TTFT and ITL may describe perceived responsiveness better.

For an automated pipeline waiting for a complete JSON object, end-to-end latency may be the number that matters most.

Tokens per second: per-user speed and total TPS

“Tokens per second” sounds wonderfully precise.

The problem is that people use it for several different things.

A benchmark might mean:

Per-user output speed

How quickly one response generates.

Request A → 90 tokens/s

Total output throughput

For aggregate output throughput, divide total output tokens from successful requests by the elapsed measurement interval. This total TPS describes work across all active requests; it is not the output speed experienced by one user. State which requests and timing boundaries the benchmark includes.

Server → 8,000 output tokens/s

Total token throughput

Some tools also report a number containing both:

input tokens
+
output tokens

These numbers answer different questions.

NVIDIA's metric definitions distinguish aggregate output throughput from per-user output throughput. The reviewed vLLM benchmark also reports total input-plus-output token throughput separately from output throughput.

So when somebody says:

This server does 10,000 TPS.

your next question should be:

Which TPS?

Per-user speed and system throughput can move in opposite directions

Suppose one request runs alone.

It might generate at:

150 tokens/s per user

Now increase concurrency.

The GPU can batch work from many requests together, becoming more efficient overall.

Perhaps the server reaches:

8,000 total tokens/s

but an individual request now receives:

80 tokens/s

Neither number is wrong.

The server improved its aggregate throughput while individual generation became slower.

That tradeoff is fundamental to LLM serving.

A provider quoting only aggregate TPS can therefore make a heavily batched system look spectacular while hiding what one user experiences.

A provider quoting only single-request TPS can make a server look responsive while telling you nothing about how much traffic it can handle.

You need both perspectives.

Requests per second measures completed jobs

Requests per second, or RPS, tells you how many requests the system completes over time.

This is particularly useful when requests have relatively consistent shapes.

For example:

12.5 completed requests/s

can be meaningful for a classification endpoint where every request has roughly:

500 input tokens
20 output tokens

But RPS becomes harder to compare when one benchmark contains tiny requests and another contains huge ones.

One request asking for a one-word classification and one request asking for a 4,000-token report both count as:

1 request

Token counts and request shapes therefore need to sit beside RPS.

Goodput asks whether the completed work was actually good enough

Throughput counts work that finished.

Goodput counts work that finished while satisfying the service's performance requirements.

Suppose a server completes:

100 requests/s

but your service requirement is:

TTFT < 1 second

and only:

72 requests/s

meet it.

Then:

throughput = 100 req/s
goodput    = 72 req/s

For the user-facing service, 72 may be the more meaningful capacity number.

vLLM can calculate request goodput against TTFT, TPOT, and end-to-end latency SLOs. A successful request must meet every configured threshold to count. This is a latency-based measure; it does not validate the answer's accuracy or usefulness.

That makes goodput one of the most useful concepts in production benchmarking.

It connects raw hardware performance to the experience the application actually promises.

Percentiles matter more than averages

Suppose a benchmark reports:

average TTFT = 450 ms

That looks good.

But imagine the distribution is:

P50 = 220 ms
P95 = 1.8 s
P99 = 6.4 s

A significant part of the traffic is having a very different experience from the average.

That is why useful benchmark reports include percentiles such as:

  • P50
  • P90
  • P95
  • P99

Current vLLM benchmarking can report percentiles independently for TTFT, TPOT, ITL, and end-to-end latency.

The higher percentiles describe tail latency.

Tail latency matters because production systems are not judged exclusively by their median user.

If an agent consists of ten sequential model calls, an occasional very slow request can also compound through the workflow.

P50, P95, and P99 answer different questions

A useful mental model is:

P50
What does a typical request experience?

P95
What does a relatively unlucky but still common request experience?

P99
How bad does the service get near the tail?

For an internal batch workload, average throughput may dominate.

For an interactive product, P95 TTFT may be far more useful.

For an API with a strict deadline, P99 end-to-end latency might determine whether the service is acceptable.

There is no universally correct percentile.

There is a percentile that corresponds to the product requirement.

Calculate fleet percentiles from distributions

Do not average each backend's P95 and call the result the fleet P95. Servers can process different numbers of requests with different latency distributions. Prometheus explains why precomputed quantiles cannot be aggregated; use compatible histogram buckets or the underlying request observations to calculate the fleet percentile.

In Hivenet Router v0.1.7, vLLM and SGLang agents can supply raw TTFT and ITL histogram buckets and the derived average and P90 gauges. Both also supply prompt-length and generation-length histograms. SGLang's ITL records one observation per generated token, as on vLLM. llama.cpp supplies no TTFT, ITL, or length histograms and no derived TTFT or ITL gauges. The decode-time histogram is vLLM-only; decode-phase token and time counters support output-speed calculations on all three engines. See the v0.1.7 engine-metrics reference.

Engine histograms and decode-phase counters carry deployment_id; scalar engine gauges do not. Scope deployment queries by organization as well, because agents declare deployment IDs. When scraping several routers connected to the same agents, deduplicate repeated engine snapshots before aggregation. Check which series your engine version exposes, and keep unlike models and workloads separate.

Likewise, subtracting engine P95 TTFT from client P95 TTFT does not yield P95 routing overhead. The two percentiles need not describe the same requests. Correlate individual requests and compare spans with matching boundaries when attributing time to a particular layer.

Monitoring TTFT and resource usage in production

Use monitoring tools to connect user-visible response times with the work occurring inside the serving stack. Track client TTFT, completion latency, errors, and goodput by model and request profile. Alongside them, track request rate, queue depth, active requests, batch size where exposed, and input/output token counts. As the number of concurrent requests increases, compare latency and total TPS together rather than treating either as an isolated success metric.

For engine diagnostics, inspect GPU memory usage, KV-cache occupancy, preemptions, and prefix-cache reuse. The vLLM production metrics reference documents these observations; metric availability and names depend on the deployed version. High resource usage alone does not prove that the model’s performance meets the application’s latency or quality requirements.

Request traces can help explain delays across the client, router, and engine, but only for the stages actually instrumented. Correlate observations with matching request IDs and timing boundaries. Keep failed and timed-out requests visible so a dashboard of successful responses does not hide overload.

Alert on sustained target breaches and investigate changes in workload, queueing, cache behavior, and backend health. Re-run representative benchmarks after engine upgrades, model changes, or new features alter the request path. These checks measure the deployment you operate; this article does not claim that they have been run on a Hivenet deployment.

Benchmark results need a defined workload

This is the single most important rule when comparing LLM inference systems.

You need to know what work was actually performed.

At minimum, a credible benchmark should identify:

Hivenet's own benchmark methodology follows the same principle: published results expose workload, hardware, environment, model or data, load profile, metrics, baseline, result, limitations, and test date.

‍

Benchmark condition Why it matters
Model and revision Different models require different amounts of work
Precision / quantization BF16, FP8, NVFP4, INT4, etc. change compute and memory
GPU model Hardware changes compute and bandwidth
GPU count Aggregate compute and memory change
Serving engine vLLM, TensorRT-LLM, SGLang, etc. behave differently
Engine version Kernels and scheduling change over time
Input length Drives prefill work
Output length Drives decode work
Concurrency Changes batching, throughput, and latency
Request rate Determines offered load
Context limit Affects serving configuration and memory
KV-cache precision Changes cache capacity
Parallelism TP, PP, DP, or EP changes communication
Dataset / prompt shape Synthetic and real traffic behave differently
Benchmark duration Short bursts can hide steady-state issues
Error count Failed work should not disappear from the report

‍

Without the test conditions, a benchmark result is difficult to reproduce and dangerous to compare.

Input and output length deserve particular attention

Imagine two RTX 5090 benchmarks.

Benchmark A

input = 128 tokens
output = 128 tokens

Benchmark B

input = 16,384 tokens
output = 2,048 tokens

Even with the same model and serving engine, these are radically different workloads.

Benchmark B has:

  • much more prefill work
  • more KV-cache use
  • substantially longer decode
  • different batching behavior
  • different opportunities for request overlap

If A produces more TPS, that tells you almost nothing about which deployment is better for workload B.

Real prompt and response distributions belong in the benchmark.

Context limit is not the same thing as prompt length

A server might be configured to support:

128K context

while the benchmark sends only:

1K prompts

That is useful information.

But it does not demonstrate performance at 128K.

The context limit tells you what the server permits.

The benchmark request shape tells you what it tested.

Our KV cache guide explains why long active context consumes increasing GPU memory and can change serving capacity dramatically.

If long-context performance matters, test long contexts.

Concurrency is where the real serving curve appears

A benchmark at concurrency 1 tells you something useful:

the low-load latency floor.

It does not tell you how the system behaves as traffic increases.

A proper serving benchmark should normally sweep concurrency or request rate.

At low concurrency:

low latency
low aggregate throughput

As concurrency rises:

batching improves
GPU utilization rises
aggregate throughput rises

Eventually:

throughput approaches saturation
queueing increases
latency rises sharply

NVIDIA's benchmarking guidance recommends sweeping concurrency to understand the throughput and latency tradeoff.

Conceptually:

Throughput
   ^
   |                       _________
   |                   ___/
   |                __/
   |             __/
   |___________/
   +-----------------------------> concurrency


Latency
   ^
   |                         /
   |                       _/
   |                    __/
   |___________________/
   +-----------------------------> concurrency

The interesting part is not necessarily the highest point on the throughput curve.

It is the best operating point before latency becomes unacceptable.

Saturation is where more traffic stops buying useful throughput

Suppose you measure:

Moving from:

Concurrency Output TPS P95 TTFT
1 150 120 ms
4 560 170 ms
8 1,020 260 ms
16 1,700 480 ms
32 2,020 1.4 s
64 2,090 4.8 s

1 → 16 concurrent requests

buys a large increase in throughput with a moderate latency increase.

Moving from:

32 → 64

adds very little throughput while P95 TTFT becomes dramatically worse.

For an interactive service, concurrency 64 is probably not the winning configuration even though it produces the largest TPS number.

This is why:

maximum benchmark throughput

and:

useful production capacity

are different things.

The best result usually sits on a latency-throughput tradeoff

Infrastructure benchmarking is a multi-objective problem.

You may want:

lower TTFT
higher throughput
lower cost
more concurrency
more model quality

at the same time.

Usually, you cannot maximize all five.

On a latency-throughput Pareto frontier, no tested configuration improves one of those metrics without worsening the other under the same workload. Compare cost separately before choosing a deployment.

You do not need formal optimization mathematics to use the idea.

If configuration B is:

slower
more expensive
and
lower throughput

than configuration A under the same workload, B is not an interesting deployment.

If B costs more but substantially improves latency, now you have a real tradeoff to evaluate.

Quantization should be benchmarked as a quality-performance trade

Suppose you compare:

BF16
FP8
NVFP4

Quantization may improve:

  • weight memory
  • GPU bandwidth pressure
  • throughput
  • GPU count
  • KV-cache headroom

But a benchmark containing only latency and TPS is incomplete.

You also need to establish that the quantized model still performs the task adequately.

Our NVFP4 guide explains why model quality needs to be evaluated alongside memory and throughput.

A 4-bit model that generates 30% faster but breaks your structured output is not a 30% improvement.

Inference benchmarking and model-quality evaluation are separate tests.

You need both.

Tensor parallelism needs cost-per-result benchmarking

Suppose the same model can run at:

TP = 2

or:

TP = 4

Four GPUs may reduce latency.

They may also double the hardware used by the replica.

The comparison should therefore include:

  • TTFT
  • TPOT / ITL
  • aggregate throughput
  • throughput per GPU
  • cache capacity
  • concurrency
  • cost

Our tensor parallelism guide explains why adding GPUs does not produce linear scaling: communication and synchronization become part of every distributed forward pass.

The benchmark tells you whether the additional GPUs actually earn their place.

PagedAttention and batching mainly show up under load

Running one request does not tell you much about a serving scheduler.

Techniques such as:

  • PagedAttention
  • continuous batching
  • chunked prefill
  • prefix caching

become important as requests overlap and compete for GPU memory and compute.

That is why our PagedAttention and continuous batching guide focuses on concurrent serving rather than single-request speed.

If you are testing scheduling efficiency, use a workload that creates a scheduling problem.

A good benchmark also reports errors

Imagine:

10,000 requests attempted
9,300 succeeded
700 failed with OOM

If the benchmark reports throughput from only the successful requests and never mentions the failures, the number is misleading.

Useful stability signals include:

  • completed requests
  • timeouts
  • OOMs
  • HTTP errors
  • cancellations
  • request preemptions
  • retry rate

Hivenet's benchmark methodology explicitly includes error rate where it is relevant because throughput is only useful while the setup remains stable.

A system that produces excellent throughput immediately before falling over is not high capacity.

Warm-up matters

The first few requests after loading a model can behave differently from steady-state traffic.

Potential causes include:

  • kernel compilation
  • CUDA graph setup
  • memory allocation
  • cache warming
  • disk/model loading
  • runtime initialization

If one benchmark includes cold-start requests and another measures an already warm server, their latency numbers may differ for reasons unrelated to normal serving performance.

State whether the benchmark is measuring:

cold start

or:

steady state

and how warm-up requests were handled.

Synthetic benchmarks are useful when you know what they represent

Synthetic requests make controlled comparison easy.

You can hold constant:

input length
output length
request rate
concurrency

and change one system variable.

That is excellent for A/B testing hardware or serving configuration.

It is less useful if your production traffic has a very different shape.

Current vllm bench serve supports synthetic and custom datasets and can output distributions for prompt, output, and combined token counts.

Use synthetic traffic to isolate variables.

Use representative traffic to predict production.

Those are different benchmark goals.

Load testing and benchmarking are related but different

A performance benchmark asks:

How does this model-serving configuration perform under defined conditions?

A load test asks:

What happens when we put realistic or extreme traffic pressure on the service?

NVIDIA distinguishes load testing from performance benchmarking: load tests examine system behavior under traffic, while performance benchmarks measure serving throughput and latency under specified conditions.

You normally want both before production.

The benchmark helps choose the configuration.

The load test helps discover where it breaks.

How to read a benchmark result in practice

This report uses illustrative numbers to explain how to read results. It is not a measured Hivenet or RTX 5090 benchmark:

Model:             Example-27B
Precision:         NVFP4
GPU:               1 × RTX 5090
Input length:      2,048 tokens
Output length:     512 tokens
Concurrency:       16

Output throughput: 2,400 tokens/s
Request throughput: 4.7 req/s

TTFT
P50: 260 ms
P95: 610 ms
P99: 940 ms

TPOT
P50: 8.2 ms
P95: 11.4 ms

Errors: 0

Do not begin with:

2,400 tokens/s

Begin with the test conditions.

One RTX 5090. A 27B model. NVFP4. 2K prompts. 512-token outputs. Concurrency 16.

Now the performance numbers have meaning.

For an interactive application, you might conclude:

  • typical first-token latency is good
  • P95 TTFT remains under one second
  • steady-state generation is fast
  • no errors occurred
  • concurrency 16 achieves 2,400 output TPS

Then compare the same workload at concurrency 8 and 32.

That tells you whether 16 is an efficient operating point or merely one arbitrary measurement.

Now imagine a competing benchmark

Provider B says:

Output throughput: 3,200 tokens/s

It appears 33% faster.

But then you discover:

Input length:       128 tokens
Output length:      128 tokens
Concurrency:        64
P95 TTFT:           5.2 seconds

These are not directly comparable results.

Provider B ran:

  • much shorter requests
  • much more concurrency
  • a dramatically worse tail TTFT

It may still be an excellent configuration for offline batch generation.

It may be a poor configuration for your interactive application.

The word faster cannot resolve that.

The workload and SLO can.

Benchmark red flags

I would be cautious with any LLM inference benchmark that does one or more of these.

Reports TPS without defining it

Is it:

per-user output TPS
total output TPS
total input + output TPS

Those are different metrics.

Gives no prompt or output lengths

Then you do not know what workload produced the result.

Gives no concurrency or request rate

Then you cannot tell whether it measures single-request latency or heavily batched throughput.

Reports only averages

Tail latency can disappear inside a good mean.

Compares different quantizations without evaluating quality

Faster output is not equivalent output if the model became materially worse.

Compares different model revisions

The hardware comparison is now contaminated by a model change.

Changes the serving engine and hardware simultaneously

You know the result changed.

You do not know which variable caused it.

Reports aggregate throughput without individual latency

Heavy batching can inflate the first while degrading the second.

Uses extremely short synthetic prompts for a long-context application

The benchmark does not represent the workload.

Ignores failures

A request that OOMs or times out did not become free merely because it is absent from the TPS calculation.

Benchmarks one concurrency level

You cannot see the latency-throughput curve or the saturation point.

Change one variable at a time when you want an explanation

Suppose you want to know whether FP8 KV cache helps.

Keep constant:

model
weights
GPU
GPU count
serving engine
input lengths
output lengths
request rate
concurrency
context limit

Change:

KV cache precision

Now you have an experiment.

If you simultaneously move from:

BF16 → NVFP4
1 GPU → 2 GPUs
vLLM → TensorRT-LLM
8K → 32K context

and throughput rises, you have demonstrated a different configuration.

You have not learned which optimization mattered.

Both kinds of benchmarks are useful.

Do not confuse them.

How to benchmark a vLLM endpoint

Current vLLM includes:

vllm bench serve

for online-serving benchmarks.

A representative command might look like:

vllm bench serve \
--backend openai \
--model "<model-id>" \
--base-url "<base-url>" \
--endpoint /v1/completions \
--dataset-name random \
--random-input-len 2048 \
--random-output-len 512 \
--num-prompts 1000 \
--max-concurrency 8 \
--request-rate inf \
--num-warmups 5 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,95,99 \
--save-result --save-detailed

This illustrative Bash command has not been executed for this article. Replace the model and base URL, configure authentication as needed, and confirm that the endpoint supports completions. Check the current vLLM CLI documentation and record the installed version. Here, --backend openai selects an OpenAI-compatible protocol; it does not require OpenAI hosting.

The example allows up to eight concurrent requests with no request-rate spacing, after five warm-up requests. It is a concurrency-limited synthetic test, not a realistic arrival-rate simulation. For a request-rate sweep, set a finite rate and report any concurrency cap: that cap can reduce the actual offered rate. Add explicit --goodput thresholds in milliseconds if reporting request goodput.

The important part is not the command.

It is controlling the workload.

Run a concurrency or request-rate sweep

Do not stop at one test.

For example:

concurrency 1
concurrency 2
concurrency 4
concurrency 8
concurrency 16
concurrency 32
concurrency 64

For each level, record:

TTFT P50 / P95 / P99
TPOT P50 / P95 / P99
ITL
output TPS
RPS
errors

Then plot:

throughput vs TTFT

or:

throughput vs TPOT

The curve will tell you far more than one headline number.

For foundational model-serving tests, record prompt shapes, concurrency, latency, and throughput together so readers can interpret the result.

Benchmark your real prompt distribution too

After the controlled sweep, test something resembling production.

If your application normally receives:

60% → 1–2K prompts
30% → 4–8K prompts
10% → 20K+ prompts

use a dataset that reflects it.

Do the same for output length.

A server optimized for:

short prompt
long output

may behave differently from one serving:

long prompt
short output

even when their average total token counts look similar.

The shape matters because prefill and decode stress the system differently.

Benchmark caching only with workloads that can use it

If you are testing prefix caching, create repeated prefixes.

If you benchmark only random unrelated prompts, the system has little opportunity to reuse prefix state.

Likewise, if you want to evaluate KV-cache pressure, use enough context and concurrency to create meaningful cache demand.

A benchmark should create the problem the optimization claims to solve.

Otherwise the result tells you almost nothing about that feature.

Convert benchmark performance into cost

Once you have a stable throughput result, infrastructure economics become easier to calculate.

Suppose the deployment costs:

C currency units per hour

and produces:

T usable output tokens per second

Then:

output tokens per hour
=
T × 3,600

and a simple infrastructure cost per million output tokens becomes:

C
──────────────── × 1,000,000
T × 3,600

But remember the word usable.

If the configuration misses your latency SLO or produces unacceptable model quality, its cheap tokens are not equivalent to the tokens from a configuration that meets the product requirement.

Use throughput that meets your latency targets when estimating capacity and cost. A peak throughput result at unacceptable latency is a poor sizing input.

Goodput can produce a better cost number

Suppose:

System A
10,000 output tokens/s
60% of requests meet SLO

and:

System B
8,500 output tokens/s
98% of requests meet SLO

Raw throughput favors A.

System B has a higher fraction of requests meeting the SLO, but these figures alone do not establish its goodput in requests per second or useful tokens per second. Report completed requests per second and count tokens from SLO-passing requests. Multiplying total token throughput by the passing-request percentage can mislead when output lengths differ.

You can apply the same logic to cost:

cost per SLO-passing request = hourly infrastructure cost / (request goodput × 3,600)

Here, request goodput is measured in requests per second over the same workload and operating conditions used for the cost estimate.

That is a much more demanding metric.

It is also closer to what the business is paying for.

A benchmark should say what it does not prove

This is a good habit in technical writing.

A result such as:

Configuration A delivered lower P95 TTFT than configuration B at concurrency 16.

does not prove:

Configuration A is the better inference system for every workload.

Perhaps B performs better with long prompts.

Perhaps B scales further at high concurrency.

Perhaps A uses a lower-quality quantization.

Perhaps B costs half as much.

Hivenet's benchmark methodology explicitly includes a limits field for this reason: a result should state what the test supports and what it cannot establish.

That makes a benchmark more trustworthy, not less impressive.

Which metrics should you care about?

The answer depends on the workload.

Interactive chat

Prioritize:

TTFT
ITL / TPOT
P95 / P99 latency
goodput

The user needs the response to begin quickly and stream smoothly.

Agents

Watch:

TTFT
E2E latency
tail latency
error rate

One agent action may depend on several sequential model calls, so slow tails can compound.

RAG

Watch:

TTFT
prompt length
prefill performance
KV-cache use
throughput under mixed contexts

Retrieved context can make prompts much larger.

Coding

Watch:

TTFT
long-context behavior
generation speed
tail latency

Large repositories can create substantial prefill workloads.

Offline batch inference

Prioritize:

aggregate throughput
cost per useful token/request
error rate

Individual TTFT may matter very little.

Structured extraction

Prioritize:

E2E latency
RPS
correct output rate
error rate

A fast malformed JSON object is not a successful result.

What is a good TTFT? Set a workload-specific target

There is no universal TTFT threshold. Interactive chatbots and AI copilots benefit from prompt initial feedback, while background jobs may tolerate a longer wait. Measure TTFT alongside TPOT and completion latency: a quick first token followed by slow generation can still make the model’s output difficult to use.

Choose targets from the application’s response-time needs, then validate them under realistic load. Compare the same model and request profile where possible; when comparing different models or inference engines, document tokenizer differences and evaluate answer quality separately.

Service level agreement versus internal latency targets

A service-level objective (SLO) specifies a measured target. A service level agreement (SLA) records a service commitment and consequences when it is not met; Google’s SRE guide distinguishes these terms. If a service provider offers an SLA, check its measurement window, request eligibility, percentile, and exclusions. A provider’s server-side latency commitment may cover a different boundary from your client TTFT.

The benchmark should help you make an infrastructure decision

A good benchmark ends with an action.

Perhaps:

NVFP4 lets the model fit on one GPU
without unacceptable quality loss.

Or:

TP=4 improves latency,
but TP=2 has much better throughput per euro.

Or:

Concurrency 32 is the highest point
that keeps P95 TTFT under our 1-second SLO.

Or:

FP8 KV cache lets us support
twice as many concurrent long-context requests.

Or:

The bigger GPU does not improve this workload
enough to justify its price.

That is the purpose of benchmarking.

Not producing the largest number.

Producing enough evidence to make a better deployment decision.

How Hivenet approaches benchmark results

Hivenet's benchmark hub describes its methodology and the areas it tracks, including GPU, inference, and VM performance. Check the individual result for its actual scope, conditions, and limitations.

For inference work, that means looking at metrics such as:

  • TTFT
  • TPOT
  • ITL
  • end-to-end latency
  • throughput
  • concurrency
  • error rate

alongside the actual model, precision, hardware, software environment, workload, and load profile.

That is also the standard we should use when comparing Hivenet with another infrastructure option.

If the workload changes, say so.

If the benchmark tests only one part of the system, say so.

If a difference sits inside normal variance, say so.

The goal of a benchmark is to make the result easier to question and reproduce, not harder.

The useful way to read any LLM benchmark

When someone shows you an inference benchmark, read it in this order.

1. What model was tested?

Check model revision, precision, and quantization.

2. What hardware was used?

Check GPU type and GPU count.

3. What did each request look like?

Check input and output token distributions.

4. How much load was applied?

Check concurrency and request rate.

5. What did users experience?

Look at TTFT, TPOT/ITL, E2E latency, and tail percentiles.

6. How much work did the server complete?

Look at output TPS and RPS.

7. Did it stay stable?

Check errors, OOMs, timeouts, and preemptions.

8. Did the model remain good enough?

Performance benchmarking does not replace quality evaluation.

9. What did it cost?

Compare useful throughput or goodput against the hardware cost.

10. What does the test not prove?

Look for the boundaries of the result.

If those ten answers are available, you probably have a benchmark worth using.

If all you have is:

12,438 TOKENS/SECOND!!!

you mostly have a number.

Source review: Tool definitions and CLI options were checked against vLLM and NVIDIA documentation on November 11, 2026. Router metric boundaries were rechecked against Hivenet Router v0.1.7 on November 16, 2026. Numerical examples and tables in this guide are illustrative; this review did not run a benchmark or measure Router overhead.

LLM inference benchmark FAQ

What is an LLM inference benchmark?

An LLM inference benchmark measures the performance of a deployed language model under defined workload conditions. Useful benchmarks normally measure latency and throughput while documenting the model, hardware, precision, request shape, and load.

What is TTFT?

TTFT means time to first token. It measures how long the user waits from submitting a request until the first generated token arrives.

What affects TTFT?

TTFT can be affected by network latency, queueing, scheduling, prompt length, prefill performance, model size, hardware, batching, and current server load.

Does engine TTFT include routing and network time?

Not necessarily. An engine measures from its own internal boundary, while client TTFT includes the path from the application to the first received content. Router queues, backend queues, transport, and buffering can contribute different delays. Record the measurement boundary beside every TTFT result.

Can I average P95 latency across servers?

No. Aggregate compatible histogram buckets or individual observations before calculating the fleet percentile. An average of per-server P95 values is not a valid fleet P95.

What is TPOT?

TPOT means time per output token. It describes average steady-state generation time after the first output token. Current vLLM calculates it from post-TTFT generation time divided by the number of remaining output tokens.

What is ITL?

ITL means inter-token latency. It describes gaps in streamed output, but measurement depends on the tool. vLLM records gaps between output events, which can contain multiple tokens, so report streaming and buffering behavior alongside the result.

Are ITL and TPOT the same?

They describe closely related decode performance, and some tools use the terms similarly. Current vLLM reports both: TPOT is a per-request average over post-first-token generation, while ITL records the individual gaps between output events.

What is end-to-end latency?

End-to-end latency measures the time from submitting the request until the complete response is received. It includes the initial wait and the generation period.

What does tokens per second mean for an LLM?

It can mean different things. A benchmark may report individual generation speed, total server output throughput, or total input-plus-output throughput. Always check the metric definition before comparing TPS results.

Is higher TPS always better?

No. Aggregate throughput can increase as concurrency rises while individual-request latency becomes worse. Evaluate throughput together with TTFT, TPOT/ITL, and your latency targets.

What is RPS?

RPS means requests per second. It measures how many requests the system completes each second.

What is goodput?

Goodput measures work completed within defined requirements. In vLLM, request goodput counts successful requests that meet all configured TTFT, TPOT, and end-to-end latency SLOs, divided by benchmark duration. It does not assess answer quality.

Why should an LLM benchmark report P95 or P99?

Averages can hide slow requests. Percentiles show tail latency and help determine what relatively unlucky users experience under load.

What is a concurrency sweep?

A concurrency sweep repeats the same benchmark at increasing numbers of simultaneous requests. It reveals how throughput and latency change as the system approaches saturation.

Why does prompt length matter in an LLM benchmark?

Longer prompts require more prefill computation and normally consume more inference memory. Comparing systems with different prompt lengths can therefore produce misleading latency and throughput conclusions.

Why does output length matter?

Longer outputs create more autoregressive decode work and keep requests active for longer. They also increase KV-cache state as generation proceeds.

What is the difference between an LLM benchmark and a load test?

A benchmark measures performance under controlled conditions. A load test focuses on how the service behaves as realistic or extreme traffic stresses capacity, scaling, queues, networking, and stability.

What is the best metric for an interactive chatbot?

TTFT and ITL/TPOT are particularly useful for perceived responsiveness, but throughput, P95/P99 latency, and goodput matter once multiple users share the service.

What is the best metric for batch inference?

Aggregate throughput, cost per completed job or token, and error rate are usually more important than single-request TTFT.

Can two LLM benchmark results be compared directly?

Only when the model, precision, hardware, request shape, load, serving configuration, and metric definitions are sufficiently comparable. Otherwise the numbers may describe different workloads.

How do I benchmark vLLM?

Current vLLM provides vllm bench serve for online-serving benchmarks, including TTFT, TPOT, ITL, end-to-end latency, throughput, percentiles, and goodput metrics.

Should model quality be part of an inference benchmark?

Performance and quality should both be evaluated when changing the model or its precision. A faster quantized model is only a useful improvement if it continues to meet the application's quality requirements.

How do I calculate LLM inference cost?

Once you know sustainable throughput under your latency SLO, divide the infrastructure cost by the amount of useful work completed. Cost per million output tokens is one common measure, while cost per successful request may be more useful for other applications.

What information should every LLM inference benchmark publish?

At minimum: model and revision, precision, hardware, GPU count, serving engine and version, input/output lengths, concurrency or request rate, benchmark duration, latency metrics, throughput, error rate, and any important limitations.

‍

Your next workload belongs on Hivenet.

Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.