
Time to first token (TTFT) measures the delay from sending an LLM request to receiving its first generated content. Pair it with time per output token (TPOT), inter-token latency, completion latency, throughput, and goodput to understand both responsiveness and serving capacity.
This guide is for developers and platform engineers evaluating LLM inference performance metrics. It explains how to compare benchmarks, separate client and Router timing, interpret percentiles, and turn measurements into monitoring and capacity decisions. All numerical examples are illustrative.
A claim such as “5,000 tokens per second” needs context: the model, hardware, precision, input and output lengths, concurrent user requests, latency, and failures. Compare how much useful work completes under the conditions your application needs.
Before looking at benchmark metrics, it helps to see what happens during one request.
A simplified request looks like this:
request sent
↓
network
↓
queue
↓
prefill
↓
first output token
↓
decode
↓
token → token → token → token
↓
final token
The prefill phase processes the input prompt.
The decode phase generates output autoregressively, normally one token at a time.
Those phases behave differently. Prefill processes many input tokens and is often relatively compute-intensive. Decode repeatedly loads model state while producing individual output tokens and can become heavily constrained by memory bandwidth and serving conditions.
The metrics used in LLM inference benchmarking are different views of this timeline. NVIDIA's current benchmarking guidance identifies TTFT, end-to-end latency, inter-token latency, tokens per second, and requests per second among the central measurements for LLM serving.
Here is the useful map:
You rarely want only one of them.
| Metric | What it answers |
|---|---|
| TTFT | How long until the user sees the first generated token? |
| TPOT | How long does steady-state generation take per output token? |
| ITL | How long are the gaps between streamed tokens? |
| E2E latency | How long until the complete response arrives? |
| Output TPS | How many output tokens does the whole system produce per second? |
| Per-user TPS | How quickly does an individual response generate? |
| RPS | How many requests does the system complete per second? |
| Goodput | How many requests complete while meeting the latency targets? |
Time to first token, or TTFT, measures the interval from sending the request until the first actual generated token arrives.
Conceptually:
request sent
|--------------------------------|
first token
<------------- TTFT ------------->
TTFT can include:
The exact boundary depends on where the benchmark client measures from, which is one reason two benchmarking tools should not be assumed to calculate every metric identically. NVIDIA explicitly warns that popular benchmarking tools can differ in how apparently similar LLM metrics are measured.
For a streaming chatbot, TTFT is highly visible.
A response can generate extremely quickly once it begins and still feel slow if the interface spends several seconds doing nothing first.
Adding a routing layer gives a request another place to wait. The router may hold it until a suitable backend has capacity; the inference engine may then queue it again before prefill. A healthy engine can therefore report a short TTFT while the client experiences a longer delay.
Measure user-facing TTFT from the client sending the request to receiving the first generated content. An HTTP response header, empty streaming event, or connection acknowledgment is not the first generated token. NVIDIA's metric definitions make this distinction explicit. For non-streaming responses, measure completion latency; arrival of the complete body does not reveal when the engine generated its first token.
Keep three measurement boundaries separate:
In Hivenet Router, hivenet_queue_depth and hivenet_queue_wait_seconds describe the per-model queue for agent capacity. They do not describe every source of waiting, such as global concurrency admission or work already forwarded to an engine. hivenet_agent_engine_waiting_requests describes a separate backend queue when the engine supplies that metric.
The Router metrics reference also distinguishes engine TTFT from HTTP request duration. Treat an HTTP duration metric as a separate observation with its own start and end points. It is not automatically a measure of routing overhead, and an engine TTFT metric does not include the client's entire network and routing path.
If these layers are new to your deployment, our guide to model selection and backend routing explains where routing sits between an application and its inference servers.
To investigate the effect of routing, run matched tests against one backend directly and through the router. Keep the model revision, hardware, serving settings, prompt distribution, output limits, cache state, client location, streaming mode, and offered load the same. Pin the routed test to that backend so a change of destination does not confound the comparison.
Repeat at low load and near the intended operating load. Compare client TTFT, completion latency, errors, and goodput, then inspect router and engine queues. A difference between the runs describes the tested paths under those conditions; it does not isolate policy-computation time. Use request-level traces for that attribution.
When testing LLM fallback, record the actual destination and model as well as elapsed time. A substituted provider model can change tokenization and output behavior, so its TPS should not be pooled into a like-for-like engine comparison. Keep failed and timed-out requests visible alongside successful recovery.
Before generating the first output token, the model processes the input prompt. This prefill time depends on the model, prompt length, cache state, and serving configuration. The attention mechanism can reuse matching cached state when the engine supports prefix caching; a new token still requires generation work.
A request containing:
500 input tokens
and one containing:
50,000 input tokens
therefore present very different workloads before decoding starts.
This is why comparing TTFT across benchmarks with different input lengths is usually meaningless.
NVIDIA's workload guidance emphasizes input and output sequence-length distributions because prompt length changes prefill work, memory requirements, and latency characteristics.
If one provider benchmarks 128-token prompts and another uses 16K-token prompts, their TTFT numbers are not measuring equivalent jobs.
Time per output token, or TPOT, describes the average time required to produce output tokens after generation has started.
The reviewed vLLM benchmark implementation calculates per-request TPOT for outputs longer than one token as:
end-to-end latency - TTFT
─────────────────────────
output tokens - 1
The first token is excluded because its latency belongs to TTFT. The formula requires more than one output token. The reviewed vLLM implementation excludes shorter outputs from its TPOT summary and assigns them zero TPOT for goodput evaluation, so check output lengths when interpreting that result.
If TPOT is:
10 ms
the corresponding steady-state generation rate is roughly:
1 / 0.010
=
100 tokens/s
for that request.
Lower TPOT is better.
Unlike TTFT, TPOT is mainly describing what happens after the answer has already begun.
Inter-token latency, or ITL, measures the delay between consecutive streamed output tokens.
Imagine:
token token token token
↓ ↓ ↓ ↓
●----8ms---●-----21ms-----●--7ms--●
TPOT gives you an average decoding rate.
ITL lets you see the individual gaps.
Current vLLM reports TPOT and ITL separately. TPOT is a per-request average; ITL records intervals between streamed output events. An event can contain multiple tokens, and buffering can change event timing. A client-observed event gap is therefore not necessarily the generation time of one token.
That distinction can expose jitter.
Two requests might have similar average TPOT while one produces:
10ms
10ms
11ms
9ms
10ms
and the other:
2ms
3ms
37ms
2ms
6ms
The averages can look acceptable while the second stream feels much less consistent.
Some benchmark literature and tools use TPOT and average ITL almost interchangeably. Others distinguish them. When comparing published results, check the tool's definition rather than assuming that identical acronyms mean identical measurement methods.
End-to-end latency measures the time from request submission until the complete response has been received.
For a simple streamed request:
E2E latency
=
TTFT
+
generation time
NVIDIA uses that same basic relationship in its inference benchmarking definitions.
End-to-end latency becomes especially important for workloads where users need the finished answer rather than the first token.
Examples include:
For a chat interface, TTFT and ITL may describe perceived responsiveness better.
For an automated pipeline waiting for a complete JSON object, end-to-end latency may be the number that matters most.
“Tokens per second” sounds wonderfully precise.
The problem is that people use it for several different things.
A benchmark might mean:
How quickly one response generates.
Request A → 90 tokens/s
For aggregate output throughput, divide total output tokens from successful requests by the elapsed measurement interval. This total TPS describes work across all active requests; it is not the output speed experienced by one user. State which requests and timing boundaries the benchmark includes.
Server → 8,000 output tokens/s
Some tools also report a number containing both:
input tokens
+
output tokens
These numbers answer different questions.
NVIDIA's metric definitions distinguish aggregate output throughput from per-user output throughput. The reviewed vLLM benchmark also reports total input-plus-output token throughput separately from output throughput.
So when somebody says:
This server does 10,000 TPS.
your next question should be:
Which TPS?
Suppose one request runs alone.
It might generate at:
150 tokens/s per user
Now increase concurrency.
The GPU can batch work from many requests together, becoming more efficient overall.
Perhaps the server reaches:
8,000 total tokens/s
but an individual request now receives:
80 tokens/s
Neither number is wrong.
The server improved its aggregate throughput while individual generation became slower.
That tradeoff is fundamental to LLM serving.
A provider quoting only aggregate TPS can therefore make a heavily batched system look spectacular while hiding what one user experiences.
A provider quoting only single-request TPS can make a server look responsive while telling you nothing about how much traffic it can handle.
You need both perspectives.
Requests per second, or RPS, tells you how many requests the system completes over time.
This is particularly useful when requests have relatively consistent shapes.
For example:
12.5 completed requests/s
can be meaningful for a classification endpoint where every request has roughly:
500 input tokens
20 output tokens
But RPS becomes harder to compare when one benchmark contains tiny requests and another contains huge ones.
One request asking for a one-word classification and one request asking for a 4,000-token report both count as:
1 request
Token counts and request shapes therefore need to sit beside RPS.
Throughput counts work that finished.
Goodput counts work that finished while satisfying the service's performance requirements.
Suppose a server completes:
100 requests/s
but your service requirement is:
TTFT < 1 second
and only:
72 requests/s
meet it.
Then:
throughput = 100 req/s
goodput = 72 req/s
For the user-facing service, 72 may be the more meaningful capacity number.
vLLM can calculate request goodput against TTFT, TPOT, and end-to-end latency SLOs. A successful request must meet every configured threshold to count. This is a latency-based measure; it does not validate the answer's accuracy or usefulness.
That makes goodput one of the most useful concepts in production benchmarking.
It connects raw hardware performance to the experience the application actually promises.
Suppose a benchmark reports:
average TTFT = 450 ms
That looks good.
But imagine the distribution is:
P50 = 220 ms
P95 = 1.8 s
P99 = 6.4 s
A significant part of the traffic is having a very different experience from the average.
That is why useful benchmark reports include percentiles such as:
Current vLLM benchmarking can report percentiles independently for TTFT, TPOT, ITL, and end-to-end latency.
The higher percentiles describe tail latency.
Tail latency matters because production systems are not judged exclusively by their median user.
If an agent consists of ten sequential model calls, an occasional very slow request can also compound through the workflow.
A useful mental model is:
P50
What does a typical request experience?
P95
What does a relatively unlucky but still common request experience?
P99
How bad does the service get near the tail?
For an internal batch workload, average throughput may dominate.
For an interactive product, P95 TTFT may be far more useful.
For an API with a strict deadline, P99 end-to-end latency might determine whether the service is acceptable.
There is no universally correct percentile.
There is a percentile that corresponds to the product requirement.
Do not average each backend's P95 and call the result the fleet P95. Servers can process different numbers of requests with different latency distributions. Prometheus explains why precomputed quantiles cannot be aggregated; use compatible histogram buckets or the underlying request observations to calculate the fleet percentile.
In Hivenet Router v0.1.7, vLLM and SGLang agents can supply raw TTFT and ITL histogram buckets and the derived average and P90 gauges. Both also supply prompt-length and generation-length histograms. SGLang's ITL records one observation per generated token, as on vLLM. llama.cpp supplies no TTFT, ITL, or length histograms and no derived TTFT or ITL gauges. The decode-time histogram is vLLM-only; decode-phase token and time counters support output-speed calculations on all three engines. See the v0.1.7 engine-metrics reference.
Engine histograms and decode-phase counters carry deployment_id; scalar engine gauges do not. Scope deployment queries by organization as well, because agents declare deployment IDs. When scraping several routers connected to the same agents, deduplicate repeated engine snapshots before aggregation. Check which series your engine version exposes, and keep unlike models and workloads separate.
Likewise, subtracting engine P95 TTFT from client P95 TTFT does not yield P95 routing overhead. The two percentiles need not describe the same requests. Correlate individual requests and compare spans with matching boundaries when attributing time to a particular layer.
Use monitoring tools to connect user-visible response times with the work occurring inside the serving stack. Track client TTFT, completion latency, errors, and goodput by model and request profile. Alongside them, track request rate, queue depth, active requests, batch size where exposed, and input/output token counts. As the number of concurrent requests increases, compare latency and total TPS together rather than treating either as an isolated success metric.
For engine diagnostics, inspect GPU memory usage, KV-cache occupancy, preemptions, and prefix-cache reuse. The vLLM production metrics reference documents these observations; metric availability and names depend on the deployed version. High resource usage alone does not prove that the model’s performance meets the application’s latency or quality requirements.
Request traces can help explain delays across the client, router, and engine, but only for the stages actually instrumented. Correlate observations with matching request IDs and timing boundaries. Keep failed and timed-out requests visible so a dashboard of successful responses does not hide overload.
Alert on sustained target breaches and investigate changes in workload, queueing, cache behavior, and backend health. Re-run representative benchmarks after engine upgrades, model changes, or new features alter the request path. These checks measure the deployment you operate; this article does not claim that they have been run on a Hivenet deployment.
This is the single most important rule when comparing LLM inference systems.
You need to know what work was actually performed.
At minimum, a credible benchmark should identify:
Hivenet's own benchmark methodology follows the same principle: published results expose workload, hardware, environment, model or data, load profile, metrics, baseline, result, limitations, and test date.
| Benchmark condition | Why it matters |
|---|---|
| Model and revision | Different models require different amounts of work |
| Precision / quantization | BF16, FP8, NVFP4, INT4, etc. change compute and memory |
| GPU model | Hardware changes compute and bandwidth |
| GPU count | Aggregate compute and memory change |
| Serving engine | vLLM, TensorRT-LLM, SGLang, etc. behave differently |
| Engine version | Kernels and scheduling change over time |
| Input length | Drives prefill work |
| Output length | Drives decode work |
| Concurrency | Changes batching, throughput, and latency |
| Request rate | Determines offered load |
| Context limit | Affects serving configuration and memory |
| KV-cache precision | Changes cache capacity |
| Parallelism | TP, PP, DP, or EP changes communication |
| Dataset / prompt shape | Synthetic and real traffic behave differently |
| Benchmark duration | Short bursts can hide steady-state issues |
| Error count | Failed work should not disappear from the report |
Without the test conditions, a benchmark result is difficult to reproduce and dangerous to compare.
Imagine two RTX 5090 benchmarks.
input = 128 tokens
output = 128 tokens
input = 16,384 tokens
output = 2,048 tokens
Even with the same model and serving engine, these are radically different workloads.
Benchmark B has:
If A produces more TPS, that tells you almost nothing about which deployment is better for workload B.
Real prompt and response distributions belong in the benchmark.
A server might be configured to support:
128K context
while the benchmark sends only:
1K prompts
That is useful information.
But it does not demonstrate performance at 128K.
The context limit tells you what the server permits.
The benchmark request shape tells you what it tested.
Our KV cache guide explains why long active context consumes increasing GPU memory and can change serving capacity dramatically.
If long-context performance matters, test long contexts.
A benchmark at concurrency 1 tells you something useful:
the low-load latency floor.
It does not tell you how the system behaves as traffic increases.
A proper serving benchmark should normally sweep concurrency or request rate.
At low concurrency:
low latency
low aggregate throughput
As concurrency rises:
batching improves
GPU utilization rises
aggregate throughput rises
Eventually:
throughput approaches saturation
queueing increases
latency rises sharply
NVIDIA's benchmarking guidance recommends sweeping concurrency to understand the throughput and latency tradeoff.
Conceptually:
Throughput
^
| _________
| ___/
| __/
| __/
|___________/
+-----------------------------> concurrency
Latency
^
| /
| _/
| __/
|___________________/
+-----------------------------> concurrency
The interesting part is not necessarily the highest point on the throughput curve.
It is the best operating point before latency becomes unacceptable.
Suppose you measure:
Moving from:
| Concurrency | Output TPS | P95 TTFT |
|---|---|---|
| 1 | 150 | 120 ms |
| 4 | 560 | 170 ms |
| 8 | 1,020 | 260 ms |
| 16 | 1,700 | 480 ms |
| 32 | 2,020 | 1.4 s |
| 64 | 2,090 | 4.8 s |
1 → 16 concurrent requests
buys a large increase in throughput with a moderate latency increase.
Moving from:
32 → 64
adds very little throughput while P95 TTFT becomes dramatically worse.
For an interactive service, concurrency 64 is probably not the winning configuration even though it produces the largest TPS number.
This is why:
maximum benchmark throughput
and:
useful production capacity
are different things.
Infrastructure benchmarking is a multi-objective problem.
You may want:
lower TTFT
higher throughput
lower cost
more concurrency
more model quality
at the same time.
Usually, you cannot maximize all five.
On a latency-throughput Pareto frontier, no tested configuration improves one of those metrics without worsening the other under the same workload. Compare cost separately before choosing a deployment.
You do not need formal optimization mathematics to use the idea.
If configuration B is:
slower
more expensive
and
lower throughput
than configuration A under the same workload, B is not an interesting deployment.
If B costs more but substantially improves latency, now you have a real tradeoff to evaluate.
Suppose you compare:
BF16
FP8
NVFP4
Quantization may improve:
But a benchmark containing only latency and TPS is incomplete.
You also need to establish that the quantized model still performs the task adequately.
Our NVFP4 guide explains why model quality needs to be evaluated alongside memory and throughput.
A 4-bit model that generates 30% faster but breaks your structured output is not a 30% improvement.
Inference benchmarking and model-quality evaluation are separate tests.
You need both.
Suppose the same model can run at:
TP = 2
or:
TP = 4
Four GPUs may reduce latency.
They may also double the hardware used by the replica.
The comparison should therefore include:
Our tensor parallelism guide explains why adding GPUs does not produce linear scaling: communication and synchronization become part of every distributed forward pass.
The benchmark tells you whether the additional GPUs actually earn their place.
Running one request does not tell you much about a serving scheduler.
Techniques such as:
become important as requests overlap and compete for GPU memory and compute.
That is why our PagedAttention and continuous batching guide focuses on concurrent serving rather than single-request speed.
If you are testing scheduling efficiency, use a workload that creates a scheduling problem.
Imagine:
10,000 requests attempted
9,300 succeeded
700 failed with OOM
If the benchmark reports throughput from only the successful requests and never mentions the failures, the number is misleading.
Useful stability signals include:
Hivenet's benchmark methodology explicitly includes error rate where it is relevant because throughput is only useful while the setup remains stable.
A system that produces excellent throughput immediately before falling over is not high capacity.
The first few requests after loading a model can behave differently from steady-state traffic.
Potential causes include:
If one benchmark includes cold-start requests and another measures an already warm server, their latency numbers may differ for reasons unrelated to normal serving performance.
State whether the benchmark is measuring:
cold start
or:
steady state
and how warm-up requests were handled.
Synthetic requests make controlled comparison easy.
You can hold constant:
input length
output length
request rate
concurrency
and change one system variable.
That is excellent for A/B testing hardware or serving configuration.
It is less useful if your production traffic has a very different shape.
Current vllm bench serve supports synthetic and custom datasets and can output distributions for prompt, output, and combined token counts.
Use synthetic traffic to isolate variables.
Use representative traffic to predict production.
Those are different benchmark goals.
A performance benchmark asks:
How does this model-serving configuration perform under defined conditions?
A load test asks:
What happens when we put realistic or extreme traffic pressure on the service?
NVIDIA distinguishes load testing from performance benchmarking: load tests examine system behavior under traffic, while performance benchmarks measure serving throughput and latency under specified conditions.
You normally want both before production.
The benchmark helps choose the configuration.
The load test helps discover where it breaks.
This report uses illustrative numbers to explain how to read results. It is not a measured Hivenet or RTX 5090 benchmark:
Model: Example-27B
Precision: NVFP4
GPU: 1 × RTX 5090
Input length: 2,048 tokens
Output length: 512 tokens
Concurrency: 16
Output throughput: 2,400 tokens/s
Request throughput: 4.7 req/s
TTFT
P50: 260 ms
P95: 610 ms
P99: 940 ms
TPOT
P50: 8.2 ms
P95: 11.4 ms
Errors: 0
Do not begin with:
2,400 tokens/s
Begin with the test conditions.
One RTX 5090. A 27B model. NVFP4. 2K prompts. 512-token outputs. Concurrency 16.
Now the performance numbers have meaning.
For an interactive application, you might conclude:
Then compare the same workload at concurrency 8 and 32.
That tells you whether 16 is an efficient operating point or merely one arbitrary measurement.
Provider B says:
Output throughput: 3,200 tokens/s
It appears 33% faster.
But then you discover:
Input length: 128 tokens
Output length: 128 tokens
Concurrency: 64
P95 TTFT: 5.2 seconds
These are not directly comparable results.
Provider B ran:
It may still be an excellent configuration for offline batch generation.
It may be a poor configuration for your interactive application.
The word faster cannot resolve that.
The workload and SLO can.
I would be cautious with any LLM inference benchmark that does one or more of these.
Is it:
per-user output TPS
total output TPS
total input + output TPS
Those are different metrics.
Then you do not know what workload produced the result.
Then you cannot tell whether it measures single-request latency or heavily batched throughput.
Tail latency can disappear inside a good mean.
Faster output is not equivalent output if the model became materially worse.
The hardware comparison is now contaminated by a model change.
You know the result changed.
You do not know which variable caused it.
Heavy batching can inflate the first while degrading the second.
The benchmark does not represent the workload.
A request that OOMs or times out did not become free merely because it is absent from the TPS calculation.
You cannot see the latency-throughput curve or the saturation point.
Suppose you want to know whether FP8 KV cache helps.
Keep constant:
model
weights
GPU
GPU count
serving engine
input lengths
output lengths
request rate
concurrency
context limit
Change:
KV cache precision
Now you have an experiment.
If you simultaneously move from:
BF16 → NVFP4
1 GPU → 2 GPUs
vLLM → TensorRT-LLM
8K → 32K context
and throughput rises, you have demonstrated a different configuration.
You have not learned which optimization mattered.
Both kinds of benchmarks are useful.
Do not confuse them.
Current vLLM includes:
vllm bench serve
for online-serving benchmarks.
A representative command might look like:
vllm bench serve \
--backend openai \
--model "<model-id>" \
--base-url "<base-url>" \
--endpoint /v1/completions \
--dataset-name random \
--random-input-len 2048 \
--random-output-len 512 \
--num-prompts 1000 \
--max-concurrency 8 \
--request-rate inf \
--num-warmups 5 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,95,99 \
--save-result --save-detailed
This illustrative Bash command has not been executed for this article. Replace the model and base URL, configure authentication as needed, and confirm that the endpoint supports completions. Check the current vLLM CLI documentation and record the installed version. Here, --backend openai selects an OpenAI-compatible protocol; it does not require OpenAI hosting.
The example allows up to eight concurrent requests with no request-rate spacing, after five warm-up requests. It is a concurrency-limited synthetic test, not a realistic arrival-rate simulation. For a request-rate sweep, set a finite rate and report any concurrency cap: that cap can reduce the actual offered rate. Add explicit --goodput thresholds in milliseconds if reporting request goodput.
The important part is not the command.
It is controlling the workload.
Do not stop at one test.
For example:
concurrency 1
concurrency 2
concurrency 4
concurrency 8
concurrency 16
concurrency 32
concurrency 64
For each level, record:
TTFT P50 / P95 / P99
TPOT P50 / P95 / P99
ITL
output TPS
RPS
errors
Then plot:
throughput vs TTFT
or:
throughput vs TPOT
The curve will tell you far more than one headline number.
For foundational model-serving tests, record prompt shapes, concurrency, latency, and throughput together so readers can interpret the result.
After the controlled sweep, test something resembling production.
If your application normally receives:
60% → 1–2K prompts
30% → 4–8K prompts
10% → 20K+ prompts
use a dataset that reflects it.
Do the same for output length.
A server optimized for:
short prompt
long output
may behave differently from one serving:
long prompt
short output
even when their average total token counts look similar.
The shape matters because prefill and decode stress the system differently.
If you are testing prefix caching, create repeated prefixes.
If you benchmark only random unrelated prompts, the system has little opportunity to reuse prefix state.
Likewise, if you want to evaluate KV-cache pressure, use enough context and concurrency to create meaningful cache demand.
A benchmark should create the problem the optimization claims to solve.
Otherwise the result tells you almost nothing about that feature.
Once you have a stable throughput result, infrastructure economics become easier to calculate.
Suppose the deployment costs:
C currency units per hour
and produces:
T usable output tokens per second
Then:
output tokens per hour
=
T × 3,600
and a simple infrastructure cost per million output tokens becomes:
C
──────────────── × 1,000,000
T × 3,600
But remember the word usable.
If the configuration misses your latency SLO or produces unacceptable model quality, its cheap tokens are not equivalent to the tokens from a configuration that meets the product requirement.
Use throughput that meets your latency targets when estimating capacity and cost. A peak throughput result at unacceptable latency is a poor sizing input.
Suppose:
System A
10,000 output tokens/s
60% of requests meet SLO
and:
System B
8,500 output tokens/s
98% of requests meet SLO
Raw throughput favors A.
System B has a higher fraction of requests meeting the SLO, but these figures alone do not establish its goodput in requests per second or useful tokens per second. Report completed requests per second and count tokens from SLO-passing requests. Multiplying total token throughput by the passing-request percentage can mislead when output lengths differ.
You can apply the same logic to cost:
cost per SLO-passing request = hourly infrastructure cost / (request goodput × 3,600)
Here, request goodput is measured in requests per second over the same workload and operating conditions used for the cost estimate.
That is a much more demanding metric.
It is also closer to what the business is paying for.
This is a good habit in technical writing.
A result such as:
Configuration A delivered lower P95 TTFT than configuration B at concurrency 16.
does not prove:
Configuration A is the better inference system for every workload.
Perhaps B performs better with long prompts.
Perhaps B scales further at high concurrency.
Perhaps A uses a lower-quality quantization.
Perhaps B costs half as much.
Hivenet's benchmark methodology explicitly includes a limits field for this reason: a result should state what the test supports and what it cannot establish.
That makes a benchmark more trustworthy, not less impressive.
The answer depends on the workload.
Prioritize:
TTFT
ITL / TPOT
P95 / P99 latency
goodput
The user needs the response to begin quickly and stream smoothly.
Watch:
TTFT
E2E latency
tail latency
error rate
One agent action may depend on several sequential model calls, so slow tails can compound.
Watch:
TTFT
prompt length
prefill performance
KV-cache use
throughput under mixed contexts
Retrieved context can make prompts much larger.
Watch:
TTFT
long-context behavior
generation speed
tail latency
Large repositories can create substantial prefill workloads.
Prioritize:
aggregate throughput
cost per useful token/request
error rate
Individual TTFT may matter very little.
Prioritize:
E2E latency
RPS
correct output rate
error rate
A fast malformed JSON object is not a successful result.
There is no universal TTFT threshold. Interactive chatbots and AI copilots benefit from prompt initial feedback, while background jobs may tolerate a longer wait. Measure TTFT alongside TPOT and completion latency: a quick first token followed by slow generation can still make the model’s output difficult to use.
Choose targets from the application’s response-time needs, then validate them under realistic load. Compare the same model and request profile where possible; when comparing different models or inference engines, document tokenizer differences and evaluate answer quality separately.
A service-level objective (SLO) specifies a measured target. A service level agreement (SLA) records a service commitment and consequences when it is not met; Google’s SRE guide distinguishes these terms. If a service provider offers an SLA, check its measurement window, request eligibility, percentile, and exclusions. A provider’s server-side latency commitment may cover a different boundary from your client TTFT.
A good benchmark ends with an action.
Perhaps:
NVFP4 lets the model fit on one GPU
without unacceptable quality loss.
Or:
TP=4 improves latency,
but TP=2 has much better throughput per euro.
Or:
Concurrency 32 is the highest point
that keeps P95 TTFT under our 1-second SLO.
Or:
FP8 KV cache lets us support
twice as many concurrent long-context requests.
Or:
The bigger GPU does not improve this workload
enough to justify its price.
That is the purpose of benchmarking.
Not producing the largest number.
Producing enough evidence to make a better deployment decision.
Hivenet's benchmark hub describes its methodology and the areas it tracks, including GPU, inference, and VM performance. Check the individual result for its actual scope, conditions, and limitations.
For inference work, that means looking at metrics such as:
alongside the actual model, precision, hardware, software environment, workload, and load profile.
That is also the standard we should use when comparing Hivenet with another infrastructure option.
If the workload changes, say so.
If the benchmark tests only one part of the system, say so.
If a difference sits inside normal variance, say so.
The goal of a benchmark is to make the result easier to question and reproduce, not harder.
When someone shows you an inference benchmark, read it in this order.
1. What model was tested?
Check model revision, precision, and quantization.
2. What hardware was used?
Check GPU type and GPU count.
3. What did each request look like?
Check input and output token distributions.
4. How much load was applied?
Check concurrency and request rate.
5. What did users experience?
Look at TTFT, TPOT/ITL, E2E latency, and tail percentiles.
6. How much work did the server complete?
Look at output TPS and RPS.
7. Did it stay stable?
Check errors, OOMs, timeouts, and preemptions.
8. Did the model remain good enough?
Performance benchmarking does not replace quality evaluation.
9. What did it cost?
Compare useful throughput or goodput against the hardware cost.
10. What does the test not prove?
Look for the boundaries of the result.
If those ten answers are available, you probably have a benchmark worth using.
If all you have is:
12,438 TOKENS/SECOND!!!
you mostly have a number.
Source review: Tool definitions and CLI options were checked against vLLM and NVIDIA documentation on November 11, 2026. Router metric boundaries were rechecked against Hivenet Router v0.1.7 on November 16, 2026. Numerical examples and tables in this guide are illustrative; this review did not run a benchmark or measure Router overhead.
An LLM inference benchmark measures the performance of a deployed language model under defined workload conditions. Useful benchmarks normally measure latency and throughput while documenting the model, hardware, precision, request shape, and load.
TTFT means time to first token. It measures how long the user waits from submitting a request until the first generated token arrives.
TTFT can be affected by network latency, queueing, scheduling, prompt length, prefill performance, model size, hardware, batching, and current server load.
Not necessarily. An engine measures from its own internal boundary, while client TTFT includes the path from the application to the first received content. Router queues, backend queues, transport, and buffering can contribute different delays. Record the measurement boundary beside every TTFT result.
No. Aggregate compatible histogram buckets or individual observations before calculating the fleet percentile. An average of per-server P95 values is not a valid fleet P95.
TPOT means time per output token. It describes average steady-state generation time after the first output token. Current vLLM calculates it from post-TTFT generation time divided by the number of remaining output tokens.
ITL means inter-token latency. It describes gaps in streamed output, but measurement depends on the tool. vLLM records gaps between output events, which can contain multiple tokens, so report streaming and buffering behavior alongside the result.
They describe closely related decode performance, and some tools use the terms similarly. Current vLLM reports both: TPOT is a per-request average over post-first-token generation, while ITL records the individual gaps between output events.
End-to-end latency measures the time from submitting the request until the complete response is received. It includes the initial wait and the generation period.
It can mean different things. A benchmark may report individual generation speed, total server output throughput, or total input-plus-output throughput. Always check the metric definition before comparing TPS results.
No. Aggregate throughput can increase as concurrency rises while individual-request latency becomes worse. Evaluate throughput together with TTFT, TPOT/ITL, and your latency targets.
RPS means requests per second. It measures how many requests the system completes each second.
Goodput measures work completed within defined requirements. In vLLM, request goodput counts successful requests that meet all configured TTFT, TPOT, and end-to-end latency SLOs, divided by benchmark duration. It does not assess answer quality.
Averages can hide slow requests. Percentiles show tail latency and help determine what relatively unlucky users experience under load.
A concurrency sweep repeats the same benchmark at increasing numbers of simultaneous requests. It reveals how throughput and latency change as the system approaches saturation.
Longer prompts require more prefill computation and normally consume more inference memory. Comparing systems with different prompt lengths can therefore produce misleading latency and throughput conclusions.
Longer outputs create more autoregressive decode work and keep requests active for longer. They also increase KV-cache state as generation proceeds.
A benchmark measures performance under controlled conditions. A load test focuses on how the service behaves as realistic or extreme traffic stresses capacity, scaling, queues, networking, and stability.
TTFT and ITL/TPOT are particularly useful for perceived responsiveness, but throughput, P95/P99 latency, and goodput matter once multiple users share the service.
Aggregate throughput, cost per completed job or token, and error rate are usually more important than single-request TTFT.
Only when the model, precision, hardware, request shape, load, serving configuration, and metric definitions are sufficiently comparable. Otherwise the numbers may describe different workloads.
Current vLLM provides vllm bench serve for online-serving benchmarks, including TTFT, TPOT, ITL, end-to-end latency, throughput, percentiles, and goodput metrics.
Performance and quality should both be evaluated when changing the model or its precision. A faster quantized model is only a useful improvement if it continues to meet the application's quality requirements.
Once you know sustainable throughput under your latency SLO, divide the infrastructure cost by the amount of useful work completed. Cost per million output tokens is one common measure, while cost per successful request may be more useful for other applications.
At minimum: model and revision, precision, hardware, GPU count, serving engine and version, input/output lengths, concurrency or request rate, benchmark duration, latency metrics, throughput, error rate, and any important limitations.
Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.