← Blog
Two illustrated computer towers with glowing turquoise components on an orange background.
Published on
2025-09-24

LLM inference in production: a practical guide

LLM inference is the process of using a trained large language model to generate output from an input prompt. Running it in production means meeting response latency, model quality, and cost targets as traffic changes. This guide is for developers and platform engineers choosing a serving engine, sizing GPU memory, and evaluating LLM performance before rollout. It covers key metrics, batching, caching, hardware, infrastructure costs, and reliability checks.

‍
Need control over the serving stack? Compute provides GPU or CPU instances where you configure and operate your model server. For a managed model endpoint, choose Hivenet Inference Service and check its supported models, variants, and regional capacity.

How LLM inference works: prefill and decoding

In autoregressive text generation, prefill processes the input tokens and builds attention state. During the decoding phase, the model uses that state to generate output tokens. Each generated token becomes part of the context for subsequent generation. The KV cache stores attention keys and values so the engine can reuse them instead of recomputing that state. Generation ends at a supported stop condition or an output limit.

The input sequence length includes the system instructions, conversation history, retrieved passages, and user prompt. Output length is how many tokens the model generates. Check the deployed maximum context length and leave room for the answer. On a GPU, weights, cache, and engine workspace share memory; multiple requests can increase the cache requirement. These stages and workload terms are explained in NVIDIA’s inference benchmarking guide.

Set targets for time to first token, completion latency, error rate, and useful throughput. Then test whether the chosen model and serving configuration meet those targets under expected traffic and overload. A peak tokens-per-second result alone does not establish production capacity.

Key metrics for LLM inference performance

Choose metrics for the application. Interactive chat needs low latency so users can begin reading promptly; offline workloads may prioritize high throughput and cost per completed job. Measure the following under the same request profile:

  • Time to first token (TTFT). How quickly the response starts. Users feel this, as it marks the initial delay before the model's output begins. TTFT is influenced by request queuing, prefill, and network latency.
  • Tokens per second (TPS). State whether you mean the output rate of one response or aggregate server throughput.
  • Throughput. Measure completed requests or output tokens per unit of time under defined latency targets. Concurrent requests describe concurrency, not throughput.
  • Inter-token latency (ITL). Gaps during streamed generation affect how smooth the answer feels. Check the benchmark definition: some tools report an average per-token delay, while others record gaps between output events that may contain several tokens. Report percentiles where available, alongside the measurement boundary.

Time per output token, requests per second, and goodput

Time per output token (TPOT) describes the average delay after the first token. For an output longer than one token, a common definition is (completion latency − TTFT) / (output tokens − 1). Check the tool’s formula and streaming behavior before equating TPOT with ITL.

Requests per second (RPS) counts successful completed requests divided by elapsed time. Goodput counts requests completed within defined service-level objectives per second. In vLLM’s serving benchmark, each successful request must meet all configured latency thresholds to count toward request goodput. That latency check does not assess answer quality.

System throughput in output TPS is total output tokens divided by the measurement interval. Record the interval, successful-request criteria, and whether warm-up is excluded. Also report average latency and P95/P99 response times; an average can hide slow requests. The inference metrics guide explains the measurement boundaries in more detail.

Track end-to-end latency and its percentiles too. Align metric definitions and use the same model, tokenizer, and request profile when comparing results. Different tokenizers make raw TPS comparisons unreliable.

Choose compute for your workload

Compare available GPU and CPU presets, choose a compatible image, and review the active price before launching. Configure and test your application on the selected instance.

Try Compute now

Serving architectures

Single GPU. Start here when model weights, KV cache, and runtime workspace fit with headroom for your target context lengths and concurrency. A parameter count such as 7B or 13B alone does not establish whether the workload fits.

Multiple GPUs. Use a supported parallelism strategy when the workload requires it. vLLM documents tensor and pipeline parallelism for distributing a model. Communication overhead and interconnect bandwidth affect performance; benchmark scaling rather than assuming that doubling GPUs doubles throughput.

Horizontal scale. Run serving replicas behind a gateway with health checks and load balancing. If you route related requests to the same replica for prefix-cache reuse, verify that the engine supports reuse and that matching prefixes remain cached. Affinity can concentrate load, so test its effect on queueing and tail latency. See our vLLM load-balancing architecture guide for routing topology and acceptance checks.

When a deployment grows beyond one server, separate choosing the model from choosing the backend that serves it. Our guide to LLM routing, model selection, and backend routing explains the difference and when a shared routing layer is useful.

Serverless endpoints. Evaluate the provider’s scaling limits, startup latency, capacity guarantees, and billing model under a burst test. The serverless label alone does not establish how quickly capacity becomes available.

Current self-serve Compute GPU presets use RTX 5090 with 32 GB per GPU; the RTX 4090 fleet is retired. Check the current GPU options and active prices in the console, and verify that your chosen runtime supports the selected hardware.

Engines at a glance

vLLM. An inference engine with continuous batching, paged KV-cache management, and an OpenAI-compatible server. Its capacity and latency depend on the model, hardware, configuration, and workload.

Text Generation Inference (TGI). Hugging Face now lists TGI as being in maintenance mode. Account for its maintenance status when choosing an engine for a new deployment.

TensorRT-LLM. An NVIDIA inference toolkit with hardware-specific optimizations. Verify support for your model and configuration, then benchmark it rather than assuming it will be the fastest option.

Ollama. Supports local model serving and concurrent requests. Its concurrency and queue controls depend on available memory and configuration. Test it against your traffic profile instead of assuming a fixed concurrency limit from the product name.

Pick based on traffic profile, model support, and how much tuning you want to own.

Context length, GPU memory, and KV caching

Longer inputs and outputs can increase KV-cache memory and latency. Requirements depend on model architecture, precision, and active requests. Size weights, cache, and workspace together, and test the longest contexts and busiest traffic you plan to support.

  • Retrieval-augmented generation (RAG). Retrieve relevant passages rather than sending an entire collection. Retrieval does not automatically shorten a prompt: cap the retrieved context and evaluate whether it supports the answer.
  • Efficient caching. Paged KV storage manages attention state in blocks. Prefix caching can reuse matching prompt prefixes and reduce prefill work; it does not reuse arbitrary similar passages or remove the cost of generating new output.
  • Quantization. Lower-precision weights, and where supported activations or cache, can reduce memory requirements. The format determines what is quantized. Check engine support and test representative requests.

Quantization, briefly

Lower precision can save memory, but throughput and quality changes depend on the method, kernels, model, and workload. Check supported quantization formats and hardware. Compare against a higher-precision baseline on your own tasks; do not assume quality loss will be small.

Hardware choices: memory bandwidth and capacity

  • Memory first. Budget for weights at the selected precision, KV cache at the target context and concurrency, and engine workspace. Measure peak memory usage and leave headroom. GPU memory is per device; combining cards requires software that distributes the workload.
  • Compute and bandwidth. Prefill and decoding stress hardware differently. Profile the actual workload before choosing more compute, more memory bandwidth, or a different batching configuration.
  • Network placement. Measure client-to-endpoint latency and, for distributed inference, communication between GPUs or hosts. Choose placement and interconnects against the application’s latency and data-handling requirements.

Compare configurations by the work they complete within your latency targets and their total provisioned cost.

Check available locations for the service and model you intend to use. Compute instance locations and Inference Service model availability can differ. For Inference Service, inspect the endpoint details to confirm placement; selecting Global lets Hivenet assign an available location.

Infrastructure costs and cost efficiency

  • GPU hours. Your main line item. Right‑size hardware to model and traffic.
  • Idle time. For a self-managed deployment, use only the scaling and shutdown controls your stack supports. Hivenet Inference Service currently uses fixed replicas without autoscaling. Stop an endpoint when it should be offline, and review the billing effect and capacity available before restarting.
  • Token workload. Long prompts and generated answers increase serving work. Set appropriate output limits and measure actual output length: a maximum-token setting is a cap, not a guarantee that every request generates that many tokens. Streaming changes delivery timing; it does not by itself reduce token generation or provisioned-capacity charges.
  • Batching. Larger active batches can improve utilization while increasing queueing or per-request latency. Sweep concurrency and request rate with realistic input and output lengths, then choose a configuration that meets the latency target.

For fixed-capacity endpoints, compare the provisioned hourly cost with useful completed work. Lower traffic does not automatically reduce the number of replicas you pay for.

For an initial capacity estimate, let D be daily output tokens, T the measured aggregate output tokens per second of one serving replica, and G the GPUs used by that replica. Estimated active replica-hours are D / (T × 3,600); active GPU-hours are that result multiplied by G. Measure T with the same prompt distribution, cache state, concurrency, and latency targets you expect in production.

This estimate describes active serving work, not the bill. Provisioned idle time, peak traffic, spare capacity, failures, storage, networking, and the provider’s billing rules affect actual cost. A daily token average cannot establish whether you have enough capacity for a burst.

If you are evaluating scale-to-zero for an intermittently used model, compare the acceptable first-response delay with the cost of keeping capacity ready. The LLM cold-start guide covers Router request holds, external wake-up signals, and validation of timeouts and scale-down behavior.

LLM performance benchmarking with representative test data

Build test data from the requests the application needs to handle: multiple prompts with short and long inputs, realistic output limits, and the expected language and content mix. Comparing models requires checking generation quality as well as inference speed. Use the same task set and record tokenizer differences instead of treating token-based metrics as directly interchangeable across models.

Keep model and engine versions, precision, sampling parameters, cache state, and hardware fixed when investigating performance bottlenecks. Record how many requests arrive per second and how many are active at once. A concurrency test and an arrival-rate test apply different pressure to an inference system. NVIDIA’s workload guidance explains why input/output lengths and load controls belong in every report.

Sweep the load and plot a latency curve against system throughput. As the number of concurrent requests increases, throughput can rise until a resource saturates; after that, queueing and latency can grow while throughput levels off or falls. Choose the operating range that satisfies your latency constraints. Trying to maximize throughput without a latency target can produce a configuration unsuitable for interactive users.

Use performance benchmarking to compare controlled serving configurations, then load-test the complete application with bursts, failures, and cancellation. Include client networking, routing, retrieval, and storage where they are part of the request path. Overall response latency is the time the application user experiences, which may exceed the engine’s reported generation time. This separates a model’s performance under controlled conditions from the capacity of the deployed service.

Reliability and observability

Pin the model revision, tokenizer, engine version, and serving configuration for each test. Record the workload and measurement boundaries so a later change can be compared with the same baseline.

Track at least:

  • Request rate, queue length, TTFT, TPS
  • GPU memory allocation, KV-cache occupancy and preemptions; prefix-cache reuse when enabled
  • Model inference resource usage (GPU utilization and memory requirements during inference)
  • Error types: OOM, timeouts, 5xx
  • Successful requests meeting latency targets, failed requests, and rejected or canceled work

Use the metrics exposed by your deployed engine version; vLLM’s production metrics reference covers queueing, cache use, preemptions, and latency. Keep client latency separate from engine metrics so network and routing delays remain visible.

Before rollout, test overload, request cancellation, backend failure, and interrupted streams. Bound queues, timeouts, and retries across the client and serving layers. Define an acceptance threshold, a rollback trigger, and a known working configuration. These checks are deployment work to perform, not results established by this guide. The LLM fallback guide separates documented retry, queue, and provider behavior from the tests needed for a deployment.

Alert on sustained breaches of your latency and error targets. If TTFT rises or throughput falls, compare queue depth, request lengths, cache behavior, GPU metrics, and backend health. Those symptoms identify a change worth investigating; they do not isolate its cause.

Security and data residency

Use encrypted connections, rotate keys, restrict access, and define what the application logs and retains. Check the locations used by the endpoint and any retrieval, storage, monitoring, and backup services; an EU endpoint location alone does not establish that the whole application keeps data in the EU.

Configure access before serving traffic.
For a Compute instance, enable the required connection option and verify the server, port, and firewall settings. For Hivenet Inference Service, use the endpoint URL, model value, and API key shown for that endpoint. HTTPS does not replace authentication or access controls.

Build or buy

Choose Compute when you want to configure and operate the instance, runtime, and model server yourself. Choose Hivenet Inference Service when a supported managed model and variant fit your application. Follow the endpoint request instructions, including the exact URL, model value, and organization API key. Test the API routes, request options, and response behavior your application relies on before switching traffic.

Further reading

Source review: November 11, 2026. Engine, hardware, and managed-service claims were checked against the linked official documentation. This guide does not report a deployment benchmark, capacity guarantee, or completed reliability test. Validate the model and software versions you intend to run.

FAQ

What is TTFT and why does it matter?

Time to first token measures the delay from sending a request to receiving its first generated content. Ignore empty stream events or response headers when measuring client TTFT. Also measure completion latency and streamed-output gaps, since a quick start can still be followed by a slow answer.

How many concurrent users can one GPU serve?

There is no fixed user count. Measure simultaneous active requests under your model, precision, context lengths, output lengths, and latency targets. Include queueing and errors in the result; registered users and requests being generated at one moment are different counts.

Does a long context beat RAG?

It depends on the application. RAG retrieves selected evidence, while a long-context request supplies more material directly. Both consume context tokens. Compare answer quality, retrieval failures, latency, and cost; a hybrid may be useful.

Do I need multi‑GPU right away?

Start single GPU if you can. Move to multi‑GPU when memory or throughput demands it. Test parallel modes and watch cache health.

Can I keep data in the EU?

If EU-only processing is a requirement, select an explicitly available EU location and verify the endpoint details. Check the locations and retention settings of the rest of your application as well. In Hivenet Inference Service, Global lets Hivenet assign an available location, so do not use it as an EU-only placement guarantee.

What is LLM inference?

LLM inference uses a trained language model to produce outputs from inputs. In autoregressive text generation, the model estimates next-token probabilities and a decoding method selects tokens. Sampling can choose a token other than the most probable one.

What are the stages of LLM inference?

LLM inference typically involves two stages: the prefill phase, where the input tokens are processed, and the decoding phase, where the model generates output tokens one at a time.

What is the difference between inference and training LLM?

Training updates model parameters using data and an optimization process. Inference uses the trained parameters to produce outputs without performing those training updates.

What are LLM inference engines?

These are software systems designed to efficiently run LLMs for generating outputs, optimizing for latency, throughput, and resource usage.

What is vLLM for?

vLLM serves language-model workloads with features such as continuous batching and paged KV-cache management. Check model and hardware support, then benchmark the deployment against your latency and throughput requirements.

What is the difference between vLLM and LLM?

LLM refers to the large language model itself, while vLLM is an engine or framework for serving LLMs efficiently in production.

Is vLLM faster than Ollama?

There is no universal winner. Compare the same model, precision, hardware, request lengths, and concurrency. Engine features and defaults can affect memory, latency, and throughput.

Why is vLLM so fast?

Continuous batching and paged KV-cache management can improve utilization and reduce wasted cache memory. The resulting speed depends on the workload and configuration; these features do not guarantee lower latency in every deployment.

What does LLM serving mean?

LLM serving refers to deploying and running large language models to respond to user requests in real time or batch modes.

What is an LLM serving engine?

It is a platform or software that hosts and manages LLMs, handling inference requests efficiently.

What is an LLM server?

A server configured to run LLM inference workloads, providing access to model predictions via APIs or other interfaces.

What does LLM as a judge mean?

It refers to using LLMs to evaluate or score outputs, such as assessing model quality or ranking responses.

What is tokens per second?

Tokens per second is a rate. State whether it counts output tokens, input plus output tokens, one response, or all concurrent requests. Use the same tokenizer and workload when comparing results.

How many words is 1,000 tokens?

There is no fixed word count. It depends on the tokenizer, language, punctuation, and content. Count tokens with the model's tokenizer when planning context or cost.

What does a token mean in AI?

A token is a unit represented by a model’s tokenizer. For text, it may correspond to a word, part of a word, punctuation, or bytes; the mapping varies by tokenizer and language.

What is TTFT?

Time to first token (TTFT) is the latency from sending a request to receiving the first generated token.

How to measure TTFT?

By recording the time difference between submitting a prompt and receiving the first output token from the model.

What is the TPOT metric in LLM?

Time per output token (TPOT) measures the average time taken to generate each output token after the first one.

What is a KV cache?

A Transformer's KV cache holds attention key and value tensors computed during prefill and decoding so later tokens can reuse that state.

What is GPU KV cache?

It is the storage of key-value cache data within GPU memory to accelerate LLM token generation.

What is the KV cache in LLM?

The KV cache holds keys and values from previous tokens to efficiently compute attention for new tokens.

What is key-value store cache?

A key-value store maps keys to values for lookup. A Transformer's KV cache stores attention tensors. The shared abbreviation does not make them the same mechanism.

What is continuous batching?

A scheduler updates the active requests between generation iterations as capacity becomes available. This can improve utilization, but queues and memory limits still apply.

What is a continuous batch?

The active batch changes between iterations: completed requests release resources and waiting requests may be admitted. It is not a guarantee of immediate admission or unlimited capacity.

What is the difference between continuous batching and in-flight batching?

TensorRT-LLM documentation also calls in-flight batching continuous or iteration-level batching. Implementations differ; these terms do not describe opposing principles.

What is LLM throughput and latency?

Throughput is how many tokens or requests an LLM can process per second; latency is the time taken to generate responses.

How to reduce latency in LLMs?

Identify whether the delay comes from networking, queueing, prefill, or decoding. Then test the relevant change, such as shorter prompts, reusable prefixes, different batching, or more capacity, against both latency and throughput targets.

What is the biggest problem with LLM?

The main constraint depends on the application. Evaluate response quality, factual errors, security, latency, memory, and cost against defined acceptance criteria.

What is the throughput of an LLM?

It is the number of tokens or requests an LLM can process per second under given conditions.

How to test LLM throughput?

By measuring tokens generated over time under controlled loads and concurrency.

Is LLM CPU or GPU intensive?

It depends on the engine and deployment. GPUs are common for matrix operations, but CPU and hybrid inference are also supported by some engines. Tokenization and other processing may remain on the CPU.

How to increase LLM throughput?

Test batching, serving-engine settings, and additional replicas or GPUs under representative load. Accept a throughput improvement only when response quality, latency, and error rates remain within the application’s requirements.

What GPU to use for LLM?

Choose hardware that fits the weights, KV cache, and runtime workspace and supports the engine’s required kernels. Compare measured performance and cost on your workload. For Compute with Hivenet, check the current GPU reference and live console rather than assuming an older GPU is still offered.

Does every LLM deployment use a GPU?

GPU support depends on the engine and configuration. CPU-only and hybrid execution are also possible; verify backend support instead of assuming every LLM deployment requires a GPU.

Do I need a GPU to run LLM locally?

Some engines support CPU-only or hybrid inference. Whether a GPU is needed depends on the model’s memory requirements and the response time you need. Test the supported backend on the target machine.

Is RTX 4090 good for LLM?

An RTX 4090 can serve workloads that fit its memory and runtime support. It is a hardware example here, not a current Compute offer: Hivenet has retired its RTX 4090 fleet.

‍

Your next workload belongs on Hivenet.

Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.