You trained a great model. It scores well on your eval set, the LoRA adapter is 40 MB, and it runs in your notebook. Then product asks the real question: can it serve two million users with a p99 latency under two seconds, without a GPU bill that bankrupts the company? The notebook that answered one request in 800 milliseconds collapses at ten concurrent users, and a naive deployment would need a fleet of GPUs sitting at 5% utilization. The hard part of fine-tuning was never the training.
Serving LLMs efficiently is a distinct engineering discipline from training, with its own bottlenecks and its own toolkit. The difference between a naive deployment and an optimized one is not 20%. It is often 10–20× in throughput per GPU, which is the difference between a viable product and an impossible bill. This is where most fine-tuning projects quietly die.
This is Part 2 of a two-part series. Part 1 covered methods, data, and training. Part 2 covers everything after you have a trained model: how inference actually works, the optimizations that make it cheap, how to serve many fine-tuned variants from shared hardware, and how to monitor quality once real users arrive.
TLDR
- LLM inference is bottlenecked by memory bandwidth and the growing KV cache, not raw compute.
- The wins come from a stack: continuous batching and PagedAttention (vLLM) for 10–20× throughput, quantization (AWQ/GPTQ/FP8) to shrink the model, and speculative decoding to cut latency.
- Serve many LoRA adapters from one base model with multi-LoRA serving instead of one deployment each.
- Autoscale on queue depth, cache prompt prefixes, and monitor TTFT, tokens-per-second, GPU utilization, and cost per token.
- Most importantly, monitor output quality drift, because a model can degrade with perfect uptime.
LLM Fine-Tuning series
This is Part 2 of a 2-part series. Part 1: Methods, Data, and Training · Part 2: Production Deployment and Monitoring (you are here)
Why LLM inference is hard#
To optimize serving you have to understand where the time goes. Generation has two distinct phases, and they have opposite performance characteristics. We explore this further in How Inference Engines Serve Models.
Prefill processes the entire input prompt in parallel to produce the first token. It is compute-bound and fast per token because the whole prompt goes through the GPU at once. Decode generates the output one token at a time, each step depending on the last. It is memory-bandwidth-bound: for every single token, the GPU must read the entire model's weights from memory. This is why a long output costs far more than a long input, and why LLM serving is fundamentally a memory-bandwidth problem, not a compute problem.
The KV cache eats your memory#
During decode, the model reuses the attention keys and values of all previous tokens. Recomputing them every step would be wasteful, so they are stored in the KV cache. The problem is that this cache grows with every token of every concurrent request, and it grows fast. Long contexts and many users can make the KV cache larger than the model weights themselves. Managing it is the central challenge of high-throughput serving, and the optimizations below are largely about doing it efficiently.
The optimization stack efficient serving needs#
| Feature | What it fixes | Typical impact |
|---|---|---|
| Continuous batching | GPU idle while requests wait for each other | Several times higher throughput vs static batching |
| PagedAttention (vLLM) | Wasted KV cache memory from fragmentation | Far higher concurrency on the same GPU |
| Quantization (AWQ / GPTQ / FP8) | Model too big / too few requests per GPU | 2–4× smaller model, more concurrency, faster decode |
| Speculative decoding | Decode is one slow token at a time | Lower latency by generating several tokens per step |
| Multi-LoRA serving | One GPU deployment per fine-tuned variant | Hundreds of adapters served from one base model |
| Prefix caching | Re-processing identical system prompts | Large savings on shared prompt prefixes |
| Tensor parallelism | Model bigger than one GPU's memory | Splits one model across GPUs for capacity and speed |
The rest of Part 2 walks this stack, then turns to deployment topology and monitoring. The path from a trained adapter to live traffic looks like this:
Continuous batching, the foundational win#
Naive serving processes requests one at a time, or in static batches where the whole batch waits for the slowest request to finish before any new request starts. Both leave the GPU mostly idle. Continuous batching (also called in-flight batching) instead adds and removes requests from the running batch at every decoding step: the moment one request finishes, a waiting one takes its slot. The GPU stays saturated.
This single technique is the biggest throughput lever in LLM serving, often several times over naive batching. It is built into modern serving engines, so you get it by choosing the right engine rather than implementing it yourself.
PagedAttention and vLLM#
The KV cache traditionally needed a contiguous block of memory per request, sized for the maximum possible length. That wastes enormous amounts of memory on fragmentation and over-allocation. PagedAttention (Kwon et al., arXiv:2309.06180), the technique behind vLLM, applies the operating-system idea of virtual memory paging to the KV cache: it stores the cache in non-contiguous fixed-size blocks and maps them with a page table.
The result is near-zero memory waste and far more concurrent requests per GPU. PagedAttention plus continuous batching is why vLLM and similar engines (TensorRT-LLM, SGLang, TGI) deliver order-of-magnitude throughput gains over a naive model.generate() loop. Do not write your own serving loop. Start from one of these engines.
# Serving a fine-tuned model with vLLM, including its LoRA adapter
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", enable_lora=True, max_loras=8)
out = llm.generate(
prompts=["Classify this ticket: my payment failed."],
sampling_params=SamplingParams(max_tokens=16, temperature=0),
lora_request=LoRARequest("support-classifier", 1, "./support-classifier-lora"),
)
Quantization for inference#
Training quantization (Part 1's QLoRA) is about fitting training in memory. Inference quantization is about serving more requests per GPU and decoding faster. It stores weights in lower precision so each decode step reads fewer bytes from memory, directly attacking the memory-bandwidth bottleneck.
There are a few main methods. GPTQ (Frantar et al., arXiv:2210.17323) and AWQ (Lin et al., arXiv:2306.00978) are post-training methods that compress to 4-bit with small, careful calibration to preserve quality. FP8 is an 8-bit floating-point format with native support on recent GPUs, a popular sweet spot of speed and accuracy. Quantization to 4-bit roughly quarters the model's memory footprint and meaningfully increases throughput, usually with negligible quality loss for well-calibrated methods. You must still measure quality on your task, because aggressive quantization can degrade it.
Speculative decoding#
Decode is slow because it is sequential: one token per forward pass of the full model. Speculative decoding (Leviathan et al., arXiv:2211.17192) breaks that limit. A small, fast "draft" model proposes several tokens ahead, and the large model verifies them all in a single forward pass. When the draft is right (which it often is for easy tokens), you get multiple tokens for the cost of one step, with output mathematically identical to the large model alone. It reduces latency without changing quality, and is increasingly built into serving engines.
Multi-LoRA: serving many fine-tuned models#
Here is where Part 1's choice of LoRA pays off enormously. If you fine-tuned ten variants (one per customer, per language, or per task), the naive approach is ten full model deployments, ten times the GPUs. Because a LoRA adapter is tiny and shares the same frozen base, you can instead serve all of them from one base model loaded once, swapping the small adapter per request.
Multi-LoRA serving (the idea behind S-LoRA, Sheng et al., arXiv:2311.03285) keeps one base model in GPU memory and a library of adapters that are applied dynamically per request, batching requests for different adapters together. This is how a platform serves hundreds of customer-specific fine-tunes economically. The marginal cost of a new fine-tuned variant becomes a 40 MB file, not a new GPU. If your product offers per-customer customization, this single capability decides whether the economics work.
Prefix caching#
Many requests share an identical prefix: a long system prompt, a fixed instruction block, or few-shot examples. Recomputing that prefix's KV cache for every request is pure waste. Prefix caching stores the KV cache of common prefixes and reuses it, so only the unique part of each prompt is processed. For applications with large shared system prompts, this is a substantial latency and cost win, and it is a built-in feature of modern engines.
Deployment topology for billions of requests#
Optimized single-node serving is necessary but not sufficient. Reaching billions of requests is a distributed systems problem on top of an inference problem, and the broader picture is in System Design for AI Products. A single request travels through several layers before it ever touches a GPU:
Tensor parallelism for sharding large models#
When a model is too big for one GPU even quantized, tensor parallelism splits each layer's matrices across multiple GPUs that work on one request together, connected by fast interconnect. This adds capacity (fit a bigger model) and can lower latency, at the cost of cross-GPU communication. It is configured in the serving engine, not your application.
Autoscaling on the right signal#
GPU instances are expensive, so you scale the fleet with demand. The mistake is scaling on CPU or memory. For LLM serving the signal that matters is queue depth and time-to-first-token, meaning how long requests wait before the engine picks them up. Scale up when the queue grows and TTFT rises, scale down when GPUs idle. Because cold-starting a GPU node and loading a model takes time, keep a warm buffer and scale ahead of demand, not after it.
Routing, load balancing, and tiering#
In front of the GPU fleet sits a router that load-balances across replicas and can do smarter things: route by adapter (send requests for the same LoRA to the same replica to exploit its warm adapter and prefix cache), tier by model size (a small fast model for easy requests, a large one for hard), and enforce per-tenant rate limits. This routing layer is where a lot of real-world cost efficiency is won.
Batch the batchable, stream the rest
Not all traffic is latency-sensitive. Route interactive chat to low-latency serving and push bulk jobs (overnight classification, summarization of a backlog) to a separate high-throughput, high-batch-size queue. Mixing them forces one configuration to compromise both; separating them lets each run at its optimum.
Monitoring is the part that never ends#
A deployed LLM is not done; it is a system whose quality can silently drift as inputs change, dependencies update, or the world moves on. Monitoring an LLM spans two layers that teams often conflate: system health and output quality. We cover the broader philosophy in Observability for AI Systems and the team-scale basics in MLOps Basics for Small Teams.
System and performance metrics#
These tell you whether the service is healthy and affordable:
- Time to first token (TTFT): the latency users feel before anything appears. The primary interactive-experience metric.
- Time per output token (TPOT) / tokens per second: how fast the response streams after it starts.
- Throughput: total tokens per second across the fleet, which is your capacity.
- GPU utilization and KV cache usage: are you actually using the hardware you pay for, or running it idle? Low utilization means wasted money; KV cache saturation means you are about to reject requests.
- Cost per token / per request: the metric that decides whether the product is viable. Track it per model and per customer.
- Error and queue-rejection rates: requests dropped because the engine was saturated.
Quality and drift metrics#
System metrics can be perfect while the model quietly gets worse. This layer is what most teams miss:
- Output quality scoring: sample production traffic and score it with an LLM-as-judge or human review against your task rubric. This is the same evaluation discipline from Part 1 and Add Evaluation to an AI Feature, now run continuously.
- Drift detection: watch the distribution of inputs and outputs over time. A shift in incoming queries, or in response length and refusal rate, is an early warning that the model is operating outside the distribution it was trained on.
- Guardrail hits: rate of safety filter triggers, refusals, and policy violations, both for safety and as a signal of misuse or prompt attacks.
- User feedback signals: thumbs up/down, regenerations, and escalations to a human are the cheapest real-world quality signal you have. Log them and trend them.
Closing the loop: canary, A/B, and the data flywheel#
Monitoring is only useful if it drives action. Roll out a new fine-tune as a canary to a small slice of traffic and compare its quality and latency metrics against the incumbent before going wide. Run A/B tests to prove a new model is actually better, not just different. And feed real production interactions (especially the failures users flagged) back into the training set for the next fine-tune. That flywheel, where production data continuously improves the model, is what separates a one-shot fine-tune from a system that compounds over time.
Common mistakes and tradeoffs#
- Do: start from a real serving engine (vLLM, TensorRT-LLM, SGLang) for continuous batching and PagedAttention
- Do: quantize for inference (AWQ/GPTQ/FP8) and measure quality on your task after
- Do: serve many LoRA variants from one base with multi-LoRA serving
- Do: autoscale on queue depth and TTFT, with a warm buffer for cold starts
- Do: monitor output quality and drift, not just latency and GPU usage
- Do: roll out new fine-tunes as canaries and feed production failures back into training
- Avoid: writing your own generate loop, which leaves 10–20× throughput on the table
- Avoid: one full deployment per fine-tuned variant when multi-LoRA exists
- Avoid: aggressive quantization without measuring task quality
- Avoid: autoscaling on CPU/memory instead of LLM-specific signals
- Avoid: monitoring only system health, because a model can degrade with perfect uptime
- Avoid: mixing latency-sensitive chat and bulk jobs in one serving config
The defining tradeoff of serving is latency versus throughput versus cost, and you cannot maximize all three. Bigger batches raise throughput and cut cost per token but add latency; aggressive quantization cuts cost but risks quality; more replicas cut latency but raise cost. The job is not to win every axis but to set the dials deliberately for each traffic class, measure relentlessly, and move them as the data tells you.
Key Takeaways#
- LLM inference is memory-bandwidth-bound, dominated by sequential decode and a KV cache that grows with every token and user. That is the bottleneck every optimization targets.
- Continuous batching plus PagedAttention (vLLM and peers) deliver order-of-magnitude throughput over a naive loop. Never write your own serving loop.
- Inference quantization (AWQ/GPTQ/FP8) shrinks the model and speeds decode; speculative decoding cuts latency with identical output.
- Multi-LoRA serving runs hundreds of fine-tuned variants from one base model, the capability that makes per-customer customization economically viable.
- Reaching billions of requests is a distributed systems problem: tensor parallelism for big models, autoscaling on queue depth and TTFT, and smart adapter-aware routing.
- Monitor two layers, system health (TTFT, TPOT, GPU utilization, cost per token) and output quality (scoring, drift, guardrails, user feedback), then close the loop with canaries, A/B tests, and a production-data flywheel.
FAQ#
Why is my fine-tuned model so slow to serve, and how do I fix it?
Almost certainly you are using a naive generation loop. The fix is a real serving engine like vLLM, TensorRT-LLM, or SGLang, which give you continuous batching and PagedAttention out of the box, often 10–20× more throughput per GPU than model.generate(). Start there before anything else.
How do I serve many different fine-tuned models cheaply?
If they are LoRA adapters over the same base model, use multi-LoRA serving: load the base once and apply the small adapters per request. You serve hundreds of variants from one deployment instead of one deployment each, making the marginal cost of a new fine-tune a tiny file rather than a new GPU.
Does quantizing for inference hurt quality?
Well-calibrated 4-bit methods (AWQ, GPTQ) and FP8 usually have negligible quality impact while substantially improving throughput and memory. "Usually" is doing work, though. You must measure on your own task, because aggressive quantization can degrade specific behaviors. Quantize, then re-run your evaluation set.
What latency metrics should I actually track?
Time to first token (TTFT) for the perceived responsiveness, and time per output token (TPOT) or tokens-per-second for streaming speed. Pair those with throughput and GPU utilization for capacity, and cost per token for viability. TTFT is the one users feel most directly.
How do I know if my deployed model is getting worse?
System metrics will not tell you, because a model can degrade with perfect uptime. Continuously sample production outputs and score them (LLM-as-judge or human review), watch input/output distributions for drift, and trend user feedback like regenerations and thumbs-down. Set alerts on quality, not just latency.
How does fine-tuning fit into a continuous improvement loop?
Production is the best source of training data. Log real interactions, especially the failures users flag, curate them into your dataset, and fold them into the next fine-tune. Then canary and A/B the new model before full rollout. That flywheel is how a fine-tuned system keeps improving instead of decaying. See Part 1 for the training side.
