Five GPU Cost Quick Wins You Can Ship This Sprint

Most teams are paying 40-70% more for GPU inference than they need to, and almost none of that is the model's fault. It's idle hardware, single-request serving, full-precision weights, and on-demand pricing for work that has no SLA. Our deep dive on GPU economics, "GPU Economics: Why Your Inference Bill Is 4x," explains why the bill compounds the way it does. This is the field-notes companion: five concrete changes, each shippable inside one sprint, each with a rough savings range and a way to verify you didn't quietly degrade quality.

Treat this as a checklist, not a research project. Pick the wins in order, measure before and after, and keep a quality eval running the whole time. The order matters: the early wins are free wins, and the later ones need a guardrail.

1. Right-size and consolidate idle GPUs

The fastest money is the GPU that's powered on and doing nothing. Pull two weeks of utilization and you will almost always find dev/staging endpoints pinned to dedicated cards, plus production replicas provisioned for a peak that arrives twice a day. Consolidate low-traffic models onto shared nodes, scale replicas to actual concurrency, and set aggressive scale-to-zero on anything that isn't latency-critical. This alone is typically 20-40% off the bill with zero model changes.

2. Turn on continuous batching

If you're still serving one request per forward pass, you're leaving the biggest single unlock on the table. Continuous (in-flight) batching injects new requests into the running batch the moment a slot frees up instead of waiting for the whole batch to finish. On vLLM and TGI this is a config flag, and it routinely delivers 2-4x throughput at the same GPU count — which is a 50-65% drop in cost per token. It's the default in modern vLLM, so for many teams the win is just adopting a current serving stack.

# vllm serve config
model: meta-llama/Llama-3.1-8B-Instruct
max_num_seqs: 256          # in-flight requests batched together
max_num_batched_tokens: 8192
enable_chunked_prefill: true   # keeps decode latency low under load
gpu_memory_utilization: 0.90   # leave headroom for KV cache spikes
# continuous batching is on by default in current vLLM

3. Quantize to FP8 or INT8

Eight-bit quantization cuts VRAM roughly in half and speeds inference ~1.8x, which lets you serve the same traffic on smaller or fewer GPUs — typically 20-30% off. On H100, FP8 is near-lossless: published benchmarks show ~33% faster output tokens/sec with sub-0.1% accuracy drift, and INT8 measured a 0.04% delta from BF16 on standard benchmarks, which is noise. The catch is you must prove it on your workload, not someone's leaderboard.

4. Move batch and non-latency work to spot

Embeddings backfills, nightly evals, document ingestion, fine-tuning, offline scoring — none of this has a user waiting on it, and all of it is probably running on on-demand GPUs. Move it to spot/preemptible capacity for 50-70% off on that slice. The work needs to be checkpointable and retry-safe, which most batch jobs already are or can become with a few lines.

• Tag every GPU workload as latency-critical or not — only the second bucket goes to spot.
• Checkpoint long jobs so a preemption costs minutes, not the whole run.
• Keep latency-critical inference on on-demand or reserved capacity; never gamble your SLA to save pennies.

5. Add per-request token accounting

You can't optimize what you can't see. Logging tokens and estimated cost per request is the cheapest win here and it's what makes the other four measurable — and defensible to finance. Attribute cost by route, customer, and model so you can find the one feature quietly burning 30% of the budget. This is the foundation; ship it first even though it's listed last.

PRICE_PER_1K = {"in": 0.0005, "out": 0.0015}  # your blended GPU cost

def log_token_cost(req_id, route, usage):
    cost = (usage.prompt_tokens / 1000) * PRICE_PER_1K["in"] \
         + (usage.completion_tokens / 1000) * PRICE_PER_1K["out"]
    logger.info("token_cost", extra={
        "req_id": req_id, "route": route,
        "in_tok": usage.prompt_tokens,
        "out_tok": usage.completion_tokens,
        "usd": round(cost, 6),
    })
    metrics.increment("inference.usd", cost, tags=[f"route:{route}"])
    return cost