Behind the Stack, Ep 10 - Batched Endpoints

Cutting LLM Costs with Batched Endpoints: What They Are and How to Self-Host Them

Introduction: The Cost Challenge in LLM Workloads

Running LLMs at scale can be expensive. Whether you’re building customer-facing chatbots, document extraction pipelines, or research tools, token usage can balloon into thousands of dollars quickly. While infrastructure teams often focus on throughput optimizations (batching requests on the GPU, prefix caching, etc.), there’s another lever to pull: endpoint design. One of the most powerful - and under-discussed - endpoint types is the batched endpoint. Instead of prioritizing instant responses, batched endpoints trade latency for cost, cutting your LLM bill in half (or more in some cases). 

In this blog, we’ll cover: 

• What batched endpoints are and how they differ from standard APIs 
• How providers reduce costs behind the scenes 
• Advanced optimization strategies (spot instances, prefix caching, request reordering) 
• How to self-host your own batched endpoint 

What Are Batched Endpoints?

Most LLM APIs now offer “batched” or “asynchronous” endpoints. The idea is simple: 

Standard endpoint → Input tokens: $1 / million; Output tokens: $2 / million; Latency: seconds. 

Batched endpoint → Input tokens: $0.50 / million; Output tokens: $1 / million; Latency: hours (up to 24h). 

The trade-off is lower cost, but there are no guarantees of real-time performance. This makes batched endpoints perfect for offline or naturally asynchronous jobs: daily/weekly document ingestion, large-scale data extraction pipelines, model evaluation, training-time data labeling. But they are not suitable for: user-facing chatbots, real-time dashboards, or anything needing sub-second latency. Think of batched endpoints as the LLM equivalent of cold storage in cloud infra: slower, but much cheaper.

How are providers able to offer up to 50% cost reductions?

At first glance, halving the token price seems too good to be true. But when you dig into GPU economics, it makes sense. 

1. Flattening GPU Demand Curves: GPU demand spikes when users are awake and drops at night. 

• Without batching → Peaks = congestion, Troughs = idle GPUs. 
• With batching → Flexible jobs move to troughs, flattening the curve. 

Providers avoid congestion, fill idle time, and improve utilization. 

2. Spot Instances: Self-hosters can use preemptible GPUs up to 80% cheaper than on-demand. Perfect for batch jobs where latency is flexible. 

• Example: On-demand A100: $3/hr vs Spot A100: $1/hr. 

3. Request Reordering + Prefix Caching: Group similar requests, reuse shared prefixes, and dramatically cut compute. Instead of computing the prefix 1,000 times, compute once and reuse.

How to Self-Host Batched Endpoints

You can replicate batched endpoints yourself with the right design. 

1. Use Priority-Aware Inference Engines: Engines like vLLM let you tag requests as high-priority (real-time) or low-priority (batch). This ensures real-time requests aren’t blocked. 

2. Add a Smart Queuing Layer: To replicate 24h contracts, add a queue in front. Queue tracks IDs, forwards low-priority requests, and promotes them if SLA is about to expire (e.g. after 23h 30m). You can even create tiers: Real-time (standard price), 24h batch (50% off), Indefinite batch (10x cheaper, no SLA). 

3. Leverage Spot Instances: Run low-priority jobs on a spot GPU pool. If they fail, retry later. If they exceed SLA, promote them to real-time. Workflow: User request → Queue → Spot pool (cheap) → Retry on failure → Promote to real-time if timeout.

Conclusion: Batched Endpoints as a Core Optimization

Batched endpoints are more than a discounted API tier - they’re a core strategy for scaling LLM workloads. Providers use them to smooth demand and maximize GPU utilization, and teams self-hosting can combine queues, spot instances, and prefix caching to design custom SLAs and pricing tiers. If your workloads don’t need instant answers, batched endpoints can easily cut your costs in half - or more. For document pipelines, ingestion jobs, or large-scale evaluation, this should be one of the first levers you pull.