AI Explainedby Milad Fahmy

AI Explained

Batch APIs: cut your LLM bill in half by waiting

Batch APIs run LLM requests in the background at 50% of the normal price. How they work, their limits, and which jobs to move to a batch.

Milad FahmyNov 10, 20262 min read

On this page
  1. Live calls vs work nobody is waiting on
  2. Half the price
  3. How a batch works
  4. Things to get right
  5. When to batch, and when not to
  6. Quick tip

You can cut your LLM bill in half just by waiting. Most of the work an app sends to a model doesn't need an answer this second, and batch APIs charge half price for exactly that kind of work.

Live calls vs work nobody is waiting on

Most apps call the model and wait for each answer. That's right for chat, where a person is watching the screen. But plenty of jobs have nobody waiting on them:

  • running evals
  • classifying a big dataset
  • embedding your docs

A batch API takes all those requests at once and processes them asynchronously, in the background.

Half the price

OpenAI, Anthropic and Google all charge 50% of the standard price for batch requests. Same models, half the cost.

How a batch works

You upload a JSONL file with one request per line. Each line has a custom_id and the normal request body:

{"custom_id": "q-1", "method": "POST", "url": "/v1/responses", "body": {...}}
{"custom_id": "q-2", "method": "POST", "url": "/v1/responses", "body": {...}}

Then you submit the batch, and it moves through states: validating, in progress, then completed. The promise is a 24-hour window, and Anthropic says most batches finish in under an hour.

Things to get right

  • Results come back in any order. Match them to requests by custom_id, never by line number.
  • Batches are big. OpenAI allows 50,000 requests per batch. Anthropic allows 100,000 requests or 256 MB, whichever comes first.
  • Rate limits are separate. At OpenAI, batch rate limits are separate from the normal per-model limits, so a huge batch doesn't eat into your live traffic.
  • Discounts stack. On Anthropic, the batch discount stacks with prompt caching, so a shared system prompt gets cheaper still.
  • Some requests can fail or expire. Check each result's status, and resubmit only the ones that didn't succeed.

When to batch, and when not to

Batch it Call it live
Evals and test suites Chat and assistants
Bulk content generation Anything a user is waiting on
Moderation of a backlog Real-time features
Embedding a document set

A common pattern: collect the day's work, submit one batch, and read the results the next morning.

Quick tip

Start with your eval suite. It runs often, nobody is waiting on it, and it's an easy 50% saving.

Not urgent? Batch it, pay half.

Sources

#batch api#llm cost#openai#anthropic#gemini#evals

Watch