Guide: batch processing

Batch jobs (translating a product catalog, classifying support tickets, extracting fields from thousands of documents) send the same kind of request many times. The work is simple; the hard parts are throughput, failures and cost. This guide covers all three using the official openai package with base URL https://api14.com/api/v1.

There is no separate batch endpoint: OpenAI's Batch API is not available on api14 (see OpenAI SDK compatibility). You run batches as normal requests with controlled concurrency.

Know your rate limit

Limits are counted in requests per minute (RPM) per API key, capped by your plan:

A key with its own RPM limit gets the lower of the two. Check yours with GET /key (rate_limit.requests is null when the key uses the plan limit), see Balance & key info. Exceeding the limit returns 429 rate_limited with Retry-After: 60.

RPM caps how many requests start per minute, not how many run at once. If an average request takes 4 seconds and your limit is 100 RPM, about 7 requests in flight already reach the limit. Pace requests by start time, and keep concurrency moderate.

A paced, concurrent runner (Python)

import asyncio, json, os, time
from openai import AsyncOpenAI, APIStatusError

client = AsyncOpenAI(base_url="https://api14.com/api/v1", api_key=os.environ["API14_API_KEY"], max_retries=4)

RPM = 100           # your plan or key limit
CONCURRENCY = 10
FATAL = {"daily_limit_reached", "insufficient_balance", "invalid_api_key", "model_not_allowed"}

class Pacer:
    """Spaces request starts evenly so we never exceed RPM."""
    def __init__(self, rpm):
        self.interval, self.next, self.lock = 60 / rpm, 0.0, asyncio.Lock()
    async def wait(self):
        async with self.lock:
            now = time.monotonic()
            self.next = max(self.next, now)
            delay, self.next = self.next - now, self.next + self.interval
        await asyncio.sleep(delay)

SYSTEM = "Translate the user's text to German. Keep product names and units unchanged. Reply with the translation only."

async def translate(item, pacer, sem):
    async with sem:
        await pacer.wait()
        r = await client.chat.completions.create(
            model="openai/gpt-5-mini",
            messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": item["text"]}],
            max_tokens=500,
        )
        return {"id": item["id"], "text": r.choices[0].message.content, "cost": r.usage.cost}

async def main(items, out_path="out.jsonl"):
    done = set()
    if os.path.exists(out_path):
        done = {json.loads(line)["id"] for line in open(out_path)}
    todo = [i for i in items if i["id"] not in done]

    pacer, sem = Pacer(RPM), asyncio.Semaphore(CONCURRENCY)
    with open(out_path, "a") as out:
        for task in asyncio.as_completed([translate(i, pacer, sem) for i in todo]):
            try:
                out.write(json.dumps(await task) + "\n")
            except APIStatusError as e:
                if e.code in FATAL:
                    raise SystemExit(f"Stopping batch: {e.code}")
                print("failed:", e.status_code, e.code)  # logged; rerun to retry

items = [json.loads(line) for line in open("items.jsonl")]  # [{"id": "...", "text": "..."}]
asyncio.run(main(items))

What this does:

• Pacing: the Pacer starts at most RPM requests per minute; the semaphore bounds how many are in flight.
• Retries: the SDK retries 429 and 5xx with exponential backoff, honouring Retry-After (max_retries=4). Upstream provider 429s are retried the same way.
• Fatal codes stop the job: daily_limit_reached and insufficient_balance will not clear by retrying, so the job exits instead of burning through attempts.
• Checkpointing: results are appended to a JSONL file as they finish. Rerunning the script skips ids that are already done, so a crash or a stopped job loses nothing.

In Node, the same structure works with Promise.all over a fixed number of workers pulling from a shared queue, each waiting 60_000 / RPM ms between starts.

Put several items in one request

For short items, one request per item wastes RPM and repeats the instructions every time. Classify 20–50 items per request with a structured output, and you divide request count by the same factor:

schema = {
    "type": "object",
    "properties": {"labels": {"type": "array", "items": {
        "type": "object",
        "properties": {"id": {"type": "string"}, "label": {"type": "string", "enum": ["billing", "bug", "feature", "other"]}},
        "required": ["id", "label"], "additionalProperties": False,
    }}},
    "required": ["labels"], "additionalProperties": False,
}

tickets = "\n".join(f"{t['id']}: {t['text']}" for t in batch_of_30)
r = await client.chat.completions.create(
    model="openai/gpt-5-mini",
    messages=[{"role": "system", "content": "Label each ticket."}, {"role": "user", "content": tickets}],
    response_format={"type": "json_schema", "json_schema": {"name": "labels", "strict": True, "schema": schema}},
)
labels = json.loads(r.choices[0].message.content)["labels"]

Check that every input id came back; resend the missing ones. If finish_reason is "length", the batch was too large for max_tokens, split it. See Structured outputs. For embeddings, pass up to 100 strings in one input array, see Embeddings.

Cut cost with prompt caching

In a batch, the instructions, examples and schema are identical in every request. Put them first and the varying item last, so providers can reuse the cached prefix:

• OpenAI, DeepSeek, Gemini 2.5 and others cache automatically once the prefix is long enough (1,024 tokens for OpenAI). Cache reads cost a fraction of normal input.
• Anthropic models need an explicit cache_control breakpoint on the static part of the prompt.

A long, detailed system prompt with a few worked examples can therefore cost little more than a short one, and usually improves quality. Confirm hits in usage.prompt_tokens_details.cached_tokens. See Prompt caching.

Choose a cheap model, then verify

For translation, classification and extraction, small models are often good enough at a fraction of the price. List text models by input price:

curl -s "https://api14.com/api/v1/models?category=text" \
  | jq -r '.data | sort_by(.pricing.prompt | tonumber) | .[:15][] | "\(.id)  \(.pricing.prompt)  \(.pricing.completion)"'

Then measure before you commit:

1. Take 50–100 representative items and label the expected output by hand.
2. Run them through two or three candidate models and compare accuracy and the summed usage.cost.
3. Multiply cost per item by your total item count to estimate the full job, and check that your balance covers it with GET /credits.

:free variants use a model's free endpoint where one exists, but they have stricter upstream rate limits, so they suit tests more than large batches. See Model ids & variants.

Before a large run

• Create a dedicated key with a daily USD limit so a bug cannot drain your balance. The job then stops with 429 daily_limit_reached. See Rate limits.
• Set max_tokens on every request.
• Run the first 1% of items, inspect the output and cost, then start the rest.
• Failed requests (any 4xx or 5xx) are not charged, so rerunning failures is safe. See Errors.