Batch API

How it works

• Upload a JSONL file of requests, create a batch pointing at it, poll until the batch reaches a terminal status, download the results file.
• The queue accepts work even while no GPU is serving the model, then drains when one comes up. If a live request would get a 503, a batch is how you leave the work with us anyway.
• The completion window is 24h: whatever hasn't run by then is refunded and the finished part is delivered.

The input file

One JSON object per line, uploaded with purpose batch (limits: 1,000 lines, 5 MB):

{"custom_id": "job-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "Qwen/Qwen2.5-7B-Instruct", "messages": [{"role": "user", "content": "Classify: great product"}], "max_tokens": 10}}
{"custom_id": "job-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "Qwen/Qwen2.5-7B-Instruct", "messages": [{"role": "user", "content": "Classify: broke in a day"}], "max_tokens": 10}}

• custom_id must be unique per line; it keys your results.
• url must be /v1/chat/completions; body takes the same fields as the live endpoint, and each line may use a different model.
• Validation is all or nothing: one bad line rejects the whole batch with the line number, and nothing is queued or charged.
• stream is ignored inside batches; results always arrive as complete responses in the output file.

Example

# 1. upload the JSONL file
curl https://tokenkiln.com/api/v1/files \
  -H "Authorization: Bearer tk_..." \
  -F purpose=batch \
  -F file=@batchinput.jsonl
# -> {"id": "file_...", ...}

# 2. create the batch
curl https://tokenkiln.com/api/v1/batches \
  -H "Authorization: Bearer tk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file_...",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'
# -> {"id": "batch_...", "status": "in_progress", ...}

# 3. poll until status is terminal
curl https://tokenkiln.com/api/v1/batches/batch_... \
  -H "Authorization: Bearer tk_..."

# 4. download results
curl https://tokenkiln.com/api/v1/files/file_.../content \
  -H "Authorization: Bearer tk_..."

Lifecycle

Every batch object carries request_counts (total / completed / failed) so a poll loop can report progress. Requests that fail with an engine error land in error_file_id with their custom_id; they cost nothing. Cancel with POST /api/v1/batches/{id}/cancel, or from the Batches page.

Billing

• At creation, the discounted worst case for every line is reserved atomically. If your balance cannot cover the whole batch, creation fails with 402 and nothing is queued.
• Each request settles individually against real token counts as it finishes, at the prices in force when the batch was created. Later catalog repricing never touches queued work.
• Cancellation and expiry refund every request that has not run. Failed requests are never billed.