Operations

Rate limits

Limits are enforced per API key using a sliding window, so a burst from one service cannot starve another.

Where limits apply

Per key. Request rate is counted against the key that made the call, not the account.
Per key, per month. An optional spend ceiling, set when you create or edit a key.
Upstream. Providers apply their own limits. Hitting one triggers a failover rather than an error, when another provider can serve the model.

Recognising a limit

Exceeding the rate returns 429 with type rate_limit_error.

{
  "error": {
    "message": "Rate limit exceeded for this API key",
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded"
  }
}

Handling it

Back off exponentially with jitter rather than retrying immediately. A tight retry loop extends the window it is trying to escape.

async function withBackoff(fn, attempts = 5) {
  for (let attempt = 0; attempt < attempts; attempt += 1) {
    try {
      return await fn();
    } catch (error) {
      if (error.status !== 429 || attempt === attempts - 1) throw error;
      const base = 2 ** attempt * 250;
      await new Promise((resolve) => setTimeout(resolve, base + Math.random() * 250));
    }
  }
}

Spread work across keys

Because limits are per key, giving each service its own key both isolates their throughput and makes the request log far easier to read.

Needing more

High-volume workloads can have their limits raised. Get in touch with the traffic shape you expect and we will size it with you.