An OpenAI-compatible API is a service that accepts the Chat Completions HTTP contract so existing OpenAI SDKs keep working after only the base URL and API key change.
The popular advice stops there: swap the connection settings and ship. That's enough for a demo, but it doesn't establish that your parser, agent, retry policy, or budget controls will behave correctly. You may also need a different model identifier, and accepting the request doesn't mean the replacement model honors every field.
Production compatibility means preserving the behaviors your application relies on, not just returning a plausible answer.
Table of Contents
- The Contract Behind an OpenAI Compatible API
- How to Swap the Base URL Without Rewriting
- Streaming and Tool Calls on the Same Channel
- Token Counts and Which Errors to Retry
- Features That Fail a Real Compatibility Test
- Fail Over Without Corrupting Agent State
The Contract Behind an OpenAI Compatible API
Compatibility means the same HTTP conversation, not the same model weights. Your client should know how to authenticate, submit conversation history, consume output, and interpret completion metadata without a provider-specific rewrite.
The Chat Completions API reference documents the v1 REST namespace, bearer authentication, message-based requests, and optional streaming, while describing a policy of avoiding breaking changes within major API versions whenever reasonably possible. Stable transport doesn't imply frozen model behavior, so pin model snapshots where available and evaluate changes before deployment.
What the client already assumes
The SDK sends a request to POST /v1/chat/completions. When the configured base URL already includes /v1, the client appends /chat/completions; adding the complete endpoint as the base URL can produce the wrong path.
The request body contains a model string and a messages array. Each message identifies its role and content, allowing the application to distinguish instructions, user input, assistant replies, and tool results.
OpenAI-compatible request contract
| Contract element | Required shape | What breaks if it differs |
|---|---|---|
| Method and path | POST /v1/chat/completions |
SDK client methods hit the wrong resource or return a not-found error |
| Authentication | Authorization: Bearer followed by an API key |
Calls fail before the model runs |
| Input | messages array with roles |
Multi-turn apps can't represent system, user, and assistant turns correctly |
| Model selection | model string on the request |
Routing and pricing can't be selected per call |
| Streaming | stream: true returns SSE deltas |
UIs wait for the complete answer or misparse chunks |
| Usage | prompt_tokens, completion_tokens, total_tokens |
Budget records and invoices stop matching calls |
| Client libraries | Official OpenAI Python and Node SDKs with a base URL override | Teams need adapters or integration rewrites |
Request acceptance is only the beginning
A non-streaming response should expose choices, the assistant message, and a finish reason. Your application needs to distinguish successful completion from output that stopped because of a limit or a tool handoff.
The old Completions interface accepted a free-form prompt; Chat Completions represents the conversation explicitly. That distinction matters because the application generally supplies its conversation history rather than assuming the server remembers it.
Newer interfaces such as Responses are separate surfaces. A service implementing Chat Completions doesn't automatically implement every SDK method.
Migration boundary: If a required row is missing, treat the endpoint as OpenAI-inspired rather than compatible with your application. Don't plan a base-URL-only migration.
How to Swap the Base URL Without Rewriting
Keep your existing client and change its connection configuration before changing application logic. This isolates transport problems from differences in model behavior.
Use a short, deterministic request first, such as asking the model to return a greeting. Don't start with a production agent that combines retrieval, tools, and structured output, because a failure won't tell you which capability broke.
Make the connection change
Keep the client. Retain the official OpenAI SDK or your existing HTTP implementation. You're replacing the destination, not introducing another conversation abstraction.
Replace the base URL. Use the gateway's documented API root. Let the SDK append its endpoint path, and inspect the final request path if you receive a routing error.
Replace the API key. Load the gateway credential from your secret manager or environment. Leave bearer-header construction to the SDK.
Select an available model. Use the exact identifier in the gateway's model catalog. A provider-prefixed slug can identify the model family or route, but don't assume its prefix guarantees a particular hosting location.
Preserve the payload. Send the same
messages, then add your existingtemperatureandtoolsfields where the selected model supports them. Preserve field values during testing so you can attribute differences correctly.Check both response modes. Confirm that the ordinary call returns a usable message and usage information. Then enable streaming and verify that your existing consumer completes normally.
The connection change leaves the application's conversation logic in place.
Equivalent requests with the same configuration
Set GATEWAY_BASE_URL, GATEWAY_API_KEY, and MODEL_ID from the gateway's documentation. The curl example appends the endpoint explicitly; the SDK examples append it internally.
curl
curl "${GATEWAY_BASE_URL%/}/chat/completions" \
-H "Authorization: Bearer ${GATEWAY_API_KEY}" \
-H "Content-Type: application/json" \
-d "{\"model\":\"${MODEL_ID}\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hello.\"}]}"
Python
import os
from openai import OpenAI
client = OpenAI(
base_url=os.environ["GATEWAY_BASE_URL"],
api_key=os.environ["GATEWAY_API_KEY"],
)
response = client.chat.completions.create(
model=os.environ["MODEL_ID"],
messages=[{"role": "user", "content": "Say hello."}],
)
print(response.choices[0].message.content)
Node
import OpenAI from "openai";
const client = new OpenAI({
baseURL: process.env.GATEWAY_BASE_URL,
apiKey: process.env.GATEWAY_API_KEY,
});
const response = await client.chat.completions.create({
model: process.env.MODEL_ID,
messages: [{ role: "user", content: "Say hello." }],
});
console.log(response.choices[0].message.content);
A unified gateway can consolidate credentials, balance management, and invoicing instead of leaving your team to reconcile separate provider dashboards. Openbase is one option: it provides provider-prefixed model access, per-key limits, and usage logs covering model, provider, tokens, latency, time to first token, and cost.
The economic reason to preserve this portability is concrete: a 2025 NBER working paper, using API usage data from OpenRouter and Microsoft Azure, found open-source language models were 90% cheaper than comparable closed-source models at the same intelligence level. That finding isn't a quote for your workload, but it explains why teams want to evaluate alternatives without rebuilding their clients.
A successful greeting establishes connectivity and basic transport compatibility. It doesn't certify the application.
Streaming and Tool Calls on the Same Channel
A chat UI can display text as it arrives, but stream: true isn't just a flag for animated typing. Your client is consuming a protocol, and a gateway must preserve that protocol through its proxy.
The streaming API documentation describes incremental, data-only Server-Sent Events and streamed tool-call deltas. The gateway must preserve event order, event boundaries, and completion semantics instead of buffering the upstream output into a final JSON document.
The important distinction is between delivering partial output and preserving the meaning of each event.
Tool arguments arrive unfinished
Ordinary text appears in content deltas. Tool requests arrive through delta.tool_calls, potentially with fragments of a function name and fragments of its JSON arguments.
Say an assistant is preparing a create_ticket call. The first argument fragment might contain the beginning of an object and a title, while a later fragment supplies the description and closes the object. Trying to parse each fragment independently will fail even though the completed arguments are valid.
The consumer needs to accumulate fragments by tool-call index, preserve the tool-call identifier, and parse arguments only when the call is complete. Never mix fragments from separate calls into the same buffer.
- Preserve identity. Carry tool-call IDs through to tool results so the model can associate each result with its request.
- Separate event types. Don't render tool arguments as assistant prose or treat text deltas as executable instructions.
- Validate before execution. Parse the completed arguments and validate them against the tool's schema before invoking application code.
For an accompanying explanation of streamed responses, use the video below.
A reconnect can duplicate work
Suppose the application executes create_ticket, then loses its connection before receiving the assistant's final acknowledgement. Replaying the conversation without checking execution state may create another ticket.
Use an application operation identifier and deduplicate inside the tool. A gateway request idempotency key helps only when the gateway explicitly implements its semantics; adding an arbitrary header doesn't make external side effects idempotent.
Record whether the failure occurred before generation, during text output, during argument assembly, or after tool execution. Those are different recovery states, not interchangeable connection errors.
For interactive performance, log time to first token and the delay between chunks. A low total duration can still hide an unresponsive start, while a healthy initial response can conceal long pauses halfway through.
Don't retry a partial stream until you know whether a side effect already happened.
Token Counts and Which Errors to Retry
Your budget code should trust metered usage, not the length of the visible answer. Input tokens represent material supplied to the model; output tokens represent generation.
The OpenAI token guide describes input and output accounting, including reasoning tokens that don't appear in the final answer but still count toward output usage and are billed as output tokens. In Chat Completions, applications consume the corresponding prompt_tokens, completion_tokens, and total_tokens fields.
Count the work you don't see
A short answer can still involve substantial internal reasoning. If your dashboard estimates output usage by tokenizing only the displayed text, it can understate billed consumption.
Keep the normalized usage fields and any provider-specific breakdown needed to explain them. Track prompt growth separately from output growth so you can distinguish an expanding conversation history from a more expensive generation pattern.
A disconnect also doesn't mean inference was free. The upstream service may have generated billable tokens before your client lost the response, so record usage at the attempt level whenever it's available.
Classify the failure before retrying
Rate limits can constrain requests and tokens over different time windows. Reducing request concurrency won't necessarily solve a token-throughput limit if each request contains a large conversation.
The retry decision must combine the failure class with execution state. The table describes operational handling, not a guarantee that replaying a request is harmless.
Which HTTP failures are safe to retry
| Failure | Retry | Why |
|---|---|---|
| 429 rate limit | Yes, after Retry-After when supplied |
Temporary quota pressure, not necessarily a bad request |
| 408 or connection timeout | Yes, with backoff and jitter, subject to execution-state checks | Transport may succeed on another attempt |
| 5xx upstream | Yes, bounded and subject to execution-state checks | Provider fault rather than invalid client input |
| Partial stream after tokens | Only with idempotency and execution-state checks | Tokens may already be billed and tools may have run |
| 400 bad request | No unchanged replay | The body will fail the same validation again |
| 404 unknown model or path | No unchanged replay | Routing or the model identifier needs correction |
Honor Retry-After when available. Otherwise use bounded exponential backoff with jitter, an overall deadline, and a retry budget shared across client and gateway layers.
Billing check: Ask whether the gateway charges for generated-but-undelivered tokens, how it reports failed attempts, and whether requests producing no tokens incur inference charges.
Provider-aware logs should connect each attempt to its tokens, cost, outcome, and final client response. That's how you detect a router that improves completion rates while steadily increasing spend through repeated generation.
Features That Fail a Real Compatibility Test
Basic chat and streaming can work while an application remains unsafe to migrate. A July 2026 comparison of 12 providers by PlugSky found those features worked broadly, but parallel tool execution and tool selection differed, JSON-mode support varied substantially, embeddings were absent from many providers, and error formats were the most inconsistent category.
The useful question isn't which vendor passed a greeting test. It's which behaviors your application requires and whether every permitted route preserves them.
Use the distinction below to separate portable transport from capabilities that require explicit testing.
Treat capability support as a routing input
What compatible usually does and does not guarantee
| Feature | What the provider checks showed | Treat it as |
|---|---|---|
| Basic chat messages | Worked broadly | Portable |
| Streaming SSE | Worked broadly | Portable if termination is tested |
| Parallel tool calls | Differences in execution and tool choice | Negotiate per model |
| Strict JSON or structured output | Support varied substantially | Negotiate, don't assume |
| Embeddings | Absent from many providers | Not part of the chat contract |
| Error response bodies | Most inconsistent category | Test before classifying retries |
An agent expecting parallel tool requests may stop early when a replacement model returns only a subset of the expected calls. Don't interpret a syntactically valid tool response as proof that the agent's workflow completed.
Structured output needs an equally precise test. A loose JSON mode that wraps an object in explanatory prose isn't equivalent to schema enforcement, and a valid object can still omit required business fields if the service ignores your schema.
Embeddings require their own endpoint and model support. Even when available, changing the embedding model without validating vector compatibility can make an existing retrieval index unusable.
Test invariants rather than sample answers
Build fixtures that assert machine-observable behavior instead of exact wording. Run them against each model-provider route you intend to permit.
- Schemas and lifecycle. Validate request handling, response structure, finish reasons, stream termination, and cancellation behavior.
- Agent execution. Check tool-call IDs, argument assembly, parallel-call behavior, and the association between calls and results.
- Output guarantees. Submit adversarial structured-output prompts and verify refusal and content-filter payloads.
- Accounting and errors. Check usage arithmetic and exercise malformed requests, unknown models, throttling, and upstream failures.
A different error envelope can hide a client failure inside a success-shaped body, leading your application to parse it as content or retry it incorrectly. Assert both the HTTP status and the error body your classifier reads.
Classify each tested capability as portable, negotiated, or not eligible for automatic routing. A missing feature should produce a deliberate rejection or an application-approved fallback, not a silent downgrade.
Fail Over Without Corrupting Agent State
Fail over when the upstream problem is transient and the application can safely resume or restart the operation. Don't use provider switching to bypass policy refusals, conceal invalid requests, or replay an agent whose external actions are already committed.
Provider diversity has a concrete availability benefit: the Presenc reliability tracker reported 20 OpenAI model-API incidents totaling 72.7 hours during a 90-day 2026 window, compared with 76.5 hours for Anthropic, with only 36 minutes of overlap. That monitoring window supports diversifying upstreams, not assuming that every incident becomes a failed request or that another provider is always available.
Use an operational gate before switching destinations.
Define acceptable fallback classes
A reachable model can still be an unsuitable fallback. Its context window may not hold the request, its tokenizer may change capacity estimates, or its reasoning behavior may reduce accuracy on the task.
Define fallback classes in application terms. A drafting service might permit any evaluated model that preserves tone and required fields; a financial extraction workflow might require a model validated against its schema and reconciliation tests.
Keep capability and policy constraints in the routing decision. Don't shorten the prompt without disclosure, remove unsupported tools, or weaken output validation to make a fallback request succeed.
Make recovery observable
Your gateway should expose attempt-level traces with the requested model, selected upstream, failure category, execution state, and outcome. Operators need to distinguish a healthy initial attempt from a successful request that required repeated upstream work.
- Use circuit breakers. Temporarily stop sending ordinary traffic to an unhealthy upstream, then probe recovery instead of repeatedly consuming application deadlines.
- Constrain route selection. Keep fallbacks inside evaluated capability, context, security, and policy boundaries.
- Inject failures. Test interrupted streams, delayed responses, malformed output, and unavailable upstreams with replayable traces.
- Preserve recovery state. Store enough application state to resume intentionally rather than relying on a fresh model to infer what already happened.
A router may pool provider capacity, select routes by priority, and move work away from degraded services. Those mechanisms improve availability only when the application can explain which model answered, what work was repeated, and whether the result still satisfies its requirements.
Enable automatic failover for capabilities your conformance suite marks portable. Pin routes that fail required checks until you've implemented and tested an explicit adapter or fallback.
The production standard is straightforward: your team should be able to explain not only why a request succeeded, but why its output, side effects, and bill remain correct after the upstream changes.
Openbase offers a unified Chat Completions-compatible endpoint, consolidated model billing, automatic upstream failover, and per-request usage logs. Visit Openbase to evaluate it with your existing SDK, then run your streaming, tool-execution, and billing conformance tests before routing production traffic.