Companies spent $37 billion on generative AI in 2025, up from $11.5 billion in 2024, a 3.2-fold year-over-year increase, according to Menlo Ventures' 2025 State of Generative AI in the Enterprise. That kind of spend changes the question. Model access isn't just which model answers best in a test. It's who controls the keys, what happens during an outage, and whether finance can prove what a successful task cost.
Table of Contents
- Direct Model Calls Versus an LLM API Gateway
- Why Model Access Outgrew a Single Vendor
- How One Request Crosses the Gateway
- Failover That Does Not Change the Task
- Why Cheaper Tokens Can Cost More
- Where to Place the Gateway
- Decide Whether a Gateway Belongs Here
Direct Model Calls Versus an LLM API Gateway
You're choosing between two shapes for the same feature. You can call each vendor directly from your application, or you can put an LLM API gateway in front of them. The gateway gives you one OpenAI-compatible chat completions endpoint, one key, one balance, and one invoice, while it forwards work to upstreams such as OpenAI, Anthropic, Google, xAI, and DeepSeek.
| Concern | Direct provider APIs | LLM API gateway |
|---|---|---|
| Credentials and billing accounts | Separate key and invoice per vendor | One key, one balance, one invoice |
| Request and response shape | Vendor-specific SDKs and bodies | OpenAI-compatible chat completions, including stream and tools |
| Adding a model | New SDK, account, and error handling | Provider-prefixed slug; same request body |
| Rate limits | Each provider quota stands alone | Pooled capacity across upstreams serving that model |
| Failure handling | That feature fails with the vendor | Retries and failover by error class |
| Billing trigger | Vendor-specific, often split dashboards | Prepaid per-token; no charge when no tokens are produced |
| Per-call visibility | Split consoles | Model, provider, tokens, latency, time to first token, and cost |
Direct calls look simpler at first because there's nothing between your code and the vendor. That simplicity holds for one model in one service. It breaks when a second team needs a different model, a vendor throttles you at peak, or someone asks which feature spent the money. Each provider brings its own account, SDK habits, dashboard, and outage plan.
What changes in your code
With a gateway, your existing streaming code usually changes only the base URL and key. The request body, stream handling, and tool calls stay in the shape your code already sends. The router orders eligible providers by priority, and every call records model, provider, input tokens, output tokens, reasoning tokens, latency, time to first token, and cost.
That single ledger matters at current scale. Enterprise LLM spending increased from $3.5 billion in late 2024 to $8.4 billion by mid-2025, based on Menlo Ventures' survey of 150 technical leaders, as reported in HPCwire's coverage of the Menlo Ventures report. When spend grows that fast, finance doesn't want five dashboards that don't reconcile. They want one record they can audit.
The practical question isn't whether a gateway has more features. It's whether your application should own provider selection, retries, and metering in every service, or whether one layer should own those jobs once.
Why Model Access Outgrew a Single Vendor
Model access stopped being a library choice and started behaving like infrastructure. Stack Overflow's 2025 Developer Survey collected more than 49,000 responses from developers in 177 countries and found that 84% of respondents were using or planning to use AI tools in their development workflow, up from 76% in 2024. The same survey reported that 51% of professional developers used AI tools daily.
Daily use changes the tolerance for failure. A script someone reruns by hand can survive a failed call. A feature customers open every morning cannot. Once a model sits inside shipped software, it takes on uptime, cost, and support expectations, the same way a database or a payment API does.
The multi-model split forces the design
The same survey found that OpenAI's GPT models were used by 82% of developers for development work, while Anthropic Claude Sonnet models were used by 45% of professional developers. The overlap is the point. Teams do not crown one model and retire the rest. They keep one for code, test another on long documents, and hold a third in reserve for price or latency.
Say your support copilot starts on one vendor. Product wants to test a second model on hard tickets. Your platform team wants a fallback during an incident. Finance wants to know why three invoices grew in the same month. A hard-coded integration handles none of that cleanly, because each new question becomes new application code.
A second provider does not automatically make the system safer. A fallback model can answer differently, refuse differently, or call tools differently, so failover is a behavior you design, not a switch you flip. The question is where that design lives.
A prototype can hard-code one vendor and survive. A production system with daily users needs a place where routing, keys, and spend are controlled once.
An LLM API gateway absorbs four jobs at that point. It standardizes one request format, centralizes authentication and billing, routes requests across models, and keeps a durable record of token usage. You can review the available model catalog in one place instead of tracking each vendor's naming and access rules inside your services.
That does not mean every team needs many models on day one. It means adding the second model should not be a rewrite, and failing over to it should not be a surprise. When usage is routine, provider-neutral access protects you from decisions you haven't made yet, including a model swap you didn't plan this quarter.
How One Request Crosses the Gateway
The application speaks one language, and the gateway books an upstream that can take the trip.
The diagram below follows that single request from your application to an upstream model and back.
The five hops
- Your app sends the familiar body. It posts a chat completions request, including streaming and tool or function calls, to the gateway base URL with one key.
- The gateway checks the key. Per-key limits let you isolate staging from production, cap a noisy internal tool, or revoke a leaked credential without rotating every vendor secret you own.
- It resolves the model slug. A provider-prefixed slug points to OpenAI, Anthropic, Google, xAI, or DeepSeek, and the gateway selects an upstream by priority, health, and pooled capacity rather than one vendor quota.
- It forwards and streams back. Tokens return in the same response shape your client already parses. The gateway fails over mid-flight only for transient failures, not for a request that was malformed from the start.
- It meters and logs the attempt. It records input, visible output, and reasoning tokens, then writes one row per attempt with provider, model, HTTP status, retry count, time to first token, total latency, token usage, final outcome, and exact cost.
Notice what doesn't change. Your request and response bodies stay the same, so you don't adopt a new SDK for every model you test. Your streaming parser, tool-call handler, and error display keep working because the contract at the edge is stable.
What is new sits behind that edge. You fund a prepaid balance, and the gateway charges along a single path only when tokens are produced. A failed attempt that emits no tokens shouldn't become a mystery line item that nobody can tie to a user-visible result.
The video below walks through the request flow so you can see where authentication, routing, and metering happen.
Two details decide whether this path is safe in production. The first is retry timing, because retrying too fast can make an overloaded provider worse. The second is tool-call deduplication, because retrying a request that already triggered an action can repeat that action.
Failover That Does Not Change the Task
A backup provider isn't a reliability strategy by itself. Independent research on public LLM services found that failures typically lasted 0.5 to 3 hours, with median recovery times of 1.23 hours for the OpenAI API and 0.77 hours for the Anthropic API. Providers also showed different failure-isolation characteristics, so they didn't fail in identical ways.
Treat failover as a behavior contract. It should say which errors trigger a retry, how many attempts are allowed, and what a retry is never allowed to change.
| Condition | Retry or fail over | Reason |
|---|---|---|
| 408 or client timeout | Yes, with capped backoff | Transient; immediate retry amplifies load |
| 429 rate limit | Yes, honor retry hints and shift upstream | Same-provider storms burn quota |
| 5xx from upstream | Yes, to the next healthy provider | Provider failure, not a bad prompt |
| Connection failure | Yes, within the attempt budget | Transport loss before a final answer |
| 400 bad request | No | Deterministic; the same body fails again |
| 404 unknown model or route | No | A retry cannot invent the model |
| Failure after a tool call | Only with deduplication | Replay can repeat a side effect |
| Partial tokens already sent | Only if duplicate output is acceptable | The client may already have rendered text |
Backoff, breakers, and attempt budgets
Retry 408, 429, 5xx, and connection failures with truncated exponential backoff with jitter. Cap the delay and cap the number of attempts per request. Immediate retries turn a 429 into a stampede that burns the same quota, stretches tail latency, and can keep a sick upstream sick.
Keep a circuit breaker and a concurrency limit per provider and model pair. When one pair crosses an error or latency threshold, shift eligible traffic instead of hammering it. The error reference for gateway responses should match this table, so application teams can tell a retryable upstream problem from a request they must fix.
The mid-generation problem
Streaming makes failover harder. If tokens or a tool call have already left the gateway, replaying a non-idempotent request can duplicate side effects. Say the first attempt created a ticket, sent an email, or charged a card through a tool. A blind replay can do it twice.
Use a request identifier, a bounded attempt budget, and tool-call deduplication. Automatic replay after partial output is acceptable only when your client tolerates duplicate text and no side effect has run yet. If either condition fails, return a clear error and let the application decide.
Practical rule: A second HTTP 200 doesn't prove the task succeeded. Prove that the answer, tool call, and side effects match what the first model would have produced.
Separate availability from correctness. A backup model may use a different tool schema, refuse borderline prompts more often, accept a smaller context window, or produce weaker answers. It can return successfully and still fail the user's task.
Measure the contract with task success, retry recovery rate, provider transitions, tokens produced before failure, and p95 and p99 latency. Run semantic equivalence tests on your real prompts before you trust a fallback route. Test tool names, required fields, refusals, and long-context cases, not just a short greeting.
Why Cheaper Tokens Can Cost More
Move the cost question from sticker price per million tokens to what a successful task consumes. The rate on a pricing page works like the fuel-economy figure on a car sticker: it describes the unit, not the trip. Per-token prices keep falling across providers, yet workload design still decides most of the bill.
A cheaper token doesn't guarantee a cheaper product. Teams respond to lower prices by sending more requests, lengthening context, retrying more often, or shifting ordinary traffic onto reasoning models built for hard problems. Each choice is reasonable in isolation. Together they can raise total spend while every price chart points down.
The reasoning token trap
Hidden reasoning tokens are often the first cost teams miss. They can occupy context capacity and be billed as output tokens even when they never appear in the final API response. Your user sees a short, tidy answer. Your bill reflects the long internal work that produced it.
That flips a common comparison. A reasoning model that prints two sentences after extensive hidden reasoning can outspend a non-reasoning model that prints a full paragraph and thinks less. If you compare only visible output length, you'll pick the wrong model, then wonder why the invoice disagrees with your spreadsheet.
Research on token-budget-aware reasoning reports that constraining reasoning budgets can reduce token costs while largely maintaining answer correctness. You don't need to guess whether a cap helps your workload. Set a budget, run your real prompts, and compare task success at each level. The right budget is the lowest one that keeps quality where your users need it.
What belongs on one ledger row
Vendor invoices often split the evidence you need. One dashboard shows tokens, another shows latency, and a third shows a monthly total with no feature attached. A gateway should keep the full story on one row so product, engineering, and finance argue from the same facts.
- Input tokens: What your prompt, system instructions, retrieved documents, and conversation history sent.
- Visible output tokens: The text or tool calls your application received and your user could see or act on.
- Reasoning tokens: Hidden computation billed as output, reported separately from visible text.
- Retry count and provider: How many attempts ran, where they ran, and whether a provider transition happened.
- Latency and time to first token: Whether the cheaper route made users wait longer for the first visible progress.
- Billing status: Whether the attempt produced tokens and was charged, or failed with no tokens and no charge.
That last line prevents ghost charges. Bill only when tokens are produced, and don't charge failed attempts that emit none. If a retry produces tokens before failing, the row should say so, because that partial work still carries a cost you need to explain.
A worked contrast without a sticker price
Take one prompt from your own product. Say it's a request to classify a support ticket, draft a reply, and choose the next action through a tool call. Run it on a reasoning model and on a non-reasoning model. Leave the published rate aside for the moment and open your gateway logs first.
| Measure | Reasoning model, short visible answer | Non-reasoning model, longer answer |
|---|---|---|
| Visible output tokens | Short | Longer |
| Hidden reasoning tokens | High, billed as output | Low or none |
| Billed output tokens, visible plus reasoning | Visible count plus a large reasoning count | Visible count plus little or no reasoning |
| Time to first token | Often slower; hidden reasoning runs before the first visible token | Often faster; output starts streaming sooner |
| Retries before success | Retry count from your log row | Retry count from your log row |
| Cost per successful task | Total billed cost across attempts divided by successful tasks | Total billed cost across attempts divided by successful tasks |
Read the table in order. The reasoning model wins on visible brevity, yet billed output includes the hidden work, so brevity alone settles nothing. User experience stays unsettled too: a model that thinks longer before streaming can show a worse time to first token even when total latency looks acceptable.
Retries belong in the same calculation. If one model needs a second attempt on a share of hard tickets, those attempts consume input tokens again and may produce billable output. A model with a higher sticker price and a cleaner first pass can cost less per resolved ticket than a cheaper model that stumbles and repeats work.
Judge models by cost per successful task rather than cost per million tokens. A failed task that a human must redo is the most expensive output you can buy.
Quality has to sit beside cost, or the comparison is rigged. Grade both models on the same tickets. Did the classification match your taxonomy? Did the tool call carry the right fields? Did a person accept the draft without rewriting it? A cheaper route that lowers acceptance pushes cost onto your support team, where it won't appear in the token report.
Controls finance and product can set
Once the ledger is honest, controls become straightforward. Set per-key limits so one environment, customer, or internal tool can't drain the balance unnoticed. Give production, staging, and experiments separate keys, and you'll know which surface caused a spike without digging through prompt logs.
Add alerts when reasoning tokens spike for a feature. A prompt change, a longer retrieved document, or a model swap can multiply hidden computation overnight. The alert should name the model, provider, and key, so the owner can act before the monthly close.
Use daily usage rollups for trend checks and a monthly invoice that matches those rows for reconciliation. If the invoice total can't be rebuilt from per-call rows, finance will keep a shadow spreadsheet, and you'll have two versions of the truth again. Check the published rates on the gateway pricing page against your own rows for input, output, and reasoning tokens.
Route by task rather than by habit. Send classification, extraction, and short drafts to the model that passes your quality bar at the lowest cost per successful task. Reserve heavy reasoning for prompts where your tests show it buys correctness you can measure. Revisit the split when prices or models change, because last quarter's bargain can become this quarter's waste.
One confusion trips up many teams: context feels free once a model accepts it. Every retrieved passage you paste into the prompt is input you pay for on every attempt, retries included. Trim history, summarize old turns, and retrieve fewer, better passages. Context discipline often saves more than a negotiated rate, and it improves latency at the same time.
Where to Place the Gateway
You can run this layer in three places, and each choice decides who owns routing policy when something breaks at night. The comparison below comes before the rule, because the trade-offs explain the rule.
| Pattern | Who operates it | Fits when | Main trade-off |
|---|---|---|---|
| Library or sidecar in the app | Application team | One service, early prototype | Routing and spend policy drift per service |
| Shared self-hosted gateway | Platform or infrastructure team | Data must stay in a chosen network | The team owns scaling, keys, and incidents |
| Managed multi-provider gateway | Vendor, with platform oversight | Several models, one key and invoice | Extra hop and a contractual trust boundary |
An in-process library or sidecar sits beside one service. It's enough for a prototype and easy to skip, which is exactly the problem later. Each service sets its own retries, limits, and provider order, so policy drifts and nobody can say which version handled a given call.
A shared self-hosted gateway centralizes routing, keys, and logs inside a network you control. That fits when prompts or logs must stay in a chosen boundary. Your platform team then owns scaling, patching, key storage, and the incident when the gateway itself becomes the outage.
A managed gateway removes the stack of provider accounts and invoices. It adds a network hop, a data-processing agreement, and trust in someone else's failover. Judge that hop with time to first token as well as total latency, because streaming users feel the wait for the first token most.
Requirements every placement must keep
Whichever pattern you pick, scope keys by environment and decide where prompts and logs live before traffic arrives. Keep independent concurrency limits per upstream so one hot model can't exhaust every provider. Keep the per-attempt log rows, or you can't prove reliability after the fact.
That logging discipline has a governance root. NIST's AI Risk Management Framework specifies that system functionality and component behavior should be monitored in production, and NIST defines reliability as an item's ability to perform as required without failure over a specified interval. Provider, model, status, retry count, latency, and outcome per attempt turn that expectation into evidence.
The gateway belongs on the path every model call already takes, not as an optional helper a team can bypass.
If a service can skip the gateway with a direct key, your routing, spend limits, and logs are advisory. Put the gateway where bypassing it requires a deliberate, reviewable change.
Decide Whether a Gateway Belongs Here
Adopt an LLM API gateway when more than one provider is already in your design, even if the second one exists only as a fallback. Adopt it when an outage measured in hours would take a customer feature down with no acceptable manual workaround. Adopt it when finance can't reconcile separate dashboards, keys, and invoices into one number they trust.
Stay on a direct API when one model is a hard product requirement and no substitute passes your quality tests. Stay direct when your prompts can't cross a specific contractual boundary that a managed layer would add. Stay direct when nobody on your team will own routing policy, because an unowned gateway becomes another piece of drift.
A four step pilot
- Inventory what you have. List current models, keys, owners, and one month of spend by feature, including the internal tools everyone forgets.
- Move one feature. Point one OpenAI-compatible feature at the gateway base URL and key, and leave the request body, streaming, and tool handling alone.
- Set the failover contract. Configure provider priority, retry only transient errors, never retry 400 or 404, cap attempts with backoff, and deduplicate tool calls before replay.
- Compare for a week. Measure per-call cost, task success, and tail latency against the old path before moving anything else.
Use per-key limits during the pilot so the test can't spend without a ceiling. Use the per-call usage log as the evidence in step four. If a provider transition happened, the log should show which provider finished the task, what it cost, and whether quality checks still passed.
Say the pilot feature is your ticket copilot. Keep the prompt fixed, keep the evaluation set fixed, and change only the path. That discipline tells you whether differences came from the gateway or from prompt edits your team shipped in the same week. Without it, you'll credit or blame the gateway for changes it never made.
If your need is one key, one balance, and one invoice, with streaming and tools on an OpenAI-compatible endpoint and automatic failover across OpenAI, Anthropic, Google, xAI, and DeepSeek, Openbase matches that path. Pilot it with the four steps above rather than migrating every service at once. One feature gives you clean before and after evidence, and it limits the blast radius if your fallback model behaves differently on real prompts.
Watch for the failure modes that stall pilots. Teams move traffic but keep direct keys alive everywhere, so half the calls bypass the new logs. Teams set a fallback model without testing tool schemas, so the first real outage becomes a behavior change. Teams compare sticker prices instead of cost per successful task, so the pilot report pleases finance and misleads product.
The success test is simple to state and hard to fake. A provider incident or a model swap should change a route, not your application. Your code keeps sending the same request, your users keep getting tasks completed to the same standard, and your logs show exactly what the detour cost.
Put one production feature behind a single OpenAI-compatible endpoint with one key, one balance, and per-call logs for tokens, latency, and cost, then judge the pilot by task success instead of HTTP status alone. Openbase gives you that path across OpenAI, Anthropic, Google, xAI, and DeepSeek, with failover and explicit reasoning-token accounting built into the same request shape. Start your pilot at Openbase and bring your toughest week of real prompts with you.