One endpoint, many models: how the Astra Gateway routes

The problem with direct provider calls

Every application ends up with the same three problems: provider SDKs drift, keys multiply, and a single upstream outage takes the product down. Adding a second provider usually means adding a second code path, not removing the first.

A gateway centralises that. Applications speak one OpenAI-compatible protocol; the gateway owns provider credentials, routing, retries and accounting.

The shape of the system

client ──► gateway ──► channel A (provider 1)
                 ├─► channel B (provider 2)
                 └─► channel C (self-hosted vLLM)

gateway owns: keys, quotas, routing, fallbacks,
              streaming, latency + cost accounting

A channel is a routable upstream plus its credentials and limits. The same model name can exist on several channels; the router picks one by weight and health, and falls through when a channel fails.

Streaming is the part people underestimate

A chat completion that streams token-by-token feels fast even when total latency is unchanged. The failure mode is buffering: one proxy layer accumulates the whole response, then flushes it at once. The client sees a spinner, then a wall of text.

The gateway streams HTTP/2 chunks with buffering disabled, so the first token reaches the client as soon as the upstream emits it. That is also why a warm connection pool matters: TLS and TCP setup on every request adds latency that no model change can hide.

Measured against direct upstream calls, warm connection pooling cut request latency by roughly 40% in our setup.

What the operator gets

• One key per customer, with quota and model allow-lists enforced server-side.
• Fallbacks: a dead channel is skipped instead of returning a 500.
• Latency and token accounting per request, which is what makes pricing possible.
• A single place to add or remove an upstream without touching application code.

Trade-offs

A gateway is another hop and another thing to operate. It also sees plaintext prompts unless you add end-to-end encryption, which most OpenAI-compatible clients do not support. Run it close to your users, keep logs minimal, and treat the gateway as production infrastructure rather than a convenience wrapper.