What overhead does an AI gateway actually add?

A well-built gateway adds single-digit to low-tens of milliseconds of forwarding overhead. LiteLLM Rust measured about 0.7 ms p99 against a mock in a July 22, 2026 head-to-head, Envoy AI Gateway measured about 2 ms against real H100 inference under streaming, and Portkey's real-world figure with features enabled is closer to 20 to 40 ms. All of these are small against an LLM inference baseline of 500 ms to 5 seconds, so gateway overhead is typically a low-single-digit percentage of total request time.

Which AI gateways handle streaming with under 20 ms of added latency?

Envoy AI Gateway is the verified one: its July 2026 benchmark measured about 2 ms of overhead against real GPU inference with streaming enabled. LiteLLM Rust, Portkey, and Bifrost each report sub-5 ms p99 overhead, which implies the same headroom, but that specific comparison ran non-streaming against a mock, so treat it as a strong signal rather than a confirmed result. Kong (P95 24.07 ms) and Portkey with guardrails on (20 to 40 ms) sit at or above the 20 ms line depending on configuration. Confirm any of these against your own provider before relying on the number.

Why are vendor benchmarks misleading?

Because most of them measure proxy forwarding against a mock upstream, which removes the provider's response time, the largest real-world variable. A headline like 40 times faster is measuring microseconds of proxy work on top of a mock, not the request a user experiences. Vendor benchmarks also tend to quote p50 rather than p99, and to test without streaming, both of which flatter the number. Read every benchmark with its upstream, its percentile, and its streaming mode attached.

Which metric should I optimize for?

TTFT under streaming for user-facing workloads, because it is what the user feels, and gateway-added overhead measured as gateway latency minus direct-to-provider latency at matched concurrency for evaluating the gateway itself. Optimizing raw proxy throughput against a mock rarely changes the real request, because provider TTFT dominates. The metric that matters for a security gateway is whether policy evaluation and audit commit stay off the critical path.

Does inline policy enforcement hurt latency?

Not when it is designed correctly. Purpose-built classifiers run in tens of milliseconds, and running identity, policy, and classification in parallel means the added latency is the slowest single check rather than their sum. Caching decisions per identity and route, and committing the audit record asynchronously, keep enforcement off the hot path. The design mistake that does hurt latency is using a full model as a judge synchronously, which adds seconds per check.

How does gateway overhead compare to provider variance?

Provider variance usually exceeds gateway overhead. Second-to-second P99 swings at major providers can move by hundreds of milliseconds, while a well-built gateway adds around 1 to a few tens of milliseconds. This is why shaving proxy overhead below a millisecond has little effect on real request latency, and why the honest way to present a gateway's cost is as a fraction of a provider call whose own timing fluctuates by more than the gateway's entire budget.

AI Gateway Latency Benchmarks: Reading the 2026 Numbers Without Getting Fooled by the Mock Upstream

TL;DR

AI gateway overhead is small and mostly mock-measured. Envoy AI Gateway's July 2026 test is the exception: about 2 ms against real GPU inference with streaming, comfortably under a 20 ms budget. LiteLLM Rust, Portkey, and Bifrost each post sub-5 ms p99 overhead, but against a mock, non-streaming. Provider TTFT runs 500 ms to 5 seconds, so gateway overhead stays a small fraction of the request either way. The engineering question that matters is keeping policy evaluation off the critical path, not shaving microseconds off the proxy.

Five orders of magnitude, and why that is suspicious

The published AI gateway latency numbers for 2026 span five orders of magnitude, from Bifrost's claimed 11 microseconds of overhead to LiteLLM's older figure of 40 milliseconds of median overhead with callbacks enabled. That range alone should make an engineer suspicious, because it is not measuring the same thing. Almost every one of these benchmarks clocks proxy forwarding against a mock upstream, which removes the one variable that dominates real requests: the model provider's own response time.

I want to read the 2026 numbers the way a principal engineer should, with the methodology attached to each figure, and then make the point the benchmarks obscure. A well-built gateway adds single-digit to low-tens of milliseconds. A model's time to first token runs from hundreds of milliseconds to several seconds. The gateway is a rounding error on the provider, and the interesting engineering question is not shaving microseconds off the proxy but keeping policy evaluation off the critical path.

The metrics that actually matter

Benchmarks get compared incorrectly because the metric is left implicit. Name it every time.

TTFT, time to first token, from request send to the first token back. Dominated by prompt processing and provider queue. This is what a user experiences as responsiveness.
ITL or TPOT, inter-token latency or time per output token, the steady-state gap between tokens after the first.
Total request latency, which is TTFT plus output tokens times ITL.
p50, p95, p99, the percentiles. p99 is where gateway and provider tail pain shows.
Throughput, sustained requests per second at a stated concurrency.
Gateway-added overhead, the only number this article cares about, defined as gateway end-to-end latency minus direct-to-provider latency at matched concurrency.

If a benchmark does not tell you which of these it measured and against what upstream, the number is not comparable to anything.

The 2026 numbers, with their caveats attached

Here is what the published benchmarks actually claim, and what each one measured.

LiteLLM's June 2026 Rust launch post first reported the Python proxy adding roughly 7.5 ms of overhead per request, with the Rust rewrite targeting about 0.05 ms. A follow-up head-to-head published July 22, 2026 puts real numbers on the same comparison: p99 added latency of about 0.7 ms for LiteLLM Rust, 2.3 ms for Portkey, 4.5 ms for Bifrost, and 257.7 ms for LiteLLM's own Python v1, at n=5,000 requests against a local deterministic mock with no logging callbacks or spend tracking enabled. Non-streaming, mock upstream, single-host, and the rig is checked in, which is the right way to publish a benchmark.

Bifrost, from Maxim, claims 11 microseconds of gateway overhead at 5,000 requests per second on a t3.xlarge, and 0.99 ms in a 500-RPS test where it reports LiteLLM at 40 ms. The caveat is load-bearing: the 11-microsecond figure is a Bifrost-only stress test against roughly 10 KB payloads with a mock upstream, and it explicitly excludes upstream response time. The 0.99 ms versus 40 ms comparison uses a 60 ms mock OpenAI response. It is a real proxy-forwarding number and it is not an end-to-end number.

Kong's July 2025 benchmark ran Kong 3.10, Portkey OSS, and LiteLLM against a WireMock upstream at 400 virtual users with 1,000-token prompts, reporting a baseline of 29,005 RPS at P95 24.07 ms and P99 30.35 ms, and stating results only as relative percentages, Kong faster than the others by stated multiples. Mock upstream, proxy-only, no absolute per-gateway overhead in milliseconds.

Portkey markets sub-1 ms latency for its bare proxy, and community reports put real-world overhead with guardrails and routing enabled closer to 20 to 40 ms. Cloudflare AI Gateway, an edge proxy, shows community figures around 10 to 50 ms added.

Envoy AI Gateway is the exception worth naming on its own, because its July 2026 benchmark is the one on this list that is not measuring a mock. Broadcom's VMware Cloud Foundation team ran a three-hour endurance test against four real H100 GPUs serving streamed LLM traffic under a bursty, agentic workload, holding 190 concurrent users before the system saturated at 224, a GPU compute ceiling rather than a gateway limit. Added overhead measured roughly 2 ms, about 0.01% of end-to-end latency, against an average time to first token of 0.103 seconds, and stayed flat through peak saturation. That is a streaming, real-upstream, third-party-run number, which makes it the strongest single answer on this page to "which gateway holds under 20 ms of added latency with streaming on."

The methodology traps

Five traps turn these numbers into misleading comparisons.

Streaming versus non-streaming. A non-streaming test folds TTFT into total latency, and a gateway that buffers the response instead of passing server-sent events through adds latency a non-streaming test cannot see. Measure TTFT under streaming.

Mock upstream versus real provider. LiteLLM, Bifrost, and Kong all benchmark against mocks. That is legitimate for isolating gateway cost, and it deletes the single largest real-world variable, which makes a headline like 40 times faster look decisive when the delta is microseconds sitting on top of a multi-hundred-millisecond provider call.

Cold start. First-request compilation and connection-pool warmup inflate p99, especially on serverless or edge gateways. Report warm and cold separately.

p50 not p99. Vendors quote the median. The tail is where retries, connection churn, and provider variance appear, and where an inline gateway either helps through failover or hurts through an extra hop.

Provider variance dwarfs gateway overhead. Second-to-second P99 swings at OpenAI or Anthropic exceed a well-built gateway's entire overhead budget. A gateway adding around 1 ms to a call whose provider TTFT varies by hundreds of milliseconds is not the latency story, and LiteLLM's own post concedes that gateway overhead is usually a small fraction of total model latency.

Which gateways stay under 20 ms with streaming on

One gateway has a verified answer, and the rest have a strong implied one. Envoy AI Gateway's July 2026 endurance run measured roughly 2 ms of added overhead against real H100 GPU inference with streaming enabled, comfortably inside a 20 ms budget, and it is the only figure on this page tested against a real model rather than a mock. LiteLLM Rust, Portkey, and Bifrost each report sub-5 ms p99 overhead in the July 22, 2026 head-to-head, which would clear 20 ms with room to spare if the same held against a real, token-by-token response, but that specific test ran non-streaming against a mock, so treat it as a strong signal rather than a confirmed one. Kong's P95 of 24.07 ms sits just over the line, and Portkey's own real-world figure with guardrails and routing enabled, 20 to 40 ms, straddles it depending on which features are on. The honest read: pick a gateway from the sub-5 ms group if raw forwarding speed under 20 ms matters, but confirm it with your own streaming test against your actual provider, because a mock-upstream number never verifies the streaming case by itself.

Put the numbers in proportion

Frame every gateway figure against the baseline. LLM inference takes 500 ms to 5 seconds. Fast, small models post TTFT under roughly 600 ms on public leaderboards like Artificial Analysis, and frontier reasoning configurations run far higher because of extended reasoning. Against that, a gateway overhead of single-digit to low-tens of milliseconds is between a fraction of a percent and a few percent of the request. The engineering that matters is not the proxy's forwarding speed. It is whether the gateway's own policy work stays off the critical path, which is the reason inline enforcement is affordable at all.

Where policy evaluation adds latency, and how to keep it off the path

Policy and classification cost is real and manageable if the design is right. Purpose-built classifiers run in the tens of milliseconds; practical inline budgets sit around sub-100 ms for an input rail. The trap is using a full model as a judge, which adds 5 to 11 seconds per check and belongs nowhere near a synchronous request. Three patterns keep evaluation invisible:

Parallel evaluation. Run identity, policy, and content classification concurrently rather than in series, so total added latency is the slowest check, not the sum.
Caching. Cache policy decisions per identity and route, and use semantic caching on repeated prompts to remove the upstream call entirely.
Async audit commit. Emit the signed per-decision record off the hot path with a write-behind commit, so producing the audit record never adds to TTFT.

Enforce inline, commit evidence asynchronously. That sentence is the whole latency design for a security gateway.

DeepInspect

DeepInspect is a stateless proxy at the AI request boundary, and it is built to the proportion the benchmarks obscure. Its end-to-end enforcement overhead measures under 50 ms in internal testing, against an LLM inference baseline of 500 ms to 5 seconds, so the enforcement decision is a small fraction of the request the user is already waiting on.

The design follows the three patterns above. Identity binding, per-route policy, and prompt classification run in parallel rather than in series. Policy decisions cache per identity and route. The signed per-decision audit record commits on a write-behind path, so evidence never sits on the critical path to the model. That keeps the gateway's own cost inside the noise floor of provider variance while still enforcing inline and failing closed. If you are evaluating gateway latency for a regulated deployment and want to see the enforcement overhead against your own traffic. Book a demo today.