Performance
Raw latency doesn't matter. Latency under load does.
On its own, Vidai adds 0.58 ms median at up to 47,200 calls a second on one machine. Across a real network it held 21,803 calls a second at 1.95 ms median, network time included, while Python-based proxies peaked under 200.
Three tests, three questions
One story, measured three ways.
On one machine
Up to 47,200 calls a second, with 0.58 ms median added
How much does Vidai itself add to a call?
Vidai, a mock provider and the load generator on one machine, so no network time is counted: the figures are Vidai's own. 32 connections, four 15-second runs, every call recorded to PostgreSQL, guardrails and TLS off.
Across a network
21,803 calls a second at 1.95 ms median, network included
What does a real deployment see, network included?
Load sent across a real network to Vidai on one 2013 Intel Xeon E3-1240 v3 (8 cores), with VidaiMock as the provider, so every figure includes network time. The test network reached its limit before Vidai did.
Head to head
13 to 17 times LiteLLM's throughput, 3.5 to 3.9 times Portkey's
How does Vidai with governance on compare with gateways doing less?
Vidai with authentication, rate limits and policy on, against LiteLLM and Portkey in minimal pass-through mode, on the same harness at three load profiles.
47,200 is the highest rate tested, not a guaranteed limit. Python-based proxies on the same harness peaked under 200 calls a second, at 197 ms median. The scripts are open source, so you can run the tests yourself: benchmark scripts. Results on your infrastructure will differ.
Why load is now the only number
A benchmark used to be a vanity metric. Agents made it a survival metric.
In the chat era, your AI traffic was paced by people: one prompt, one call, human typing speed as the ceiling. Agentic AI removes that ceiling. One task now fans out into ten or more autonomous calls, and the traffic through your in-path layer is climbing 10× and still rising.
The layer in front of every AI request is no longer handling chat-pace load. It is handling agent-pace load, and its speed under that load is now your AI's speed.
See what 10× looks like at your scale →What it means for the business
Fast enough that nobody notices it's there.
Every 100 ms of added latency measurably reduces engagement. Vidai sits in the path of every AI request and adds under a millisecond of its own: 0.58 ms median, measured on one machine with no network in between. It doesn't become the bottleneck when traffic spikes 10×.
Performance is a property of being a control plane, not a feature you tune. What's a control plane? →
Across a network
21,803 calls a second, network included.
| Gateway | Peak RPS | Median | p95 | p99 | Status |
|---|---|---|---|---|---|
| Vidai (Rust) | 21,803 | 1.95 ms | 23.5 ms | 40 ms | Stable |
| Python-based proxies | < 200 | 197 ms | 1.3 s | 6.0 s | Past its limit |
| Cloud-managed gateways | ~ 640 | 100 ms | 395 ms | 1.2 s | At its limit |
Method: a single legacy 8-core node, an Intel Xeon E3-1240 v3 from 2013, chosen so the figure is a floor, not a best case, with VidaiMock as the provider. 21,803 is where the test network reached its limit, not Vidai. Full methodology and reproduction steps are in the technical write-up.
Per-core scaling
Why a single node carries so much.
The whole-machine number above is the headline. The number underneath it is the more telling one. Each core in the test added roughly 7,000 requests per second of headroom in a linear scaling curve, up to about 29,000 RPS in total before shared-hardware contention (NIC, cache) prevented further increase. On the same harness, Python-based AI proxies bottlenecked around 75 to 110 RPS per core: a per-core efficiency gap of roughly 94×.
That gap is the reason a fleet of Python nodes does not bridge the difference: 94 of them, behind a load balancer, replicate the bottleneck 94 times. The fastest path through this is not more nodes, it's a faster runtime.
94×
per-core throughput gap
Infrastructure footprint
What decides your node count: headroom, or a ceiling.
Adding nodes is never free
"Just load-balance it" is a real answer, with a real bill: each node is compute, load-balancer capacity, and one more thing to patch and page for. The question is whether you add nodes by choice or by necessity.
Vidai nodes scale for HA, not headroom
A single node already carries the load with margin. The second node is the one you fail over to, not the one that stops you redlining.
A redlining node has no margin for the spike
When traffic jumps 10× for an hour, headroom absorbs it. A node already near its ceiling has nowhere to put it, so the fleet has to be sized for the peak, all the time.
Efficiency per node is the lever
Cut what one node clears and you multiply the fleet, the load balancer and the on-call surface with it. Agent-pace traffic only sharpens that maths.
Why each node carries more
The headroom is a property of a Rust hot path.
No garbage collection
No tail-latency spikes when the collector runs mid-request.
No GIL
Real multi-core throughput on every node, not a single-threaded process that scales only by adding more processes.
Zero-copy networking
The request isn't re-serialised three times on the way through.
~25 MB engine
Small footprint, low hardware bill, trivial to replicate.
High availability, today
Run several Vidai servers behind your load balancer, with autoscaling. Add a server for resilience, not because one ran out of room.
More on operations →How we start together
Want the benchmark methodology.
- 01
Pick one app or team
Point it at the Control room. One configuration change; nothing else changes.
- 02
Watch for two weeks
Every call priced, checked and recorded. Rules start in observe mode.
- 03
See the evidence
What it cost, what it caught and what it would have stopped, from your own traffic.