PayPal’s gateway brings multi-vendor AI access under shared rules
Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy · arXiv:2610.35509
In plain words
This paper describes the Cosmos LLM Gateway, a system PayPal built to give enterprise applications one governed way to reach models from different vendors.
A separate service checks identity, access and budgets, while a filter inside the traffic proxy translates requests and tracks token use as responses stream.
In tests with a simulated upstream service, added request-path time stayed at 4 ms median as load rose to about 2,300 requests per second. The test did not measure response-body work or performance near saturation.
The research
When applications connect directly to multiple model vendors, credentials, request formats, spending controls and data rules can become fragmented. Can an organization put those functions behind one shared entry point while keeping added request delay low?
Cosmos LLM Gateway centralizes model access and governance, splitting control decisions from in-proxy translation to limit delays on streamed traffic.
Cosmos LLM Gateway centralizes model access and governance, splitting control decisions from in-proxy translation to limit delays on streamed traffic. With two instances, doubling load from 1,137 to 2,257 requests per second left median, 95th- and 99th-percentile gateway-added time unchanged at 4, 10 and 11 ms. The authors measured time to the upstream response, minus the simulated upstream’s service time; four-instance results were similarly steady, supporting their claim of little added delay in the tested conditions.
These measurements support the authors’ conclusion that one enterprise can centralize governance across vendors with a median of a few milliseconds of measured request-path overhead.
The latency test covered a synthetic upstream and only the request path before the upstream response; it excluded response-body work, and the authors did not test near saturation. The system and its production results describe one organization.
What the evidence shows
- One entry point for models
The Cosmos LLM Gateway is PayPal’s production system for giving enterprise applications one governed way to reach models from multiple vendors. The paper addresses a problem with connecting each application directly to each vendor: credentials, data formats, spending controls and governance can become fragmented across teams.
The authors ask whether one shared gateway can bring those functions together while keeping the request path responsive.
- Separate decisions from translation
The gateway divides work according to what each task needs. A separate service handles identity, credentials and policy decisions that depend on external information.
A filter inside the traffic proxy translates requests and measures token use as response chunks pass through. This keeps streamed response chunks from waiting on an external call by default, although some data-protection uses require response handling.
- Test the request path
The authors measured added request-path time by sending requests through the gateway to a simulated upstream with fixed latency, then subtracting the upstream’s service time. With two instances, doubling load from 1,137 to 2,257 requests per second left the median, 95th-percentile and 99th-percentile overhead at 4, 10 and 11 milliseconds. Four-instance results were also steady across the tested loads, with no failed requests.
- Add production evidence
The latency result covers only the request path before the upstream response begins. The paper adds production figures and live-voice deployment observations: as of September 2026, the gateway served more than 12,000 developers, and more than ten deployments during live traffic sustained over 2,000 concurrent calls with no dropped calls reported.
These observations show operation at PayPal, while the authors’ conclusion remains bounded to one enterprise and the conditions described.
Deep
- Original paper authors
- Mithun Sasidharan, Srinivasan Manoharan, Rupesh Thakur, Abhinav TK, Ashwin Hegde
- Submitted to arXiv
- License
- arXiv non-exclusive