Langsmart Publishes Industry’s First p95 Semantic Cache Benchmarks for On-Premises AI Gateway, Challenges Market: “Show Me the p95”
SAN JOSE, Calif.--( BUSINESS WIRE)--NVIDIA GTC 2026 — Langsmart, the enterprise AI governance company, today announced the successful completion of a rigorous enterprise evaluation with a Fortune 200 financial institution. The testing confirms that Langsmart’s Smartflow platform delivers a 10.2x speedup in response times, achieving sub-300ms latency on standard, low-resource hardware.
We published our p95 semantic cache latency at GTC. No other AI gateway vendor has. Show me the p95.
As enterprise adoption of AI gateways accelerates – with analysts projecting 70% of engineering teams will use them by 2028 – the industry has struggled with a lack of standardized performance data. Langsmart’s latest results provide a transparent blueprint for organizations requiring high-performance AI governance within strict on-premises and air-gapped environments.
Enterprise Evaluation: Performance Under Pressure
The evaluation focused on real-world financial services workloads, prioritizing reliability and speed within a secure infrastructure. Unlike cloud-based gateways that require data to leave the perimeter, Smartflow was deployed as a Docker container on a modest 4vCPU, 8GB server.
Key performance milestones included:
“For banking, insurance, and healthcare, routing prompts and model responses through a third-party cloud is a liability,” said Craig Alberino, Founder and CEO of Langsmart. “Smartflow eliminates that risk by deploying entirely within the client’s network, delivering performance that actually exceeds cloud-hosted alternatives.”
Raising the Industry Standard: "Show Me the p95"
While the evaluation highlights Smartflow’s technical achievements, it also exposes a critical transparency gap in the AI gateway market. Langsmart’s research found that while many vendors promise efficiency, none currently publish p95 or p99 latency data – the metrics most critical for production-grade enterprise stability.
“Enterprise buyers deserve to see real numbers on real hardware, not marketing claims,” said Alberino. “We are calling on all AI gateway vendors to follow our lead and publish standardized benchmarks. If you’re providing enterprise infrastructure, show me the p95.”
Langsmart’s push for transparency aims to provide CISOs and CTOs with the empirical data needed to evaluate AI governance tools effectively, ensuring that security does not come at the cost of performance.
For the full benchmarking methodology and results, visit langsmart.ai/blog/show-me-the-p95.
About Langsmart
Langsmart is the enterprise AI governance company building Smartflow – the on-premises AI firewall, gateway, and governance control plane for regulated industries. Smartflow enables financial services, healthcare, and insurance organizations to govern AI model traffic at the network layer without data leaving their infrastructure. Langsmart is headquartered in the New York City metro area with teams in Connecticut and Austin, TX.