# AI Gateway & Model Evaluation Set 📊 > A professional, reproducible evaluation layer for [Awesome AI Gateway](README.md): how the **models** behind the gateways actually perform, what they **really cost** on concrete workloads, and how the **gateways themselves** score on compliance, price, security and stability. > > **Languages:** English · [简体中文](BENCHMARKS.zh-CN.md) · Last reviewed: **see footer** Every number here is **sourced and dated**. Cost cells are *computed* from a public pricing table by a unit-tested script ([`scripts/cost_calc.py`](scripts/cost_calc.py)), never hand-typed — re-run it and you get the same table. Model scores are copied from primary leaderboards with links. Gateway scores follow the published [rubric](#scoring-rubric-apply-consistently) below. ## Contents - [Part 1 — Authoritative model benchmarks](#part-1--authoritative-model-benchmarks) - [Part 2 — Pick a model by scenario](#part-2--pick-a-model-by-scenario) - [Part 3 — Real-world token cost (computed)](#part-3--real-world-token-cost-computed) - [Part 4 — Gateway scorecard: compliance · price · security · stability · observability](#part-4--gateway-scorecard-compliance--price--security--stability--observability) - [Part 5 — Real-world reviews: what production users report](#part-5--real-world-reviews-what-production-users-report) - [Part 6 — Gateway observability: the factors that matter](#part-6--gateway-observability-the-factors-that-matter) - [Part 7 — Identity & governance: the SSO-tax table](#part-7--identity--governance-the-sso-tax-table) - [Methodology & caveats](#methodology--caveats) - [Sources](#sources) --- ## Part 1 — Authoritative model benchmarks How capable is each model? These are the most-cited public benchmarks as of the review date. **Read them with the [caveats](#methodology--caveats)** — leaderboards get gamed and contaminated; pair them with the human-preference Arena and the real-world cost tables below. Ranked by the **Artificial Analysis Intelligence Index** (the most-cited one-number composite), on the current **v4.1** scale (rebased 2026-06-15, re-weighted toward agentic work — scores sit ~5 points below v4.0, so **never compare across scales**). Sourcing is deliberately single-source per column: GPQA♦ and HLE are AA's own independent runs; SWE-bench Verified is the independent [BenchLM](https://benchlm.ai/benchmarks/sweVerified) harness (board dated 2026-07-27) — where a vendor card claims a score but the independent board doesn't carry the model, we show `—`, not the vendor's number. Reasoning models appear in their highest-effort config (what AA headline-lists). `♦` = GPQA Diamond. | # | Model | Provider | Weights | Context | GPQA♦ | SWE-bench Verified | HLE | Arena Elo | AA Index | |---|---|---|---|---|---|---|---|---|---| | 1 | **Claude Opus 5** | Anthropic | Closed | 1M | 93.2% | **96.0%** 🥇 | 52.6% | 1495ˢ | **60.7** 🥇 | | 2 | **Claude Fable 5** | Anthropic | Closed | 1M | 92.6% | 95.0% | **53.3%** 🥇 | **1508** 🥇 | **59.9** | | 3 | **GPT-5.6 Sol** | OpenAI | Closed | ~1M | 94.1% | — | 47.2% | 1485 | **58.9** | | 4 | **Kimi K3** | Moonshot | 🔓 Open | 1M | 93.5% | — | 44.4% | 1486ˢ | **57.1** | | 5 | **Claude Opus 4.8** | Anthropic | Closed | 1M | 92.0% | 88.6% | 45.7% | 1484 | **55.7** | | 6 | **GPT-5.5** | OpenAI | Closed | ~1M | 93.5% | — | 44.3% | 1482 | **54.8** | | 7 | **Grok 4.5** | xAI | Closed | 500K | 93.1% | — | 40.3% | 1468 | **53.8** | | 8 | **GLM-5.2** | Z.ai (Zhipu) | 🔓 Open · MIT | 1M | 89.5% | — | 40.1% | 1469 | **51.1** | | 9 | **Muse Spark 1.1** | Meta | Closed | 262K | 89.8% | — | 45.1% | 1491 | **50.6** | | 10 | **Gemini 3.5 Flash** | Google | Closed | 1M | 92.2% | — | 41.0% | 1476 | **50.2** | | 11 | **Gemini 3.6 Flash** | Google | Closed | 1M | 92.8% | — | 38.3% | 1482 | **50.1** | | 12 | **Gemini 3.1 Pro** | Google | Closed | 1M | 94.1% | — | 44.7% | 1486 | **46.5** | | 13 | **Qwen3.7 Max** | Alibaba | Closed | 1M | 92.3% | 80.4%⁴ | 38.1% | 1475 | **46.0** | | 14 | **DeepSeek V4 Pro** | DeepSeek | 🔓 Open · MIT | 1M | 88.8% | 80.6% | 35.9% | 1457 | **44.3** | | 15 | **Kimi K2.6** | Moonshot | 🔓 Open | 256K | 91.1% | 80.2% | 35.9% | 1461 | **44.2** | | 16 | **GLM-5.1** | Z.ai (Zhipu) | 🔓 Open | 200K | 86.8% | — | 28.0% | 1469 | **40.2** | | 17 | **Grok 4.3** | xAI | Closed | 1M | 90.1% | — | 35.0% | 1443 | **37.6** | | 18 | **Claude Haiku 4.5** | Anthropic | Closed | 200K | 67.2% | 73.3% | 9.7% | 1412 | **29.6** | | 19 | **Mistral Large 3** | Mistral | 🔓 Open | 256K | 68.0% | — | 4.1% | 1415 | **15.9** | ˢ Arena Elo still settling — Claude Opus 5 has only ~2.4K votes (±12), Kimi K3 ~3.6K; read those two as provisional. Fable 5's 1508 (±6, 16K votes) is settled. ⁴ BenchLM marks the Qwen3.7 Max (and Gemini 3.5 Flash Pro-board) rows as lower-confidence. > 🛡️ **Contamination-resistant cross-check.** On **SWE-bench Pro** (harder to game than Verified, [BenchLM](https://benchlm.ai/benchmarks/swePro)): Fable 5 **80.0%** 🥇 · Opus 5 79.2% · Opus 4.8 69.2% · Grok 4.5 64.7% · GPT-5.6 Sol 64.6% · GLM-5.2 62.1% · Muse Spark 1.1 61.5% · GPT-5.5 / Kimi K2.6 58.6% · GLM-5.1 58.4%. (Claude Mythos 5, limited availability, posts 80.3% — the figure sometimes misattributed to Fable 5.) The frontier clusters at 89–94% on GPQA — at that ceiling, 1–2 point gaps are noise; **HLE is the column that still separates** (4–53% spread). **What each column means** - **GPQA Diamond** — graduate-level science questions, Google-proof by design (AA independent runs). - **SWE-bench Verified** — fixes real GitHub issues; the headline *agentic coding* score (BenchLM independent harness). - **HLE (Humanity's Last Exam)** — frontier-difficulty closed-book exam; the strongest remaining separator (AA independent runs). - **Arena Elo** — blind human preference on [Arena (ex-LMArena)](https://arena.ai/leaderboard); the hardest metric to game. - **AA Index** — [Artificial Analysis](https://artificialanalysis.ai) Intelligence Index v4.1, a composite across agentic/coding/reasoning/knowledge benchmarks. - AIME math figures (where published) remain in [`data/models.json`](data/models.json); the column was retired here for sparse coverage. --- ## Part 2 — Pick a model by scenario Benchmarks rank capability in the abstract; most teams have one concrete job. This maps the common jobs to a *capability* pick and a *value* pick (good-enough, far cheaper). Cross-check the price in [Part 3](#part-3--real-world-token-cost-computed). | Your job | 🏆 Capability pick | 💸 Value pick (good-enough, far cheaper) | Why | |---|---|---|---| | **Agentic coding** (SWE-bench) | Claude Opus 5 (96.0) / Fable 5 | Kimi K2.6 · DeepSeek V4 Pro | Open models hit ~80% SWE-bench Verified at a fraction of flagship cost | | **Long-context / RAG** (100K+) | Gemini 3.1 Pro (1M ctx) | DeepSeek V4-Flash (1M ctx) | Cost floor on input-heavy work; mind Gemini's >200K surcharge | | **Hard reasoning / math** | Gemini 3.1 Pro (98.2 AIME'26) | GLM-5.1 · Kimi K2.6 | Open models reach 95%+ AIME — math is the most commoditized frontier skill | | **Bulk generation** (emails, content) | Claude Haiku 4.5 | DeepSeek V4-Flash · GPT-5.4 nano | Output-heavy → output price dominates; see [3.1](#31-write-a-100k-token-report-generation-heavy) | | **Cheapest acceptable chat** | GPT-5.4 nano | DeepSeek V4-Flash | ~$0.21 per 1M-token chatbot month vs $17.50 for GPT-5.5 | | **Open-ended chat** (human pref) | Claude Fable 5 (Arena 1508, #1) · Muse Spark 1.1 | GLM-5.2 (1469 at $1.40/M in) | Arena Elo is the metric that tracks "feels good to use" | | **On-prem / data-sovereign** | Kimi K3 · DeepSeek V4 Pro (MIT) | GLM-5.2 (MIT) · Kimi K2.6 | Open weights you can run inside your own VPC — zero data egress | | **Compliance-bound enterprise** | Claude Opus 4.8 / GPT-5.5 via Azure / Bedrock / Vertex | — | Route flagships through a [first-party cloud](#part-4--gateway-scorecard-compliance--price--security--stability--observability) with HIPAA/FedRAMP | > A **gateway** is what lets you act on this table without rewriting code: set the capability pick as primary and the value pick as fallback, or route per-request by task. That's the whole point of the [list](README.md). --- ## Part 3 — Real-world token cost (computed) > "Benchmarks tell you what's *best*. Your invoice tells you what's *affordable*." These tables price four concrete workloads across a representative model set, computed from the pricing in [`data/models.json`](data/models.json) by [`scripts/cost_calc.py`](scripts/cost_calc.py). Pricing is USD per 1M tokens; reasoning models bill hidden thinking tokens at the output rate. ### 3.1 Write a 100K-token report (generation-heavy) **Write a 100K-token report** (input 2,000 tok · output 100,000 tok) | # | Model | Provider | Cost | |---|---|---|---| | 1 | DeepSeek V4-Flash | DeepSeek | $0.028 | | 2 | GPT-5.6 Luna | OpenAI | $0.12 | | 3 | GPT-5.4 nano | OpenAI | $0.13 | | 4 | Mistral Large 3 | Mistral | $0.15 | | 5 | Kimi K2.6 | Moonshot | $0.40 | | 6 | GLM-5.2 | Z.ai (Zhipu) | $0.44 | | 7 | Claude Haiku 4.5 | Anthropic | $0.50 | | 8 | Grok 4.5 | xAI | $0.60 | | 9 | Gemini 3.6 Flash | Google | $0.75 | | 10 | GPT-5.6 Terra | OpenAI | $1.20 | | 11 | Gemini 3.1 Pro | Google | $1.20 | | 12 | Kimi K3 | Moonshot | $1.51 | | 13 | Claude Opus 5 | Anthropic | $2.51 | | 14 | Claude Opus 4.8 | Anthropic | $2.51 | | 15 | GPT-5.6 Sol | OpenAI | $3.01 | | 16 | GPT-5.5 | OpenAI | $3.01 | | 17 | GPT-6 Astra | OpenAI | $5.02 | | 18 | Claude Fable 5.1 | Anthropic | $5.02 | > 📊 Cheapest is **~178×** less than the most expensive for this task. ### 3.2 Summarize a 100K-token document (input-heavy) **Summarize a 100K-token document** (input 100,000 tok · output 2,000 tok) | # | Model | Provider | Cost | |---|---|---|---| | 1 | DeepSeek V4-Flash | DeepSeek | $0.015 | | 2 | GPT-5.6 Luna | OpenAI | $0.022 | | 3 | GPT-5.4 nano | OpenAI | $0.023 | | 4 | Mistral Large 3 | Mistral | $0.053 | | 5 | Kimi K2.6 | Moonshot | $0.10 | | 6 | Claude Haiku 4.5 | Anthropic | $0.11 | | 7 | GLM-5.2 | Z.ai (Zhipu) | $0.15 | | 8 | Gemini 3.6 Flash | Google | $0.17 | | 9 | Grok 4.5 | xAI | $0.21 | | 10 | GPT-5.6 Terra | OpenAI | $0.22 | | 11 | Gemini 3.1 Pro | Google | $0.22 | | 12 | Kimi K3 | Moonshot | $0.33 | | 13 | Claude Opus 5 | Anthropic | $0.55 | | 14 | Claude Opus 4.8 | Anthropic | $0.55 | | 15 | GPT-5.6 Sol | OpenAI | $0.56 | | 16 | GPT-5.5 | OpenAI | $0.56 | | 17 | GPT-6 Astra | OpenAI | $1.10 | | 18 | Claude Fable 5.1 | Anthropic | $1.10 | > 📊 Cheapest is **~76×** less than the most expensive for this task. ### 3.3 Coding-agent session (mixed + reasoning tokens)

> **Capability *and* cost on one axis.** Every model with both an independent SWE-bench Verified score (BenchLM) and a price, plotted on the shared coding-agent session. Open weights (green) hit ~80% — flagship-*tier* coding — for a fraction of the spend: **DeepSeek V4 Pro (80.6%) costs ~$0.07 a session**, while the 96% ceiling (Claude Opus 5) runs ~23× that and Arena-topping Fable 5 ~46×. The cost axis reuses the unit-tested engine below; capability is the dated `swe_bench_verified` figure (models without an independent score — e.g. Gemini 3.1 Pro, absent from the BenchLM board — don't appear). Rendered by [`scripts/make_coding_chart.py`](scripts/make_coding_chart.py) — re-run it and you get the same picture. **Coding-agent session** (input 50,000 tok · output 20,000 tok · +30,000 thinking for reasoning models) | # | Model | Provider | Cost | |---|---|---|---| | 1 | DeepSeek V4-Flash | DeepSeek | $0.021 | | 2 | Mistral Large 3 | Mistral | $0.055 | | 3 | GPT-5.6 Luna | OpenAI | $0.07 | | 4 | GPT-5.4 nano | OpenAI | $0.073 | | 5 | Kimi K2.6 | Moonshot | $0.13 | | 6 | Claude Haiku 4.5 | Anthropic | $0.15 | | 7 | GLM-5.2 | Z.ai (Zhipu) | $0.29 | | 8 | Grok 4.5 | xAI | $0.40 | | 9 | Gemini 3.6 Flash | Google | $0.45 | | 10 | GPT-5.6 Terra | OpenAI | $0.70 | | 11 | Gemini 3.1 Pro | Google | $0.70 | | 12 | Kimi K3 | Moonshot | $0.90 | | 13 | Claude Opus 5 | Anthropic | $1.50 | | 14 | Claude Opus 4.8 | Anthropic | $1.50 | | 15 | GPT-5.6 Sol | OpenAI | $1.75 | | 16 | GPT-5.5 | OpenAI | $1.75 | | 17 | GPT-6 Astra | OpenAI | $3.00 | | 18 | Claude Fable 5.1 | Anthropic | $3.00 | > 📊 Cheapest is **~143×** less than the most expensive for this task. ### 3.4 1M-token chatbot month (balanced) **1M-token chatbot month** (input 500,000 tok · output 500,000 tok) | # | Model | Provider | Cost | |---|---|---|---| | 1 | DeepSeek V4-Flash | DeepSeek | $0.21 | | 2 | GPT-5.6 Luna | OpenAI | $0.70 | | 3 | GPT-5.4 nano | OpenAI | $0.72 | | 4 | Mistral Large 3 | Mistral | $1.00 | | 5 | Kimi K2.6 | Moonshot | $2.48 | | 6 | GLM-5.2 | Z.ai (Zhipu) | $2.90 | | 7 | Claude Haiku 4.5 | Anthropic | $3.00 | | 8 | Grok 4.5 | xAI | $4.00 | | 9 | Gemini 3.6 Flash | Google | $4.50 | | 10 | GPT-5.6 Terra | OpenAI | $7.00 | | 11 | Gemini 3.1 Pro | Google | $7.00 | | 12 | Kimi K3 | Moonshot | $9.00 | | 13 | Claude Opus 5 | Anthropic | $15.00 | | 14 | Claude Opus 4.8 | Anthropic | $15.00 | | 15 | GPT-5.6 Sol | OpenAI | $17.50 | | 16 | GPT-5.5 | OpenAI | $17.50 | | 17 | GPT-6 Astra | OpenAI | $30.00 | | 18 | Claude Fable 5.1 | Anthropic | $30.00 | > 📊 Cheapest is **~143×** less than the most expensive for this task. **Pricing gotchas a gateway buyer must know** 1. **Reasoning tokens are billed as output.** A "cheap" reasoning model can cost more than a flagship once it thinks for 30K tokens. The coding table above includes them. 2. **Cached input is 5–10× cheaper on most models — and up to ~120× on DeepSeek.** Reusing a long system prompt? The cached-input rate, not the headline input rate, is your real cost. (Writes cost 1.25–2×, so it only pays above a break-even hit rate — see [caching economics](docs/caching-economics.md).) 3. **Batch APIs are ~50% off** for non-interactive work (Anthropic, OpenAI, Google all offer this). 4. **China models are priced in RMB** and often have off-peak discounts (DeepSeek) — the USD figures here are conversions and move with the exchange rate. --- ## Part 4 — Gateway scorecard: compliance · price · security · stability · observability This is the part buyers actually lose sleep over. Models are interchangeable; the gateway is where your keys, prompts, and audit trail live. Each gateway is scored ★1–5 on five axes using the rubric below, so scores are comparable rather than vibes. The observability axis grades what a gateway actually exposes (per-gateway evidence in [`data/gateways_eval.json`](data/gateways_eval.json)); [Part 6](#part-6--gateway-observability-the-factors-that-matter) explains what each pillar means in practice. ### Scoring rubric (apply consistently) | ★ | Compliance | Security | Stability / reliability | Observability | |---|---|---|---|---| | ★5 | SOC 2 Type II **+** ISO 27001 **+** HIPAA BAA **+** EU residency **+** ZDR | Guardrails + PII redaction + RBAC + SSO/SAML + audit logs + key vault | Public uptime SLA ≥99.9%, status page, multi-provider failover, sub-ms overhead | All 5 pillars: metrics export + trace export + per-key token/cost attribution + log export + dashboard | | ★4 | SOC 2 + one of {ISO, HIPAA, residency} + ZDR option | Most of the above, missing one enterprise control | SLA or strong failover + healthy maintenance | 4 of 5 pillars (typically missing trace export or a bundled UI) | | ★3 | SOC 2 **or** GDPR posture, ZDR on request | RBAC + audit logs + keys encrypted | Failover/fallback, active releases, no public SLA | Cost/usage accounting + dashboard, no standards-based (Prometheus/OTel) export | | ★2 | Privacy policy only, no third-party audit | Basic auth + key storage, few controls | Best-effort, community-maintained | Basic request logs/stats only | | ★1 | None stated | Known unpatched issues / minimal controls | Sporadic maintenance or unproven | Little beyond the provider invoice | | 🏠 | *Self-hosted: **you** own these. Score reflects the controls the software gives you to comply.* | | | | **Markup** = what the gateway charges on top of provider token cost. Self-hosted = $0 markup, you pay infra + ops. #### Hosted multi-provider gateways | Gateway | Compliance | Markup | Security | Stability | Obsv | One-line | |---|---|---|---|---|---|---| | **Cloudflare AI Gateway** | ★★★★½ | **0%** | ★★★★ | ★★★★½ | ★★★★★ | CF holds SOC 2 II / ISO 27001 / PCI; free DLP + fallback; 100% SLA at Business+ | | **Portkey** (cloud) | ★★★★½ | usage-based | ★★★★½ | ★★★★ | ★★★★★ | SOC 2 II + ISO + HIPAA; 50+ guardrail marketplace, RBAC/SSO; 99.99% SLA | | **Vercel AI Gateway** | ★★★★ | **0%** | ★★★½ | ★★★★ | ★★★★ | SOC 2 II + 99.99% SLA (Enterprise); true 0% even on BYOK | | **Helicone** (cloud) | ★★★½ | **0%** passthrough | ★★★½ | ★★★ | ★★★★½ | SOC 2 + HIPAA (Team); PII detection; OSS core → VPC/self-host option | | **Requesty** | ★★★½ | ~5% | ★★★½ | ★★★ | ★★★ | EU residency + PII masking + ZDR; SOC 2 Type II "in progress — expected Q3 2026" (slipped from Q2; the vendor's own FAQ markup contradicts the badge). Q3 is the current quarter — re-verify before relying on it | | **OpenRouter** | ★★★½ | ~5.5% credit fee | ★★★ | ★★★ | ★★★★½ | ~90 providers, auto-failover, free ZDR; **no public SLA** (enterprise only) | | **Eden AI** | ★★★½ | ~5.5% platform fee | ★★★ | ★★★½ | ★★★ | France-based, EU-default residency, GDPR-first; SOC 2 UNVERIFIED | | **Martian** | ★★★ | volume (undisclosed) | ★★★½ | ★★★ | ★★½ | "Airlock" compliance vetting + cost-routing; certs UNVERIFIED | #### First-party clouds (single-vendor, strongest certs) | Gateway | Compliance | Markup | Security | Stability | Obsv | One-line | |---|---|---|---|---|---|---| | **Azure OpenAI** | ★★★★★ | N/A | ★★★★★ | ★★★★½ | ★★★★½ | SOC 2 / ISO / HIPAA-BAA / **FedRAMP High**, region pinning, ZDR endpoints | | **AWS Bedrock** | ★★★★★ | N/A | ★★★★★ | ★★★★½ | ★★★★ | ISO / SOC / CSA STAR / HIPAA / FedRAMP High; multi-model within Bedrock | | **Google Vertex AI** | ★★★★½ | N/A | ★★★★★ | ★★★★½ | ★★★★ | First GenAI platform to FedRAMP High (2025); SOC 2 / ISO / HIPAA | | **OpenAI** (direct) | ★★★★ | N/A | ★★★★ | ★★★★ | ★★★★ | SOC 2 II, HIPAA-BAA, ZDR; but single-vendor = no cross-provider failover | > ⚠️ First-party clouds win compliance but **can't survive a provider outage** — that cross-vendor failover is exactly what a gateway in front of them buys you. #### Open-source self-hosted (🏠 you own compliance; $0 markup, you pay infra) | Gateway | Compliance | Security | Stability | Obsv | One-line | |---|---|---|---|---|---| | **Portkey Gateway** (OSS) | ★★★🏠 | ★★★★ | ★★★★ | ★★ | MIT; full guardrails, MCP OAuth, fallbacks free; 2.65 ms measured overhead (the marketed <1 ms did not reproduce) | | **Kong AI Gateway** | ★★★½ | ★★★★ | ★★★★ | ★★★½ | Prompt Guard + mature general security plugins (OSS); PII sanitization, Model Armor and RBAC are Enterprise-only (verified absent from the OSS tree, 2026-07-28) | | **Envoy AI Gateway** | ★★★🏠 | ★★★★ | ★★★★ | ★★★★ | Multi-provider + MCP gateway w/ OAuth+CEL authz; native K8s/Istio | | **Bifrost** (Maxim) | ★★★🏠 | ★★★½ | ★★★★½ | ★★★★★ | Go; measured 0.62 ms overhead (the ~11µs figure is the vendor's own benchmark); cluster mode; one high SSRF advisory, fixed 1.5.16 | | **TensorZero** | ★★★🏠 | ★★★ | ★★★★ | ★★★★½ | Rust; <1ms p99 at 10k+ QPS; routing + built-in observability; ⚠️ archived Jun 2026 | | **Higress** | ★★★🏠 | ★★★½ | ★★★★ | ★★★★½ | Istio/Envoy AI-native, Wasm plugins, console; Alibaba-backed | | **Apache APISIX** | ★★★🏠 | ★★★ | ★★★★ | ★★★½ | ai-proxy / ai-prompt-guard plugins on mature ASF gateway | | **LiteLLM** | ★★★🏠 | ★★½ ⚠️ | ★★★★ | ★★★★★ | SOC 2 I + ISO (Enterprise); **run ≥1.84.0** — 12 advisories published in 2026, 3 of them critical (one RCE chain on CISA's KEV list); all fixed, but the floor moved past 1.83.7 | | **GPT-Load** | ★★🏠 | ★★½ | ★★★½ | ★½ | Go key-pool rotation + encrypted key store + dual auth; proxy-level only | | **new-api** | ★★🏠 | ★½ ⚠️ | ★★★ | ★★½ | Active, but **cluster of 2026 CVEs** (IDOR/SSRF/SQLi) — sandbox + patch fast | | **one-api** | ★★🏠 | ★★ | ★★½ | ★★ | The MIT original; maintenance slowed — new-api is the more active successor | > ⚠️ **CVE honesty.** Popularity makes OSS gateways targets. LiteLLM (pre-auth SQLi + unauth RCE) and new-api (IDOR/SSRF/SQLi) both had serious 2026 advisories — *patched*, but the lesson is: pin to current stable, restrict egress, and don't expose the admin panel publicly. Absence of found CVEs (TensorZero, Higress, Envoy, GPT-Load) ≠ proven-secure; it can mean less scrutiny. > 📊 **Obsv column** = the five-pillar observability score from the rubric; per-gateway evidence (which pillars, which docs) is machine-readable in [`data/gateways_eval.json`](data/gateways_eval.json). Standouts: **LiteLLM / Bifrost / Cloudflare / Portkey cloud** cover all five pillars; **Portkey OSS v1.x ships near-zero observability** (its telemetry lands in the unreleased 2.0 branch); **Envoy AI Gateway** is the strongest standards-first pick (OTel GenAI-semconv, no UI); CN-panel gateways (new-api/one-api/GPT-Load) invert — strong billing UI, no Prometheus/OTel. > ⏱️ **Gateway overhead, independently measured.** Vendors market conflicting overhead claims (µs-level vs ms-level) with no third-party numbers — so this project measures it: a reproducible harness (mock OpenAI upstream, interleaved rounds, median-of-medians; no API keys) runs monthly in CI on the same neutral runner. 2026-07 results, added latency per request: **Bifrost 0.62 ms** (IQR 0.59–0.78) · **Portkey Gateway OSS v1.15.2 2.65 ms** (2.57–2.83) · **LiteLLM v1.91.1 5.83 ms** (5.55–5.86). Read against marketing: Bifrost's "fastest" holds directionally (~10× vs LiteLLM — not the marketed 50×, which refers to loaded throughput); Portkey's "<1 ms" didn't reproduce on shared CI hardware (it did on a fast desktop: 0.47 ms). Data: [`llm-gateway-bench/data/overhead.json`](https://github.com/cuihuan/llm-gateway-bench/blob/main/data/overhead.json) · [methodology](https://github.com/cuihuan/llm-gateway-bench/blob/main/docs/methodology.md) — PRs adding Kong/Envoy/Higress are welcome. > 🔌 **Protocol-translation fidelity, independently measured (2026-07).** The most-reported gateway failure in real bug trackers isn't routing — it's the gateway **corrupting tool-calls / streaming / usage in translation** ("claude code" appears in 400+ LiteLLM issues; the top-commented issues on Portkey/OpenRouter/Bifrost are all this). So this project measures it: a mock upstream returns spec-correct responses (a tool_call, a genuine multi-chunk SSE stream with `usage`), and the harness checks what actually arrives **through** each gateway (no keys, reproducible in CI). Result, self-hosted at a custom upstream: **LiteLLM v1.91.1 — 3/3 · Bifrost — 3/3 · Portkey Gateway OSS v1.15.2 — 1/3.** All three relay a tool_call intact; but **Portkey OSS in custom-host mode threw an internal error on *every* streaming request** (client got 0 chunks, no `usage`) while non-streaming worked — reproduced on a clean CI runner. Fair caveats: this is the OSS gateway's *self-host custom-host path* (Portkey's hosted product / standard provider integrations may stream fine, and v2.0 is unreleased). That result is OpenAI-format **passthrough**; the harder **cross-format** path — an Anthropic client (e.g. Claude Code) routed to an OpenAI upstream, where the hardest bugs live — is now measured too (neutral CI runner): **LiteLLM v1.91.1 — 3/3 · Bifrost — 3/3 · Portkey Gateway OSS v1.15.2 — not offered.** LiteLLM (`/v1/messages`) and Bifrost (`/anthropic/v1/messages`) both translate a Claude-Code / Anthropic-SDK client to an OpenAI upstream cleanly — tool_use, streaming and usage all survive; Portkey OSS's `/v1/messages` is **Anthropic-provider-only** — with an OpenAI provider it rejects the request (`messages is not supported by openai`), so it doesn't offer this path in header-config self-host mode. One finding worth pinning your version for: LiteLLM's `/v1/messages` transport **changed across releases** — ≤1.57.x via OpenAI *Chat Completions* (drops streaming usage, 2/3), ≥~1.9x (and Bifrost) via the OpenAI **Responses API** (upstream `/v1/responses`), which fails `KeyError('created_at')` against a chat-completions-only upstream (recorded **inconclusive**; a gateway that doesn't offer the path at all is **unsupported** — neither is ever a 0/3). **Takeaway: before committing, run your actual agent — tools + streaming — through the gateway, not a hello-world.** Data: [`fidelity.json`](https://github.com/cuihuan/llm-gateway-bench/blob/main/data/fidelity.json) (passthrough) · [`xformat.json`](https://github.com/cuihuan/llm-gateway-bench/blob/main/data/xformat.json) (cross-format). > 🏠 **Self-hosted shifts the burden to you.** LiteLLM/Bifrost/Kong score on the *controls they hand you* (RBAC, audit logs, key vaulting, on-prem) — but SOC 2 / HIPAA compliance of the *deployment* is yours to earn. That's the trade for $0 markup and full data control. --- ## Part 5 — Real-world reviews: what production users report Benchmarks rank capability; this ranks **what actually breaks once a gateway is in production**. Sourced from incident postmortems, status pages, security research and acquisition news — every dated event below links to a primary or recognized source, summarized fairly (what users praise *and* complain about). Read it alongside the [scorecard](#part-4--gateway-scorecard-compliance--price--security--stability--observability): stars measure popularity; this measures the 3am pager. | Gateway | Praised for | Recurring gripe | Dated event worth knowing | |---|---|---|---| | **LiteLLM** | Default OpenAI-compatible multi-provider proxy; unmatched model breadth | Latency/memory overhead and degradation at high RPS | ⚠️ **PyPI supply-chain attack (Mar 2026)** — v1.82.7/1.82.8 were backdoored with a credential stealer after a Trivy CI-token compromise (TeamPCP); PyPI quarantined the project in ~3h. Pin to a clean release. [Trend Micro](https://www.trendmicro.com/en/research/26/c/inside-litellm-supply-chain-compromise.html) · [Datadog](https://securitylabs.datadoghq.com/articles/litellm-compromised-pypi-teampcp-supply-chain-campaign/) | | **OpenRouter** | 400+ models behind one key and one bill; hard spend caps | ~5.5% credit fee at scale; provider quality/quantization varies; no public SLA | **~50-min database outage (Aug 28 2025)**, then two more on **Feb 17 & 19 2026** — all DB-layer failures, not upstream providers (the Feb pair: the caching layer dropped every DB connection, returning 401s with ~80–90% request failures); ~99.96% uptime over 8 months, but no SLA or downtime credits. [StatusGator](https://statusgator.com/services/openrouter) · [Feb 2026 postmortem](https://openrouter.ai/blog/announcements/openrouter-outages-on-february-17-and-19-2026/) | | **Portkey** | Genuinely production-grade; strong observability, fallbacks, prompt management | Much "review" content is SEO — weigh G2/Gartner over listicles | **Acquired by Palo Alto Networks (closed May 2026)** — now the control plane for Prisma AIRS; a neutrality/roadmap question if you want vendor-independent infra. [Palo Alto Networks](https://www.paloaltonetworks.com/company/press/2026/palo-alto-networks-completes-acquisition-of-portkey-to-secure-ai-agents) | | **Vercel AI Gateway** | One endpoint, 0% markup on BYOK, native to the AI SDK | Durable independent sentiment still thin | **Vercel breach (Apr 2026)** — a Context.ai OAuth supply-chain compromise exposed environment variables for a subset of customers. Not the gateway product itself, but it bears on trusting Vercel with your keys. [TechCrunch](https://techcrunch.com/2026/04/20/app-host-vercel-confirms-security-incident-says-customer-data-was-stolen-via-breach-at-context-ai/) · [Vercel](https://vercel.com/kb/bulletin/vercel-april-2026-security-incident) | | **Cloudflare AI Gateway** | Zero-infra observability / caching / rate-limiting on CF's edge; dollar spend limits | Limited request-placement / routing control | Newer entrant; independent production-review depth is still thin. | | **Kong AI Gateway** | Inherits a battle-tested data plane + broad plugin ecosystem | Open-core: several important plugins are Enterprise-only (some prefer Apache APISIX for predictable OSS) | The AI Gateway is newer; most Kong-vs-LiteLLM/Portkey benchmarks are vendor-published — reproduce before trusting. | | **TensorZero** | Ambitious OSS gateway + observability + optimization "flywheel" | — | ⚠️ **Archived June 2026** — company wound down; Apache-2.0 code and community forks remain, but plan for self-support, not a vendor roadmap. [GitHub](https://github.com/tensorzero/tensorzero) | | **Helicone** | ~2-minute onboarding; fast cost/usage and token-limit debugging | As a proxy it sits in the request path (a SPOF unless you use async logging) | **Acquired by Mintlify (Mar 2026), now maintenance mode** — security fixes, bug fixes and new models keep shipping; Mintlify supports migration. [Mintlify](https://www.mintlify.com/blog/mintlify-acquires-helicone) · [Helicone](https://www.helicone.ai/blog/joining-mintlify) | | **Bifrost** | Markets itself as a fast Go drop-in for LiteLLM | Independent production evidence is still thin | Its 50–90× lower-p99 claims are **vendor-reported** — reproduce on your own workload before betting on them. | > **How to read this.** Every gateway here is a *legitimate* project with a real maintainer or company — the kind this list includes. We surface their warts on purpose: a gateway with *no* public criticism usually just has no public users. These notes are point-in-time and dated; sources move fast, so verify before you sign. --- ## Part 6 — Gateway observability: the factors that matter *Why this is its own axis — separate from the [scorecard](#part-4--gateway-scorecard-compliance--price--security--stability--observability) and from generic APM: a gateway sits between many internal consumers and metered, $-per-token providers, so the unit of analysis is **tokens and dollars attributed per key / team / user / model** — and the gateway's own value-add (retries, fallback, caching, guardrails) actively **masks** cost and failure unless it's instrumented. The cross-vendor standard is the [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) (`gen_ai.*` spans + metrics), now natively consumed by Datadog/Honeycomb/Grafana — but most `gen_ai.*` attributes are still **"Development"** status in 2026, and several heavily-marketed capabilities (online evals, drift detection) are real **product features, not standards**. This rubric grades what a gateway actually exposes, flags standardized vs aspirational, and stays neutral. The premise is survey-backed: cost is the **#2 most-monitored production metric** after quality/task success, with nearly half of teams actively monitoring cost in prod ([Amplify 2026 AI Engineering Report](https://www.amplifypartners.com/blog-posts/the-2026-ai-engineering-report), n>1,000).* ### Table-stakes — miss these and it's barely instrumented (mostly maps to Required/Stable OTel) | Factor | What a well-instrumented gateway exposes | How to verify | |---|---|---| | **Core inference telemetry** | `gen_ai.operation.name` + `gen_ai.provider.name` on every span (**both Required**), span named `{op} {model}`, and **both** `gen_ai.request.model` (alias requested) **and** `gen_ai.response.model` (concrete model that answered) — so a silent reroute / alias-swap is visible. | Call two upstreams via a model alias; confirm distinct `provider.name`, and `request.model`=alias vs `response.model`=resolved version. | | **Token & $ cost attribution** | input/output token counts + the `gen_ai.client.token.usage` histogram split by `gen_ai.token.type`; cost read from the **provider usage payload** (not just model-name inference), broken down by token type (prompt / completion / cached-read / reasoning) and rolled up per key/team/user/model. | Two identical calls under different keys → separate cost; a cache-read costs less than a miss; an **unknown model still yields token counts**, not a silent $0. | | **Decomposed latency** | total **and** upstream-provider **and** gateway-overhead as separate numbers, plus **TTFT** on streams, as **histograms** (p95/p99) — `gen_ai.client.operation.duration` is the **only** metric the spec marks Required. | One trace shows total/upstream/overhead separately; a streaming call records TTFT distinctly; the metric is a histogram, not an average gauge. | | **Error taxonomy by origin** | failures carry the **Stable** `error.type` + provider status + class, separated by origin (client/gateway rejection vs upstream failure vs guardrail block) — not one flat 5xx counter. | Trigger a provider 429, an oversized-prompt 400, a budget/auth rejection, and a guardrail block → each labeled distinctly. | | **Open export / no lock-in** | emits **and** ingests OTLP (`gen_ai.*` / OpenInference), a Prometheus `/metrics` endpoint, webhooks, and bulk raw export to S3/warehouse — not a dashboard-only silo (a real risk after the 2026 shakeout: TensorZero archived, Helicone→Mintlify maintenance, Portkey→Palo Alto). | Point its exporter at a throwaway OTel Collector; `curl /metrics`; configure a test webhook; ask for a documented warehouse export. | | **Cardinality discipline** | metrics use only **bounded** labels (model / provider / region / status / deployment); unbounded ids (prompt text, conversation/request id, raw user id) live in trace/log attributes, **never** labels — else the monitoring bill exceeds the inference bill. | Scrape `/metrics` and look for any unbounded label; ask "what's my metrics bill at 10k RPS with 1M distinct users?" | ### Differentiating — instruments the gateway's own value-add (which otherwise hides cost & failure) | Factor | What a well-instrumented gateway exposes | How to verify | |---|---|---| | **Reliability visibility** (retry / fallback / failover / cooldown) | counters for successful vs failed fallbacks labeled `requested→fallback` model, retry-attempt counts, circuit-breaker/cooldown events — because a request that succeeds after 3 retries + a fallback otherwise looks like a clean 200. | Force the primary to 500 → trace shows retry spans, the fallback source→target, and which deployment finally served. | | **Cache visibility** | per-request HIT/MISS + hit-rate % + cost/time saved; exact **and** semantic caching; provider prompt-cache read/write tokens tracked **separately** from the gateway's own response cache. | Send the same request twice → 2nd flagged HIT with lower cost/latency; dashboard shows hit rate + cumulative savings. | | **Budget / quota / rate-limit telemetry** | live remaining-budget gauges per key **and** team ($ + hours-to-reset), remaining RPM/TPM per model, upstream provider rate-limit headroom, graduated alerts, and a **hard cutoff** at the limit (not just an alert). | Set a small team budget, drive spend → gauge ticks down, alert fires, **and traffic is actually cut off**. | | **Streaming lifecycle** | a `gen_ai.request.stream` flag, TTFT + per-output-token latency, and **mid-stream failure** detected as a failure/partial (a stream can send 200 OK then stall or error). | Stream a request → TTFT distinct from total; cut the connection mid-stream → recorded as failure/partial, not a clean success. | | **Prompt/response capture *with* redaction** | content capture **off by default** (the OTel default, toggled via `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`), landing in `gen_ai.input.messages` / `output.messages`; paired with **PII redaction before storage and before the provider call**. Raw logging without redaction can breach GDPR + the EU AI Act (high-risk provisions enforceable **2026-08-02**). | With capture unset, only metadata appears; a planted SSN/email is masked on both input **and** output. | ### Advanced — the 2026 frontier, mostly **non-standard product features** (grade as bonus, verify claims) | Factor | What a well-instrumented gateway exposes | Honest caveat | |---|---|---| | **Retention + tail/eval-driven sampling** | per-data-class retention (keep metrics long, expire raw prompts fast) + sampling that keeps **100% of errors** and high-cost/anomalous traces while sampling the boring bulk — head-only sampling drops the rare hallucination you needed. | Only OTLP transport is standardized; the **sampling policy is product-specific**. | | **Eval / quality & drift signals** | online evals (LLM-as-judge or programmatic) attachable to live traces and chartable over time, + capture of the **resolved** `gen_ai.response.model` so a silent provider re-point is visible, + trace→eval behind a CI gate. | Online evals / LLM-as-judge are **vendor product features, NOT the OTel standard** — only `gen_ai.response.model` is standardized. Treat vendor eval/latency claims as vendor-reported until reproduced. | **Exemplars among listed gateways** (illustrative, not endorsements): **LiteLLM** ships the reference self-hosted label set (`litellm_spend_metric`, cached/reasoning token splits, distinct request/upstream/overhead latency, fallback + budget metrics, Prometheus + Grafana). **Helicone**, **Langfuse**, **Arize Phoenix**, **Pydantic Logfire** and **Braintrust** are observability-first (OTLP-native, cost-by-segment, evals). **Portkey** exposes OTLP export + hard budget enforcement + cache/guardrail telemetry. **Kong AI Gateway** maps the `gen_ai.*` span set and PII sanitization. **Cloudflare AI Gateway** adds spend limits + free DLP/PII scan. *Verify against your own workload — most performance/eval figures are vendor-reported.* > **Questions to ask before you trust a gateway's dashboard** > - Show me a sample exported span: do you **emit and ingest OTLP** with `gen_ai.*` / OpenInference names, or is telemetry proprietary-only? > - Is there a Prometheus `/metrics` endpoint, and are all labels **bounded**? (Any unbounded label — prompt text, conversation/request id, raw user id — is a walk-away.) > - Can cost be attributed **simultaneously per virtual-key / team / user / model AND by token type**, read from the provider usage payload (not just inferred from the model name, which breaks on new/renamed models)? > - Can I read **total, upstream, and gateway-overhead latency separately**, with TTFT on streams, as histograms (p95/p99) — not one average gauge? > - When a request succeeds only after retries + a fallback, does the trace show the **retry spans and the fallback source→target**, or just a 200? > - Do budgets **hard-cut-off** at the limit or only alert, and where do alerts route? > - Is content capture **off by default**, with PII redacted **before storage and before the provider sees it**? > - Do you record the **resolved** `gen_ai.response.model`, so a silent provider re-point or model re-tune is detectable? > - Under ZDR / self-host, **what observability do I lose** (typically you keep metadata/metrics, drop prompt bodies)? > **Grounded in** the [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) ([spans](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md) · [metrics](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md)). Most `gen_ai.*` attributes are still **Development** status in 2026 (`error.type` / `server.*` are Stable) — pin `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` so dashboards don't silently break across releases. Reference label sets: [LiteLLM Prometheus](https://docs.litellm.ai/docs/proxy/prometheus), [OpenLLMetry/Traceloop](https://github.com/traceloop/openllmetry), [OpenInference](https://github.com/Arize-ai/openinference). Last reviewed **2026-06-25**. --- ## Part 7 — Identity & governance: the SSO-tax table Enterprise evaluations burn hours on four unglamorous questions — **SSO, SCIM, RBAC, audit logs** — and no neutral party has published the tier gates on one page. Here they are, from primary vendor docs only ([machine-readable with per-cell quotes + URLs](data/identity_matrix.json), verified **2026-07-27**). Control-plane features only: data-plane auth plugins (APISIX `openid-connect`, Higress `key-auth`, …) protect *proxied* traffic, not the gateway's own console, and don't count. "Not documented" = the vendor's docs are silent, **not** a confirmed "no". The recurring pattern is the **SSO tax**: identity and governance are precisely where open source and low tiers stop and the enterprise contract begins. **Legend:** ✅ available at the stated tier · 💰 paid/enterprise gate · ❌ absent · ❔ not documented ### Self-hosted / open-core | Vendor | SSO (admin / control plane) | SCIM provisioning | RBAC | Audit logs | |---|---|---|---|---| | **LiteLLM** | ✅ OIDC + SAML, **free ≤5 users** (v1.76.0+) · 💰 Enterprise beyond | 💰 Enterprise (`/scim`, 6 IdPs; deprovision also deletes the user's API keys) | ✅ global roles + teams/keys free · 💰 org/team-admin roles Enterprise | 💰 Enterprise (key/team/user/model changes; UI + S3 JSON export) | | **Portkey** | 💰 Enterprise (OIDC + SAML, hosted control plane) · ❌ not in OSS gateway | ✅ users + groups→workspaces (Azure AD, Okta) · tier ❔ (docs sit under "Enterprise Offering") | ⚠️ vendor self-conflict: pricing says Production $49/mo, docs banner says Enterprise · ❌ not in OSS | 💰 Enterprise (admin actions, indefinite retention, JSON Admin API; no bulk file export) | | **Kong** | 💰 Konnect Enterprise (OIDC + SAML) · ❌ none in OSS | ❔ not documented for the gateway (Kong's SCIM is Insomnia-only) | 💰 Konnect Plus+ / Gateway Enterprise · ❌ zero `rbac` keys in OSS `kong.conf.default` | 💰 Enterprise (Konnect SIEM webhook, CEF/JSON — ⚠️ **7-day retention**, then deleted) | | **Apache APISIX / API7** | ❌ OSS (single shared admin key) · 💰 API7 Enterprise (OIDC/SAML/LDAP) | ❔ OSS · 💰 API7 Enterprise (Okta, Entra ID) | ❌ OSS · 💰 API7 Enterprise (custom roles + IAM-style policies) | ❔ OSS · 💰 API7 Enterprise (180-day default, JSON/CSV export) | | **Higress (OSS)** | ❌ single local admin account (console login mgmt is a roadmap item) | ❔ not documented | ❌ single-admin model, no console roles | ❔ not documented | ### Hosted | Vendor | SSO | SCIM | RBAC | Audit logs | |---|---|---|---|---| | **OpenRouter** | 💰 Enterprise (SAML: Okta, Entra ID, Google Workspace, custom) | 💰 Enterprise (group→workspace mappings; IdP deactivation kills org-scoped keys) | ✅ 2 roles only (Admin/Member) + per-workspace assignment · tier ❔ | ❔ **no admin-action audit log documented** — only usage CSV/PDF export | | **Vercel AI Gateway** | 💰 Enterprise, or Pro add-on at **$300/month** (SAML, 20+ IdPs) | 💰 Enterprise only (Directory Sync — not purchasable on Pro at any price) | ✅ Pro team-level · 💰 Enterprise adds project-level | 💰 Enterprise (CSV + SIEM streaming / Audit Log Drains) | | **Cloudflare AI Gateway** | ✅ **free on all plans** (dashboard SSO, any Cloudflare One IdP, SAML incl.) | 💰 Enterprise only (Okta, Entra ID, Authentik) | ✅ 70+ account roles + domain/resource scoping (domain-scoped shipped to Enterprise; docs silent on tier today) | ✅ all plans, 18-month retention, CSV/API · 💰 Logpush export Enterprise | | **Higress hosted (Alibaba Cloud APIG)** | ✅ free (RAM SAML 2.0, account plane) | ✅ CloudSSO SCIM 2.0 (Okta, Entra ID, Keycloak) | ✅ RAM roles + custom JSON policies, resource-level (service code `apig`) | ✅ ActionTrail (90-day free, SLS/OSS delivery) | > **How to read the SSO tax.** (1) **Six of the nine vendors gate SSO behind a paid or enterprise tier.** Cloudflare is the only gateway vendor shipping dashboard SSO free on every plan; Vercel is the only one with a public sticker price for the tax ($300/month on Pro); LiteLLM has the only OSS carve-out (free up to 5 users since v1.76.0); Higress OSS has no console SSO at any price. (2) **Don't confuse data-plane auth with control-plane identity.** APISIX's `openid-connect`/`saml-auth` and Higress's auth plugins protect *proxied* APIs — the OSS consoles behind them are single-admin with no SSO, RBAC, or audit trail documented at all. (3) **Audit-log fine print bites hardest.** Kong Konnect keeps audit logs **7 days** then deletes them (the SIEM webhook is effectively mandatory); OpenRouter documents no admin-action audit log at all; LiteLLM and Portkey put audit logs behind Enterprise. (4) **Check *where* the feature lives.** Portkey's OSS repo is the data plane only — every identity feature requires the hosted/enterprise control plane; Cloudflare, Vercel and Alibaba Cloud inherit identity from the platform account, not the gateway product — fine if you're all-in on that platform, an extra dependency if not. Per-cell vendor quotes and source URLs: [`data/identity_matrix.json`](data/identity_matrix.json). --- ## Methodology & caveats - **Benchmarks are necessary, not sufficient.** Public sets leak into training data (contamination), and vendors optimize for the leaderboard. We therefore show *several* benchmarks + blind human preference (Arena) + real cost, and weight none of them alone. - **"Verified" matters.** We prefer SWE-bench **Verified** over the raw set, and official model cards / [Artificial Analysis](https://artificialanalysis.ai) independent runs over vendor press releases. Vendor-reported numbers are labeled. - **Cost ≠ price.** Headline $/token hides reasoning-token inflation, cached-input discounts and batch pricing. Part 3 prices *workloads*, not tokens, and the script is open for you to plug your own token mix. - **Gateway scores are point-in-time estimates** from public trust pages, status pages and docs. Compliance certs lapse and ship; verify before you sign. Corrections welcome via PR — see [CONTRIBUTING](CONTRIBUTING.md). - **No affiliation.** This list takes no money from any vendor listed. Self-hosted and commercial options are scored on the same axes. ## Sources Primary leaderboards and pricing references (verify live — these move weekly): - [LMArena](https://lmarena.ai) — blind human-preference Elo - [Artificial Analysis](https://artificialanalysis.ai) — Intelligence Index, price & speed - [SWE-bench](https://www.swebench.com) — agentic coding leaderboard - [Vellum LLM Leaderboard](https://www.vellum.ai/llm-leaderboard), [OpenRouter rankings](https://openrouter.ai/rankings) - **Agentic & tool-calling:** [Terminal-Bench](https://www.tbench.ai/leaderboard) (shell/CLI agents), [τ²-bench](https://github.com/sierra-research/tau2-bench) (tool-agent policy adherence), [BFCL v4](https://gorilla.cs.berkeley.edu/leaderboard.html) (Berkeley function-calling), [Aider polyglot](https://aider.chat/docs/leaderboards/) (multi-language code edits) - **Contamination-resistant / frontier:** [LiveBench](https://livebench.ai) & [LiveCodeBench](https://livecodebench.github.io/leaderboard.html) (questions refreshed monthly), [FrontierMath](https://epoch.ai/frontiermath) (research-level math) - Official pricing: [Anthropic](https://www.anthropic.com/pricing), [OpenAI](https://openai.com/api/pricing/), [Google](https://ai.google.dev/pricing), [DeepSeek](https://api-docs.deepseek.com/quick_start/pricing) - Observability standard (Part 6): [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) — `gen_ai.*` spans & metrics Per-cell sources are listed in [`data/models.json`](data/models.json) and the gateway research notes. --- *Maintained as part of [Awesome AI Gateway](README.md). Model scores and prices change fast; this set is reviewed on a published cadence and every figure is dated at its source.* **Last reviewed: 2026-07-28** · benchmark & pricing snapshot in [`data/models.json`](data/models.json), gateway scores in [`data/gateways_eval.json`](data/gateways_eval.json).