AI Gateway — agentic plane

NEW

Govern every AI request. Tokens, cost, and risk — one gateway.

Track every token. Cap every budget. Route across a 17-provider, 111-model catalog behind one OpenAI-compatible endpoint. Apinizer governs your LLM, MCP, and agent traffic on the same runtime that already runs your REST APIs — same audit, same identity, same operators on call.

Request a demo Read the docs

Catalog17 providers · 111 models
StandardsOpenAI · MCP · A2A
ModalitiesChat · Embed · Audio · Image

Token economics and cost control — first-class

Every prompt, every response, every embedding — counted, attributed, and capped. Set token and USD budgets on the gateway scope chain — global, anonymous, credential, credential-org, proxy, and tenant — plus per-model and per-provider ceilings, in any time window. The same three-tier permission model (System / Project / Team) that owns REST access owns who may edit those budgets, so the people who own the workload also own the bill.

Live token tracking — input, output, cached, and total — per request
Limits in tokens or in dollars, per minute, hour, day, or month (TPM + USD; request-per-minute is the platform throttle policy)
Six scope tiers — global → anonymous → credential → org → proxy → tenant — crossed with provider and model
Hard caps plus threshold alarms at 50%, 80%, 90%, and 100% of budget
Cost attribution back to a project or cost center — finance gets a line item, not a mystery
Auto fall-back to a cheaper model or a cached answer when the budget tips
Per-model unit pricing you edit in the UI, with cached tokens billed separately

Tracked per request: input · output · cached · total tokens
Quota windows: minute · hour · day · month
Quota scopes: global · anonymous · credential · org · proxy · tenant × provider · model
Measured in: tokens or USD
Enforcement: hard cap · threshold alarm · graceful fallback
Reporting: cost by project · model · team · time range

Cost-aware multi-LLM routing

Write the application once against an OpenAI-compatible endpoint. The gateway decides which model actually answers — cheapest, fastest, load-balanced, or whatever your routing rules say. Drop in a cheaper model for summarisation, send tier-one customers to the frontier model, fall back to a self-hosted model when a provider degrades. Requests are translated into each provider's native format on the way out, so a fall-back chain can cross vendor families mid-flight.

OpenAI-compatible request and response shape — no client rewrites, drop-in for the openai and langchain_openai SDKs
Seven routing algorithms — primary/fall-back chain, load-balanced pool, rule-based conditional, least-cost, least-latency, prefix affinity, and cheaper-model downgrade on budget overflow
Two-way OpenAI ↔ native translation — Anthropic, Gemini, Vertex, and Bedrock bodies rebuilt per leg, including cross-family failover
Provider fall-back chains with idempotent billing — a retried request is never counted twice, and a leg over its cost ceiling is skipped
Real per-chunk SSE streaming with backpressure and disconnect handling, trailing usage collected on stream end
Unary agentic tool-call loops with a turn ceiling and per-turn metering (streaming agentic loops are on the roadmap)

Native adaptersOpenAI · Anthropic · Gemini / Vertex · Bedrock · vLLM
Catalog17 providers · 111 models · UI-editable pricing
OpenAI-compatibleany provider or self-hosted model on the OpenAI wire format
Outbound authBedrock SigV4 · Vertex OAuth2 · bearer keys
Endpointschat/completions · responses · embeddings · audio · images

Exact and semantic response cache

Skip the token bill on repeat prompts. Apinizer answers an identical prompt from the Hazelcast cluster the REST gateway already runs, and a re-worded prompt from a real vector layer — embeddings scored against your own VectorDB above a similarity threshold you set. The cache stores the masked form of a response, so a cache hit can never replay data the PII policy stripped on the way out.

Exact-match keying by SHA-256 over the request, cluster-wide on Hazelcast
True vector layer — embedding similarity over pgvector, Qdrant, or Redis with your own threshold and top-K
Tune what gets keyed — last N messages, system / assistant / tool roles in or out, and `Cache-Control: no-store` and `no-cache` honoured
Masked-first caching — the semantic layer never stores the raw pre-mask body
Configurable TTL and size ceiling, proxy-scoped so two AI proxies never share a bucket
Atomic invalidation on redeploy, plus a TTL purge job for expired traces
Live hit-rate plus estimated token-spend saved (average miss cost × hits) and latency-saved reports per cache bucket

Exact match: SHA-256 request key on Hazelcast
Semantic match: embedding similarity · threshold + top-K
Vector stores: pgvector · Qdrant · Redis (RediSearch KNN)
Key tuning: message countback · role filters · cache-control
Safety: masked body cached · proxy-scoped isolation
Reports: hit rate · tokens saved · latency saved

Prompt templates and decorators on the gateway

The client sends a template name and a few parameters; the gateway assembles the real prompt. Templates live in a catalog scoped built-in, platform-wide, or per project, and decorators prepend the system blocks every team keeps forgetting — privacy, bias, security, scope. Application code stops carrying the prompt, and a prompt change ships through APIops instead of a release.

Template catalog with `{{param}}` placeholders, required flags, and defaults — three scopes: built-in, platform, project
Gateway-side expansion — single-pass substitution, closed to template and JSON-structural injection, with fail-closed size ceilings
Prompt decorators — prepend or append system blocks per role, merge into an existing system message, with `${env}` values resolved at runtime
Nine built-in responsible-AI presets: bias & fairness, explainability, privacy, content moderation, ethical AI, security, regulatory compliance, short responses, technology concealment
Enforced assembly order — template, then decorator, then RAG, then guards and cache — regardless of how the policies were arranged
Templates are exportable artifacts: wizard, REST, or APIops promotion from dev to test to prod

json

// The client sends this…{ "model": "gpt-4o-mini",  "template": "support-summary",  "template_params": { "tone": "formal", "ticket": "INC-4417" } }// …the gateway expands the catalog template, prepends the// privacy + regulatory-compliance decorators, injects RAG// context, then runs the guards and the cache — in that order.

RAG on your own VectorDB — nothing leaves the cluster

Ground answers in your documents without shipping them anywhere. Upload to a knowledge base, and Apinizer chunks, embeds, and indexes into a VectorDB you run. At request time the gateway retrieves and injects context before the guards and the cache see it — and the retrieval is bounded by project and knowledge base, so one tenant's question can never reach another tenant's chunks.

Knowledge bases with document upload, PDF text extraction, chunking, and embedding indexing
Four injection modes — prepend to the user turn, prepend or suffix the system message, or a template
Top-K and similarity threshold per policy, with an explicit behaviour when nothing matches
Multi-tenant isolation enforced at query time — project and knowledge-base filters on every retrieval
Fail-closed by default: when the project scope cannot be resolved, injection is skipped rather than widened
GDPR / KVKK erasure on indexed documents — delete the source, drop the vectors
pgvector, Qdrant, or Redis as the runtime store, with pooled connections and atomic redeploy

Sources: document upload · PDF extraction · chunk + embed
Stores: pgvector · Qdrant · Redis (RediSearch KNN)
Injection: prepend user · prepend system · system suffix · template
Tuning: top-K · similarity threshold · no-match behaviour
Isolation: project + knowledge-base filter, fail-closed
Order: retrieve → inject → guards → cache

Prompt firewall — injection, jailbreak, topic, and data loss

Block the patterns that put regulated AI projects on hold: prompt injection, jailbreak chains, conversation-structure tampering, credential exfiltration, off-topic prompts that burn budget, and runaway loops. The guards run inline — the bad request never reaches the model — and you can put a second-opinion model behind them when a regex is not enough.

Prompt guard — injection and jailbreak detection with preset pattern libraries, in inline, async, or shadow mode
Topic guard — embedding-similarity allow-lists and deny-lists with separate thresholds; deny wins, and it fails closed
Context-integrity guard — control tokens and forged role sequences caught before the model sees a tampered conversation
DLP guard — secret and credential presets, sticky block across both directions of the exchange
Oversize and loop guards — body ceilings and repeat-request patterns, so one client cannot drain a budget
External guardrail providers through one adapter — an LLM-as-judge endpoint, or any OpenAI-compatible judge (vLLM, Ollama)
Configurable AI and MCP error responses — your wording, your status codes, on every guard decision

Inline guards: injection · jailbreak · context integrity
Topic guard: allowed + denied topics · deny-first · fail-closed
Data guards: DLP presets · credentials · secrets
Budget guards: oversize body · loop detection
External engines: LLM-judge · OpenAI-compatible (vLLM · Ollama)
Latency modes: inline · async · shadow

PII masking that survives streaming

Turkish identity data is a first-class citizen, not a custom regex you maintain. TCKN, IBAN, and Turkish phone numbers are validated by checksum before anything is masked, so the gateway redacts real identifiers instead of any eleven digits it happens to see. Masking runs on a sliding window across SSE chunks — an identifier split across two chunks is still caught — and on MCP and A2A tool results through the same policy.

Twelve built-in types — TCKN, IBAN, Turkish phone, email, credit card, IP, URL, passport, US SSN, TR driver's licence, generic bank account, crypto wallet
Checksum validation wherever the format defines one — mod-97, Luhn, base58check, bech32, EIP-55
Six mask modes — fixed, keep-first, keep-last, mask-first, placeholder sequence, or synthetic-but-valid replacement — plus delete, encrypt, hash, and detect-only operations
Placeholder sequence keeps references consistent inside a request, so the model can still reason about "the same customer"
Chunk-boundary-safe masking on streaming responses, including the terminal events of the Responses API
Custom regex and element-name rules, plus a three-scope Turkish PII preset catalog
The same masking applied to MCP and A2A tool results — the tool output is not a side door

Turkish types: TCKN · IBAN-TR · phone-TR · driver's licence
International: email · card · IP · URL · passport · SSN · IBAN · wallet
Validation: checksum-verified before masking
Mask modes: fixed · keep-first · keep-last · mask-first · placeholder · synthetic
Streaming: sliding window across SSE chunks
Also covers: MCP and A2A tool results

AI observability — every prompt, every token, every model

The Analytics Engine ingests AI traffic next to REST traffic — 28 sparse AI fields on every log record. One query answers cost-by-team, latency-by-provider, error-rate-by-model, and which guard tripped last night. When you already run Grafana, the gateway exports native OTLP with GenAI semantic conventions and a Prometheus metric family, so AI traffic lands in the dashboards your SRE team already watches.

Token spend by user, project, team, provider, model, and time window — eight report breakdowns
Time-to-first-token and time-per-output-token per provider and per model, not just total latency
Live trace with a request DAG — a chained agent call is one tree, not five unrelated log lines
Timeline viewer and replay — re-run a captured request on the worker to reproduce what happened
Guardrail Hits and Cache Efficiency reports, plus a cost breakdown finance can read
Native OTLP export (HTTP and gRPC) with OpenTelemetry GenAI semantics, a Prometheus metric family, and a Grafana dashboard you can import
AI alerts and anomaly detection on spend and latency — the bill never surprises you twice

Cost dashboardstokens · dollars · by project, team, model
Latency dashboardsTTFT · TPOT · p50 · p99 · by provider
Tracinglive trace · request DAG · timeline · replay
Securityguardrail hits · masking counts · injection attempts
ExportOTLP GenAI semconv · Prometheus · Grafana dashboard
Anomaliesspend spikes · latency spikes · cache miss surges

MCP and A2A as first-class proxies

MCP and Agent2Agent are not a bolt-on server next to the gateway — they are proxy types alongside REST, SOAP, and AI, with the same policy chain, the same deploy history, the same rollback. When you open an MCP or A2A endpoint you pick one of two modes: expose the REST and SOAP APIs you already publish as tools, or pass through to an external MCP server or agent. Either way the agent authenticates through the same policies as every other client, and every call lands in the same audit record.

Two MCP modes — turn the REST and SOAP APIs you already publish into MCP tools, or pass through to external MCP servers with credential exchange and an allow-list
Two A2A modes — publish a proxy as an agent with a discoverable agent card, or front an external agent and rewrite its card
Standard policy chain for auth — JWT, OIDC, OAuth 2.0, API key, mTLS — no separate agent auth scheme to maintain
Default-deny identity ACL with tool slicing: which credential, role, or org may call which tool
Tool-catalog drift detection — the tool schema is pinned by hash, and a health job flags a silent change
Argument constraints per tool — regex, max length, enum, required — evaluated with a ReDoS deadline
Per-tool call quotas, and client-token passthrough off everywhere until you explicitly opt in
Aggregate up to ten backend MCP servers behind one path with deterministic namespacing and fail-soft discovery
A2A task lifecycle with streaming relay, resubscribe, push notification, and a cancel that actually interrupts the worker

MCP modes: expose REST / SOAP APIs as tools · passthrough
A2A modes: agent-expose · passthrough
MCP versions: 2024-11-05 · 2025-03-26 · 2025-06-18 (default) · 2025-11-25 · 2026-07-28
Access: default-deny ACL · per-tool RBAC · output filter
Governance: schema drift pin · argument constraints · call quota
Outbound: mTLS fail-closed · OAuth 2.0 · SSRF-guarded probes

One gateway, one audit, one runtime

AI Gateway is not a side-car. It is a layer of policies on the same gateway that runs your REST, gRPC, WebSocket, SOAP, and GraphQL traffic. Same identity, same audit, same observability, same operators. There is no second control plane to learn, no second pager rotation, no second invoice.

Same gateway runtime — REST, gRPC, WebSocket, SOAP, GraphQL, MCP, A2A, and AI on one process
Same identity surface — OAuth 2.0, OIDC, JWT, mTLS, SAML for humans, agents, and partners
Provider API keys stay on the LLM connection — applications authenticate with their own credentials, never with the upstream key
LDAP identity sync for AI consumers — paged fetch, scheduled jobs, sync history
Same audit at the persistence layer — bypass rejected at compile time
Same three-tier permission model — System, Project, Team — plus dedicated AI roles for development, budgets, and analytics
Same explicit deploy path — save, review, deploy, roll back; prompts and routes change without restarting a pod
Same Kubernetes posture — self-hosted, air-gap friendly, no data leaves the cluster

Runtimeone gateway process · API + AI
Identityhumans · agents · partners on one surface
Auditpersistence-layer, immutable, replayable
RBACSystem / Project / Team — everywhere
Deployhot — prompts, routes, models
PostureKubernetes-native · air-gap friendly

Use cases

Where teams put it to work.

Stop the AI bill from running away

Token budgets per project. Response cache on the hot path. Cheap-model fall-back when the budget tips. Cost attribution by team and project. The AI line item stops being a surprise, and finance gets a cost report instead of a Slack message.

Token and USD ceilings on credential, organization, proxy, and project scopes
Exact and semantic response cache with hit-rate and savings reports
Cheaper-model downgrade and least-cost routing off the catalog price
Cost attribution back to the project that ran the workload
Monthly cost exports for finance

In the box

What's included

The capabilities below are part of the standard install — no add-on SKUs and no separate licenses.

AI traffic types

Chat completions — unary and streaming
Responses API — unary and streaming
Embeddings — client-facing, shared with RAG and semantic cache
Audio — speech-to-text and text-to-speech
Image — generation
Function calling, tool use, and unary agentic loops
Agent-to-Agent (A2A) messages and tasks
Model Context Protocol (MCP) tool calls

Cost & token governance

Live token tracking — input, output, cached, total
Ceilings per window — minute, hour, day, month
Ceilings per owner tier — credential, organization, proxy, project
Ceilings per provider and per model, measured in tokens or dollars
Threshold alarms at 50 / 80 / 90 / 100 percent, with monthly reset
Auto fall-back to cached answer or cheaper model on budget tip
Cost exports back to the project that ran the workload

Security & guardrails

Prompt injection, jailbreak, and context-integrity guards
Topic guard with allow-lists and deny-lists on embedding similarity
Twelve built-in PII types with checksum validation, including TCKN and IBAN-TR
Six masking strategies, from redaction to synthetic-but-valid values
Masking that holds across streaming chunks and on tool results
DLP, oversize, and loop guards
External guardrail engines through one adapter (LLM-judge)
Default-deny tool RBAC per identity
Audit trail at the persistence layer

RAG & knowledge

Knowledge bases with upload, PDF extraction, chunking, and indexing
pgvector, Qdrant, or Redis as the vector store — self-hosted
Four RAG injection modes with top-K and similarity threshold
Project and knowledge-base isolation enforced at query time
Fail-closed when the project scope cannot be resolved
GDPR / KVKK erasure on indexed documents

Operability

Same gateway runtime as REST, SOAP, gRPC, WebSocket, MCP, and A2A
Same identity, same audit, same RBAC across API and AI
Explicit deploy with history and rollback; no pod restart for a prompt change
Three-tier permission model (System / Project / Team) plus dedicated AI roles
Live cost, latency, and reliability dashboards
OTLP, Prometheus, and Grafana export for existing SRE tooling
Kubernetes-native, air-gap-friendly deployment

Resources

Keep going

AI Gateway docs

Configure providers, set token budgets, write prompt firewall policies, and observe AI traffic alongside REST.

Read the docs

Cost & token playbook

Patterns for project budgets, cost attribution, response cache tuning, and cheap-model fall-back chains.

See the playbook

Provider quickstarts

Drop-in recipes for OpenAI, Anthropic, Bedrock, Azure OpenAI, Gemini, and self-hosted Llama or vLLM.

Browse providers

Prompt firewall reference

The guard catalog — injection, jailbreak, PII, credentials, off-topic, tool-use — with policy snippets.

Read the catalog

AI observability guide

Cost, latency, reliability, and firewall dashboards in the Analytics Engine — one query for API and AI.

Open the guide

Architecture overview

How the AI plane shares one runtime with the API Gateway, Identity Manager, Cache, and Analytics Engine.

See the suite

Migration from a side-car gateway

A short field guide for teams running a dedicated AI gateway today — what to keep, what to retire, what to consolidate.

Read the guide

Govern every AI request

Bring tokens, agents, and risk under one control plane.

A 30-minute walkthrough of the Apinizer AI Gateway — token budgets, multi-LLM routing, exact and semantic cache, prompt firewall, PII masking, RAG on your own VectorDB, MCP and A2A governance, and AI observability — on a Kubernetes of your choice.

Request a demo Read the docs