DVARA 1.12.0 does. The new Cost Advisor reads your last 30 days of calls and lists ways to spend less, each with what it saves a month. It also checks a cheaper model on your own traffic before it calls the switch safe. This post covers 1.11.0 too, which shipped the day before.

Ways to spend less, priced from your own traffic​

Cost → Cost Advisor in Flightdeck shows four kinds of suggestion:

• Use a cheaper model. When a workspace spends at least $50 a month on a model, the advisor looks for a model from the same provider that can do the same things: tools, structured output, reasoning, images and enough context. It picks the closest step down that costs at most 70% as much, not the cheapest one.
• Turn on the response cache. If the cache is off and some requests repeat one it would have answered, the advisor shows what those calls cost. One click turns the cache on.
• Route by tier. For a model whose traffic is often simple, it prices what routing by complexity would save, using a tier map you already have.
• Large system prompts. It flags an API key that sends a prompt of 2,000 tokens or more on every call, and what prompt caching would save. DVARA never changes a prompt.

Every figure is priced the way your bill is: on the tokens each call used, at your prices. A suggestion under $5 a month is not shown. Each one can be dismissed with a reason, and comes back only if its saving doubles. Every apply and dismissal goes on the audit trail. See Cost Advisor.

A cheaper model is checked before it is called safe​

A cheaper model only saves money if its answers are as good. So a workspace admin can let DVARA check it on the workspace's own requests.

After an answer has gone back to your application, the Gateway sends a small sample of requests (2%, at most 200 a day per Gateway) to the cheaper model. The request goes through the same policies, PII rules and guardrails as any other. A judge model then compares the two answers. Only the verdict is kept: no text of the request or either answer is stored.

After 30 checks, the suggestion shows Validated when quality held in at least 90% of them. Otherwise it shows Quality dropped, and DVARA stops trying that model. The checks are billed to the workspace, up to $10 a month unless you set another limit, and they never change an answer. It is off until the admin turns it on. See Check cheaper models on your traffic.

The current Anthropic SDKs work​

Anthropic's new API version, 2026-10-21, is what its current SDKs send: Java 3, Python 2, TypeScript 1, Go 2, C# 13, PHP 1, Ruby 2 and the ant CLI 2. Up to 1.11, DVARA's /v1/messages accepted only 2023-06-01, so a client that upgraded its SDK was refused.

1.12.0 serves both. Under 2026-10-21, DVARA refuses what that version removed, the same way Anthropic does: temperature, top_p and top_k, a document in an assistant message, and citations on a user's text. Clients on 2023-06-01, including Claude Code, work as before. See Use the Anthropic Messages API.

DVARA's own calls to Anthropic move to 2026-10-21 as well. When DVARA translates a request for Anthropic, for example one that came in on /v1/chat/completions, it now leaves out temperature and top_p, and the X-Gateway-Params-Not-Applied response header says so. To keep the old version, set ANTHROPIC_API_VERSION=2023-06-01 on the Gateway.

Token counts that match the bill​

DVARA counts tokens itself, for rate limits and for checking a call's cost before it is sent. In 1.11.0 we measured what Mistral bills. The text count was already exact. Mistral adds 8 tokens to every request, and 28 more when it has tools. DVARA now adds the same. You can set your own figure per model family. See Counting tokens.

Smaller fixes​

• A streamed answer from Cohere failed on its first event. It works now.
• The Python MCP SDK 2.3.0 works with DVARA on MCP's stateless revision, 2026-07-28, as a client and as a server behind the Gateway.
• Two quick edits to the tier catalogue both take effect. Before, the second could be missed until the file changed again.
• A Gateway that joins the cluster late now learns the agent sessions that were killed before it joined.

Read this before upgrading​

Nothing you set before stops working. Four things to know:

1. Clients on the current Anthropic SDKs now reach /v1/messages. Under 2026-10-21, leave out temperature, top_p and top_k.
2. DVARA calls Anthropic on 2026-10-21, so a translated request no longer sends temperature or top_p. ANTHROPIC_API_VERSION=2023-06-01 keeps the old behaviour.
3. Mistral counts are 8 tokens higher per request, 28 more with tools, as Mistral bills them.
4. Flightdeck adds one table for the Cost Advisor when it starts. Back up the database first.

The upgrade guide has every change.

Try DVARA 1.12​

The images ghcr.io/dvarahq/dvara-gateway:1.12.0 and ghcr.io/dvarahq/dvara-flightdeck:1.12.0 and the Helm chart 1.12.0 are out now. DVARA Open Source 1.12.0 is out too: get the release, follow the Open Source quickstart, or book a walkthrough of the Enterprise platform.

DVARA 1.10.0 is built around that. Routing explains each decision. Cost and tokens can be counted before a call. Live answers can be judged. And rules that are meant to block now block when they cannot check.

See what routing decided, and what it saved​

Intelligent routing sends each request to the cheapest model tier that can answer it. In 1.10.0 every routed answer says which tier it got, which model was asked for, which was chosen, and why. The audit record carries the same.

A route can now name a model for a reasoning tier, used when the caller asks for high reasoning effort. A workspace can set its lowest tier, so it is never served below it. The Usage page shows what routing saved against the model that was asked for. When the router itself calls a model to classify a request, that call goes through your policies, budgets and audit like any other. See Intelligent routing.

Count tokens and cost before you call​

POST /v1/tokens/count gives a request's token count, where it would go, how much of the context window it uses, and what it would cost, for up to ten models, without calling any of them. Flightdeck has a Token Calculator under Cost. Mistral and DeepSeek models are now counted with their own vocabularies.

Usage and spend are also split by the API a call came in on: Chat Completions, Responses, Messages, Gemini and the rest. Since each coding agent uses its own API, that shows what each agent costs. See Cost management.

Judge answers on live traffic​

DVARA can now score a sample of live answers as they are served: valid JSON when JSON was asked for, refusals, toxic content, length and patterns you set. Scoring never slows or fails a request, and every score is a signed audit event.

You can also name a judge model to grade sampled answers for accuracy, tone, policy adherence or your own rubric. The judge's call goes through the gateway, so your PII rules and budgets apply to what it is sent. Its spend has its own monthly limit and is kept out of chargeback. See Prompts and evals.

Rules that are meant to block now block​

A rule that cannot check a request used to let it through. In 1.10.0:

• A data-residency rule refuses a call whose region is unknown, and a budget rule refuses when budget use cannot be read.
• The ML injection classifier and the PII service can be set to refuse when they are down, and every time one lets a request through unchecked, it is recorded.
• A guardrail plugin set to fail closed treats a broken answer as a failure.
• Failover keeps to a workspace's allowed regions. A workspace can choose to be served elsewhere instead, and each time it is, that is recorded.

DVARA also finds model output that acts when it is shown: terminal escape sequences, and images or links that would send data to an untrusted host. A model file you supply can be pinned by its SHA-256. Gateways now talk to each other over TLS 1.3, and only those holding a certificate from the cluster's own authority can join. See Guardrails and ML and plugin guardrails.

Change more in Flightdeck, with no restart​

Providers can be added, turned on and off, and removed in Flightdeck. Retry, circuit breaker, timeouts and failover can be changed globally or per provider, with history and rollback. Each workspace has its own response cache and turns it on itself. Prompts can be kept for audit, apart from the signed record, and deleted after a retention period you set. See Resilience and Operations.

More from each provider​

Groq and Cohere now take tools and images, and Mistral takes images. Every Mistral model family, including Pixtral and Codestral, reaches Mistral by its name, with no route needed. See Multimodal.

The Enterprise Gateway also does more work per CPU core. On a laptop benchmark it went from stalling near 3,100 small requests a second, with the CPU idle, to 5,000 a second with every request served.

Read this before upgrading​

Most of 1.10.0 needs no action. Check these first:

1. The Gateways' cluster has a new port, 7800, and new metric and health names. Update network rules, dashboards and alerts that name the old ones.
2. Gateway pods talk TLS to each other. The Helm chart sets this up. With Compose or VMs and more than one Gateway, give each one a certificate.
3. Data-residency and budget rules that cannot check now refuse.
4. Each workspace's response cache is off until the workspace turns it on.
5. Auto-kill is on by default, so a looping agent session ends. There is no message-count cap by default.

Flightdeck applies the database changes when it starts. The upgrade guide has every change and how to keep the old behaviour.

Try DVARA 1.10​

The images ghcr.io/dvarahq/dvara-gateway:1.10.0 and ghcr.io/dvarahq/dvara-flightdeck:1.10.0 and the Helm chart 1.10.0 are out now. DVARA Open Source 1.10.0 is out too: get the release, follow the Open Source quickstart, or book a walkthrough of the Enterprise platform.

DVARA 1.9.0 brings them back onto the governed path. It also serves the newest MCP and A2A protocol versions, and it signs agent-to-agent delegation so each hop can prove whose authority it carries.

Govern coding agents​

Point a coding agent at DVARA, and every model call it makes goes through your keys, policies, guardrails, budgets, metering and audit. 1.9.0 supports:

• Codex CLI, through the Responses API, on its default settings.
• Gemini CLI, through a new Gemini API at /v1/gemini.
• GitHub Copilot: the Copilot CLI and VS Code Copilot Chat's Custom Endpoint models.
• OpenCode, through Chat Completions.
• Cursor, as a preview, through a base URL at /v1/cursor, in chat and agent modes.
• Claude Code, as before, through /v1/messages.

Copilot's inline completions, cloud agent and code review stay on GitHub. No gateway can govern them.

Tools that run at the provider, such as web search and code interpreter, are refused. DVARA cannot check what they do, so it does not let them through. Each setup page shows how to turn them off in the client. See Codex CLI, Gemini CLI, GitHub Copilot, OpenCode and Cursor.

Serve the newest MCP version​

DVARA now serves MCP 2026-07-28, the stateless version, beside 2025-11-25 and 2025-06-18. It passes the official MCP conformance suite.

Each governed server also has its own address, /mcp/{serverId}. A client that connects there sees just that server's tools, with their own names.

DVARA can now sign in to an upstream MCP server in more ways: static headers, OAuth client credentials, mutual TLS, AWS SigV4 and OAuth token exchange. See Authenticate to an upstream.

Speak A2A 1.0, and sign delegation chains​

DVARA serves A2A 1.0 by default, beside 0.3. Both official A2A SDKs and the official test kit pass against it.

When one agent hands work to another, DVARA now signs the delegation chain. It checks the chain at every hop, ties it to one workspace, and lets you revoke a single chain or everything one issuer signed. A hop that carries a signed chain cannot claim more authority than it was given. See the A2A delegation guard.

The Agent Card DVARA serves is written in your client's A2A version and signed. Flightdeck also adds an agent catalog, so you can register a known agent in one click.

Make passing on a caller's token a choice​

Some upstream servers and agents want the caller's own token. Sending it on unchanged lets anyone holding that token reach further than you meant. In 1.9.0 it is a choice you make on each server or agent, and each call that does it is recorded. Where the upstream supports it, DVARA can trade the token for a new one made only for that upstream.

See more of where spend goes​

Usage now shows spend by end user and by tag. It counts cached, cache-write and reasoning tokens, and the access log records them. A stream that fails is audited as failed, not as a served answer. Flightdeck warns you when a workspace's IP rules are saved but not enforced.

Read this before upgrading​

Check four things before you move production traffic:

1. A refused A2A JSON-RPC call now answers HTTP 200, with the error in the JSON-RPC body, so the official SDKs can read it. A bad key still gets 401. Set a2a.protocol.json-rpc-error-status=refusal to keep the old status codes.
2. Passing a caller's own token to an upstream is now an explicit, audited choice. The upgrade switches existing delegated servers and agents to it, so they keep working.
3. Tools that run at the provider are refused on /v1/responses and the Gemini API.
4. Workspace IP rules apply only when IP access is on at the Gateway, with DVARA_LLM_GATEWAY_IP_ACCESS_ENABLED=true.

Flightdeck applies the database changes when it starts. The upgrade guide has the details.

Try DVARA 1.9​

The images ghcr.io/dvarahq/dvara-gateway:1.9.0 and ghcr.io/dvarahq/dvara-flightdeck:1.9.0 and the Helm chart 1.9.0 are out now. DVARA Open Source 1.9.0 is out too: get the release, follow the Open Source quickstart, or book a walkthrough of the Enterprise platform.

Carry a signed delegation chain with the call: a record of which agents it has already passed through. Check it before calling the next agent. With strict agent matching and enforcement turned on, a return to an agent already in the chain stops there. In DVARA, you can also allow a return for a different skill. The choice matters, as does requiring agents to pass the chain along: a signature prevents edits, but it cannot stop an agent from leaving the chain out.

Updated October 2, 2026 for DVARA 1.8.5.

The problem: two good agents make one loop​

Take two agents in a travel workspace. Planner builds trip plans and Budget prices them.

A user asks Planner for "a 3-day Lisbon trip under €1,500". Planner asks Budget to price its draft. Budget needs the day-by-day itinerary, so it asks Planner. To build the itinerary, Planner needs a price, so it asks Budget again.

Each agent is asking for information it needs. The problem is that neither knows the other is waiting on it. Every trip around that loop adds work and model calls to your bill.

Adding agents just makes the loop harder to see. Planner → Budget → FX → Tax → Planner has the same problem, with more steps before anyone returns to Planner.

Why the usual fixes fall short​

Timeouts​

A timeout limits how long a loop runs, but the work before it expires has already happened. At one hop every two seconds, a 60-second timeout leaves room for roughly 30 hops.

Loop detectors and rate limits​

A rate limit stops calls after they reach a threshold. A session loop detector looks for repeated patterns. Both let some work through before acting. For example, a detector that waits for three repetitions of a short cycle handles Planner and Budget like this:

Six hops go through before the seventh is refused. The detector also needs to know which calls belong together. DVARA looks for X-Session-Id first, then the session in the delegation chain, then the message's contextId. Losing the header alone is therefore recoverable. Lose all three, and DVARA has to create a new session id; it can no longer connect the call to the earlier work.

A per-session list of agents​

You could record every agent a session has touched and refuse a repeat. But that refuses valid work. Planner may call Budget and Weather at the same time, and later call Budget again for a different trip in the same session. One shared list mixes those branches and refuses calls that are fine.

Letting agents name themselves​

If the check relies on a header in which the caller names itself, any caller can claim any name, or leave out the part of the chain that would get it refused.

The general answer, rule by rule​

A gateway, message bus or orchestrator can make the decision before forwarding each call. It needs a few things from the caller and a few rules of its own:

1. Put the chain on the call. Each call carries the path that led to it: planner → budget. The agent that receives it passes it on with its own onward calls, the way it already forwards a trace id.
2. Refuse a cycle. If the target is already in the chain, refuse the call. You can choose how strict to be. "Same agent" refuses any return to an agent. "Same agent and same job" lets Planner be asked for a draft and later for a check, but refuses the same request twice.
3. Set a depth limit. Decide how many agents a task should need. Refuse the next hop when it exceeds that depth, even if every agent or skill name is different.
4. Let the router own and sign the chain. The component that forwards the call writes the chain and signs it. An agent can carry the chain but cannot change it. A chain that was changed, has expired, or belongs to another workspace or session is refused. Fail closed: a bad chain is never treated as "no chain".
5. Keep parallel branches separate. Check planner → budget against its own chain and planner → weather against its own. A session detector running behind that check must respect those branches too.
6. Expire the chain. Count its lifetime from the first call, without renewing it at each hop. Require the chain on onward calls so an agent cannot reset its depth or age by omitting it. Use a separate session hop budget to limit total work across branches.
7. Tell the caller to stop this delegation. Return 409 Conflict with a cycle error code. The agent needs a handler for it: finish with the information available, return an incomplete result, or ask a person for help. Retrying the same delegation will fail again.

Keeping the chain on the call means another gateway with the same signing key can check it without fetching its history. That check is stateless. Counting calls across a session is a separate job, and multiple gateways need shared accounting to enforce one session limit.

How DVARA does it​

The DVARA A2A Gateway has checked delegation chains since 1.8.3. Version 1.8.5 adds a hardened preset and a session hop budget, and fixes how the session detector handles parallel sibling calls.

These are Enterprise platform capabilities. You can use them without a licence in Development for non-production work within 3 workspaces and 100,000 governed calls a month. Production rights and support require a licence. The DVARA Open Source distribution does not include the A2A Gateway.

With a usable signing key and the guard enabled, the gateway signs a delegation chain and sends it with each hop, in a header called X-DVARA-Delegation. The agent passes it back to the gateway on its next outgoing call, together with its session id. Before the gateway forwards a hop, the guard checks the chain for a cycle and for depth. A refused hop never reaches the peer, is never billed, and does not count toward the workspace's monthly hop cap.

Here is the same Planner and Budget example in enforce mode, with each agent preserving its chain:

With the strict rule, Budget gets this on hop 3, and Planner is never called:

409 a2a_delegation_cycle
Delegation cycle: agent planner is already in chain planner → budget.

Budget now has a choice to make in its error handler. It can price the draft it already has, or return an incomplete result if the draft is not enough. It must not retry a2a_delegation_cycle. DVARA stops the call at hop 3; Budget still has to decide how to answer Planner.

The default rule, same agent and same skill, lets Budget ask Planner for the itinerary once, because that is a new job. It refuses the repeat on hop 4. The default depth limit is 8, and each workspace picks its own rule and depth.

Enable the hardened preset in 1.8.5​

If you want to stop the first return to an agent, choose a2a.delegation.preset: hardened for the fleet or workspace. This is opt-in. It selects the agent rule, requires chains from agent-bound keys with the root-key exception below, and sets a 200-hop session budget unless you specify another value. You still choose the mode separately: observe records what would be refused; enforce refuses it.

Existing installations keep the standard defaults: mode: enforce, cycle-rule: agent-skill, max-depth: 8, and require-chain-for-agents: false, all under a2a.delegation. The session safety budget is off under that preset. a2a.delegation.chain-ttl defaults to 15 minutes from the root call. The delegation guard guide covers the full configuration.

Check that the gateway can sign chains​

Set DVARA_A2A_DELEGATION_SIGNING_SECRET to a private random secret of at least 32 bytes, shared by the gateway pods. If it is unset, DVARA derives a delegation key from a usable DVARA_AUDIT_HMAC_SECRET. Published placeholder secrets are not usable signing keys.

Check gateway status before testing a loop. In 1.8.5, a production profile refuses to start without a signing secret if the fleet asks for an active guard. A development profile can start with the guard effectively off. Gateway status shows the configured and effective fleet modes; check those alongside your workspace's mode. An enforce setting is no help if the pod cannot sign chains.

Separate root keys from onward-agent keys​

Bind each agent's API key to that agent. When chains are required and the guard enforces them, an ordinary agent-bound key without a chain receives 403 a2a_delegation_chain_required.

Some calls have to start somewhere. A key marked “may start chains” can start new work without an incoming chain; DVARA puts its bound agent at the beginning of the new chain. Unbound application keys can start roots too. Keep those credentials at trusted entry points and give agents separate keys for onward calls. Otherwise, an agent can use its root permission to start again.

Under the standard defaults, dropping the token from both the header and message metadata starts a fresh chain. DVARA also accepts the token in message.metadata["dvara.delegation"], so dropping only the header does not reset the chain when that metadata remains. The session detector still provides a backstop when calls remain correlated.

Observe the effect before enforcing​

Start in observe mode. Review would-refuse events, the Console's hop markers, and the depth metric before switching to enforce. In observe mode, cycle, depth, missing-chain and hop-budget violations are recorded without blocking the call. Invalid signatures still fail closed, and other controls such as session loop detection remain active.

The API-key binding identifies the caller. Possessing a chain does not establish an agent's identity. DVARA removes dvara.delegation metadata from replies, including nested values and escaped JSON keys in streams, and keeps the token out of its delegation logs. It does not scrub arbitrary text in which a peer has copied the token. Agents need to treat it as sensitive too.

Bound session work separately from billing​

In 1.8.5, a2a.delegation.max-hops-per-session limits how many hops a session forwards. It is separate from chain depth and from the workspace's monthly billing allowance. Under the standard preset, 0 means no safety limit. Under hardened, 0 selects the preset's 200-hop limit; an explicit positive value replaces it.

Once the budget is used up, enforce mode returns 409 a2a_delegation_hop_budget_exceeded before the next call reaches the peer. Handle that as a stop for this work, not a reason to open another session. If DVARA cannot read the count, hardened enforcement returns 503 a2a_delegation_hop_budget_unavailable. Standard mode lets that call through uncounted and records a warning and metric. In observe, exceeding the budget is recorded but does not block the call.

If you run several gateway pods, make sure they share the same accounting cluster. The local fallback only counts calls on its own pod. Separate clusters, including those split by a partition, each count their own traffic. Cluster-backed counts also expire after an hour idle. These are limits on session work, not a permanent spending cap.

Do not rely on the monthly A2A allowance to stop a runaway session. An unlimited allowance or billable overage permits more calls, and an unavailable usage count does not stop them either. Set the session safety budget for that purpose.

What stays as a backstop​

The delegation guard works alongside session loop detection, the session kill switch, billing controls, A2A policies and configured human-approval rules. In 1.8.5, identical sibling hops carrying the same verified parent chain count once for repetition and cycle detection. Each still counts toward the session rate limit and any enabled safety budget. A rate refusal uses 429 a2a_session_rate_limited and does not kill the session.

That sibling handling depends on a verified chain. Calls without one still go through the session detector's heuristic checks. Across multiple pods, shared loop history depends on the accounting cluster; a local fallback sees only the calls on that pod.

One choice deserves particular care: the default agent-skill rule trusts the skill name the caller sends. Changing that name avoids a cycle match, although a preserved chain will still hit its depth limit. If a return to the same agent should always stop, use strict matching. Keep onward calls on keys that require a chain, and preserve their session context, so those checks have the history they need.

Try it​

Stop recursive delegation between AI agents covers the settings, error codes, and what an agent builder must forward. Govern peer-agent calls with the A2A Gateway covers registration, policy, PII and approvals on each hop.

DVARA 1.8.0 puts those calls on one governed path. It introduces a clean Open Source foundation for LLM traffic and strengthens the Enterprise controls used across models, tools, and agents.

Start with Open Source​

DVARA Open Source is an Apache-2.0 LLM Gateway that runs with file configuration and no database. It gives you a practical way to put policy, PII checks, guardrails, rate limits, and a local tamper-evident audit file in front of model traffic.

You can run it as a standalone gateway or embed its governance libraries in a Spring Boot application. The configuration lives in gateway.yaml, so the first deployment is easy to inspect, version, and reproduce.

The Enterprise platform builds on the same foundation. It adds Flightdeck, central persistence, fleet operations, advanced LLM controls, and the DVARA MCP and A2A Gateways.

Make permissions real​

API-key scopes are now enforced across the LLM, MCP, and A2A planes. A key can be limited to the operations its workload needs—for example, model discovery, embeddings, MCP reads, tool calls, or A2A sends.

A valid key used outside its scopes is refused with 403. That turns scopes from descriptive metadata into an authorization boundary.

Say exactly what happens to sensitive data​

In 1.8, REDACT and TOKENIZE mean different things:

The distinction matters. A compliance review should not have to guess whether “redacted” data can later be recovered.

Hold protected streams until the decision is final​

An observational streaming policy can still deliver events immediately while DVARA records evidence. A control that can block or transform content now holds the complete response, makes one decision, and then releases governed content or refuses it.

DVARA also treats a provider stream that ends without a finish reason as an error. A truncated answer no longer looks like a successful, billable response.

Govern tools and agent handoffs​

The Enterprise MCP and A2A Gateways apply the same identity, policy, PII, budget, approval, loop-detection, and audit controls to tool calls and agent handoffs.

Streaming LLM tool calls now retain their IDs, types, function names, and arguments through provider translation. DVARA records the intent; your application still decides whether to execute the tool.

Give operators evidence they can use​

The release adds clearer operational signals around refused requests, protected stream buffers, incomplete streams, webhook delivery, cluster membership, and provider failures. File-backed audit chains can be verified with coverage as well as a pass/fail result.

Provider failures are also classified more carefully. A bad key or most other provider 4xx responses no longer open a shared circuit for every workspace. Rate limits, timeouts, and fallback behavior now follow the reason for the failure instead of treating every error alike.

Read this before upgrading​

Several controls that were previously permissive or inactive now take effect. Check these before moving production traffic:

1. A non-empty API-key scope list is enforced.
2. REDACT is irreversible; use TOKENIZE when you need reversible request-side substitution.
3. Tool allowlists and denylists now match and enforce their rules.
4. Context limits come from the serving provider, and the maximum-input-token guardrail is off by default.
5. Provider 4xx responses other than 408 and 429 do not count against the shared circuit breaker.

The 1.8 upgrade guide covers the complete preflight, rollout, verification, and rollback sequence.

Try DVARA 1.8​

Download the Open Source 1.8.0 release, follow the Open Source quickstart, compare capability availability, or book a walkthrough for the Enterprise platform.

Most platform leads can't answer. Not roughly, not which providers, not which teams. That gap has a name — shadow AI — and it is a visibility problem long before it is a policy problem.

How it happens​

Nobody decides to have shadow AI. It arrives one reasonable decision at a time.

One team adds an OpenAI key to a service. Another wires up Anthropic for a different feature. A third is calling a model through a SaaS tool that nobody logged as an AI vendor. A fourth is running an agent that calls tools on its own.

Four sensible decisions. Four paths out of your network, four sets of credentials, four logging conventions. No single place the traffic passes through — so no single place to see it.

Four questions that measure it​

Ask these about yesterday, not last quarter.

1. Which applications called a model, and which model did each one use?
2. What did they send? Would you know if customer data had left?
3. Who authorised it? For an agent: which tool did it call, on whose behalf?
4. Prove it. Produce a record you would hand an auditor.

If those take a week of Slack archaeology, the problem isn't that your policy is badly written. There is nothing in the request path that could enforce a policy even if it were perfect.

Why the usual answers don't close it​

• A written policy describes intent. It doesn't change what happens on the next request.
• Your API gateway sees an HTTPS POST to a hostname. Not the model, not the tokens, not the prompt, not the tool call.
• Per-app logging is four formats in four places that stop the moment someone is in a hurry.
• Provider dashboards are organised around billing, one per provider, and know nothing about that SaaS tool.

Each is useful. None produces an inventory.

The fix is an old one​

When services multiplied, we stopped asking every team to get retries and mTLS right and put a service mesh in the path. When APIs multiplied, we put a gateway in front.

Same move. Route model traffic through one point and visibility stops depending on whether every team remembered. It becomes a property of the architecture.

The control has to sit in the request path. Something that observes and reports is a monitoring tool — if it can't refuse a call, it can't enforce anything.

What it costs to adopt​

Almost every provider now speaks an OpenAI-compatible API, so this is a base URL, not a rewrite.

Python — before. Each service holds its own key and makes its own way out.

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

After. Same SDK, same call sites.

client = OpenAI(
    base_url="https://dvara.internal/v1",
    api_key=os.environ["DVARA_API_KEY"],
)

Java, Spring AI. Configuration only — no Java changes.

spring:
  ai:
    openai:
      base-url: ${DVARA_URL:http://localhost:8080/v1}
      api-key: ${DVARA_API_KEY}

One gotcha worth an afternoon of your life: Spring AI 2.0 uses the official OpenAI SDK, which POSTs to ${base-url}/chat/completions. The base URL must include /v1 yourself — unlike Spring AI 1.x, the SDK does not prepend it.

Agents travel the same road. Point the MCP client at the governed endpoint and tool calls are governed too:

spring:
  ai:
    mcp:
      client:
        streamable-http:
          connections:
            dvara:
              url: ${DVARA_MCP_URL:http://localhost:8080/mcp}

That matters more than it looks. Most AI-governance conversations stop at the model call. But an agent that can call tools can read a database, hit an internal API, or move money. The model call is the least dangerous thing an agent does.

What this actually produced​

Everything below is from a local run against the mock provider — no real provider traffic, so the model names are stand-ins. What is real is the record the gateway kept without any application being changed.

A hundred and twenty-four governed calls later, the inventory that did not exist:

That table is a GROUP BY over the stream above it — every governed call, with the model it went to and the workspace it came from, in one place. Question one, answered by a query rather than a week of archaeology.

Two requests carrying a US Social Security number were detected on the way out and recorded as PII_DETECTED, with the entity type — before the call left the boundary.

And the audit trail is a hash chain rather than a log file that claims to be one. 429 signed envelopes, each carrying its own HMAC and its predecessor's hash.

The chain is per writing process, which is the part worth knowing before you check one: this install had four — the gateway, the console, and two more from restarts — so there are four chains, not one, and each numbers its own envelopes from 1. Checking each chain against itself:

• every chain runs 1 to N with no gaps (420, 4, 3 and 2 envelopes)
• zero envelopes, across all 429, whose previous_hash failed to match the preceding hmac

Compare envelopes across chains instead and you will get breaks that are not there — the sequence numbers interleave. It is the first thing to get wrong.

Remove or edit a record and the chain stops matching at that point. That is the difference between logs and evidence.

Finally, visibility is only worth having if you can act on it. A rule denying one model:

version: "1"
rules:
  - id: no-expensive-model
    conditions:
      model:
        denylist: [mock/gpt-4o-expensive]
    action: DENY

The refusal names the rule that fired, and it happens before the provider is called — nothing sent, nothing billed. Every other model keeps working.

The honest objection​

One point in the path is one point of failure, and anyone who has run infrastructure will say so immediately. That has to be answered in the design, not the pitch: capability-aware failover so one provider outage doesn't take the path down, circuit breakers, and the ability to run inside your own perimeter so the control point isn't a new third party holding your prompts.

If the thing you added for governance becomes the reason you're down, you've made the system worse, and no audit trail compensates for that.

Where DVARA fits​

This is what we build. DVARA is a self-hosted control plane that sits in the request path for model calls, MCP tool calls and agent-to-agent hops, applying one policy set and one audit trail across all three. Everything above is its actual configuration and its actual output.

We work across all three planes rather than the model call alone for the reason in the middle of this piece: by the time an agent is invoking tools, the model call is the least consequential hop in the chain.

If you can already answer the four questions at the top, you don't need us. If answering them would take a week, the gap is architectural — and it doesn't close by writing it down.

Two editions, one download away​

There is no separate Community build to hunt for. DVARA decides the edition at startup:

• A valid, signature-verified license → Enterprise Edition. The full governance graph activates.
• No license, an invalid one, or one expired beyond its 14-day grace → Community Edition. The gateway boots and passes traffic on the Community feature set instead of failing to start.

The current edition is reported as edition (COMMUNITY / ENTERPRISE) on /actuator/gateway-status, so an operator can always confirm which one is running.

What is new is that the free image is now genuinely free of the paid code. The public dvara-llm-gateway and dvara-flightdeck images are a slim build that leaves out every paid Enterprise engine — policy, FinOps, signed audit, SSO and security, advanced routing, agentic, and cache. A security team evaluating Community Edition is no longer, byte-for-byte, holding the enterprise code. The full Enterprise build ships as separate private images (dvara-llm-gateway-ee, dvara-flightdeck-ee), and the runtime license still gates Enterprise on those.

One thing that does not change by edition: Postgres is required for every edition. Community Edition is a persistent, single-environment deployment — not an in-memory or dev-only mode.

What Community Edition includes​

The whole LLM gateway, unrestricted:

• The OpenAI-compatible surface — /v1/chat/completions, /completions, /responses, /embeddings, /models, /batches, /files
• 15+ providers, provider switching by model string, default fallback, and the resilience layer (circuit breakers, retries)
• API-key authentication with workspace plumbing, request and access logging
• Raw token-usage metering, Prometheus metrics
• The Console, with built-in username/password login and RBAC roles

On top of that, four scaled-down governance features so a free install shows what governance actually feels like rather than being a bare proxy:

• Static model allow/deny list — a single list, no policy DSL and no dry-run. A blocked model returns POLICY_DENIED.
• Local request log — who, when, which model, how many tokens, viewable in the Console, retained about 7 days. It is unsigned: no HMAC, no hash chain, no SIEM export.
• Regex-only PII detection — the built-in patterns (SSN, checksum-validated card numbers), with log or block as the action. No redaction, no tokenization.
• Read-only cost estimate — per-model spend visibility. No budgets, no caps, no enforcement.

Where the line is​

The shape of that split is deliberate. In Community Edition you can see your traffic, catch an obvious SSN, and block an expensive model — enough to feel what governance does. What you cannot do is prove any of it to an auditor, enforce it richly, or protect the data itself. Those are the things a regulated production deployment actually needs, and they are what Enterprise Edition is for.

Start in five minutes​

A single docker run of the public Community Edition image — no API key, no license — takes you from nothing to a first governed request, verified against the published image. A second step shows governance actually doing something: the Community deny-list teaser blocks a request with POLICY_DENIED, so the first thing a new user sees is a policy firing, not a bare proxy. The quickstart walks the whole path, including how to upgrade to Enterprise later.

A trust surface for a closed-source product​

Community Edition is free, but not open source — DVARA does not publish its source. So 1.6.0 ships the instrument that stands in place of source access: a responsible-disclosure process — SECURITY.md, with a coordinated-disclosure SLA and safe harbor for good-faith research — backed by a live /.well-known/security.txt.

If you cannot read the code, you can at least see how vulnerabilities are handled.

Opt-in usage telemetry. Community Edition can now send a small usage signal — off by default, and Community-Edition only. A licensed or air-gapped install never sends anything, and not because a switch is set the right way: the telemetry simply isn't active outside Community Edition. When you turn it on, five non-sensitive fields leave the box: an anonymous install id, the edition, the version, a coarse request-volume bucket (a range, never an exact count), and a timestamp. Nothing else — no prompts, no responses, no PII, no provider or model names. The full payload is documented field-by-field, so you can see exactly what would be sent before enabling it.

No lock-in​

Community Edition changes what happens when a paid license lapses, and it is worth being precise.

An Enterprise license that expires gets a 14-day grace period, fully operational, with warning headers and audit events. After the grace period the data plane returns 402 LICENSE_EXPIRED — administrative reads stay available so an operator can still diagnose and export configuration. That block is intentional: a paid product should be able to stop on non-payment.

What Community Edition adds is a floor beneath it. An operator who does not want to renew can drop to the free edition and keep running — free and unblocked, at the Community feature set. So the honest promise is not "Enterprise never stops." It is you can always fall back to the free edition and keep serving traffic.

Turning on Enterprise​

Because the public base-name images are now slim Community Edition, turning on Enterprise is a re-image, not a key-swap. Running Enterprise means pulling the private -ee image with a customer registry credential (docker login ghcr.io) and applying a license. Concretely, moving from a free install to a paid one:

• Point your deployment at dvara-llm-gateway-ee / dvara-flightdeck-ee instead of the public names.
• Add an imagePullSecret (or docker login) for the private registry.
• Apply your license as before.

Existing Enterprise deployments that already pull a full image and apply a license are unaffected — this only changes the base-name images.

Also in 1.6.0 — signed Agent Cards on the A2A plane​

Alongside the Community Edition work, the agent-to-agent governance plane gets the piece everything downstream depends on: knowing a peer agent is who it claims to be.

When DVARA discovers a peer agent, it now verifies the cryptographic signature on that agent's Agent Card, following the A2A specification's signed-card format. It surfaces a trust state — VERIFIED, SELF_SIGNED, UNSIGNED, or INVALID — on the served card, in the agent registry, and on the discovery audit event, and it enforces it: a signature that fails against the operator-pinned key is rejected outright, and a strict mode (require-signed-cards) refuses anything that is not fully VERIFIED. By default it is fail-soft, so you can adopt it without breaking existing peers. The operator-pinned public key on each agent — set through the Automation API or the Console, with a trust badge on the A2A agents page — is the trust anchor the delegation chain will build on in 1.7.0.

1.6.0 also freezes the design for cryptographic delegation: a signed delegation chain with strict scope-subset attenuation, where a delegated token can only ever narrow authority, never widen it. And two quiet correctness gaps are fixed — skill-scoped policy rules and approval.required-skills now actually govern A2A hops, and a peer-supplied context id is now recorded in per-hop metering instead of coming through empty.

Upgrade​

1.6.0 is a plain version bump over 1.5.0 — same image names, same public/private split, same registry model. Pull the 1.6.0 images (or the oci://ghcr.io/dvarahq/charts/dvara:1.6.0 chart), keep your license and configuration, and roll.

If you are new to DVARA: it is self-managed, activated by a signed license key, with bring-your-own-key credentials so provider keys and request data never leave your perimeter. Start free with the public Community Edition quickstart, or get a 30-day, all-features trial license — every feature unlocked, including the MCP and A2A planes. See the editions or, for production licensing or a managed deployment in your region, talk to us.

This post is a practical definition of A2A: what the protocol actually is, why the delegation hop between two agents is a blind spot your existing controls don't cover, and what it takes to govern it.

What is agent-to-agent (A2A)?​

Agent-to-agent (A2A) is a protocol for one AI agent to delegate work to another. Instead of a single monolithic agent that does everything, you get specialists: a scheduling agent, a billing agent, a research agent, a customer-record agent. When the scheduling agent needs a customer's billing status, it doesn't reimplement billing — it asks the billing agent. That request, and the reply, is an A2A hop.

The emerging A2A protocol standardizes what that looks like on the wire. A few concepts matter for the rest of this post:

• Agent Cards. Each agent publishes a machine-readable description of itself at a well-known discovery endpoint (/.well-known/agent-card.json) — its identity, endpoint, and the skills it offers. Other agents read the card to decide who can do what.
• Messages and tasks. A caller sends a message (message/send), optionally streams the response (message/stream), and can manage long-running work through a task lifecycle (tasks/get, list, cancel, resubscribe).
• Push notifications. For work that outlives a single request, a caller can register a webhook so the peer can call it back when a task completes.
• Delegated authority. A hop often carries an on-behalf-of credential — the calling agent is acting for an end user, and that user's authority rides along with the request.

None of this is exotic. It's a normal RPC pattern. What makes it dangerous is where it sits.

Why the A2A hop is a new attack surface​

Your LLM gateway governs the call from your app to the model. If you've adopted an MCP Gateway, you also govern the call from your agent to its tools. The A2A hop is the one in between those two that nobody instruments — and it is arguably the highest-risk of the three, because a hop to another agent is a hop to something that will itself go make more calls.

Left ungoverned, the delegation hop is invisible in five specific ways:

1. No caller identity. When Agent A calls Agent B directly, B often has no reliable idea which workspace, which user, which application is really behind the request. Authority gets muddled the moment it's delegated.
2. No policy on what may be delegated. There's no control point that says "the scheduling agent may ask the billing agent for status, but never for card numbers." Any agent can ask any peer for any skill it advertises.
3. No scan of the data that crosses. A message from one agent to another can carry PII, secrets, or a whole customer record in its payload — and it leaves your perimeter for the peer with nobody looking at it.
4. No record that it happened. When an auditor asks "show me every time an agent acted on this customer's behalf," there is no trail. The hop simply isn't logged anywhere that survives.
5. No brake on runaway delegation. Agent A asks B, which asks C, which asks A again. Without loop detection across the delegation chain, a multi-agent system can spiral — burning budget and hammering downstream systems — with no kill switch.

On top of those, the protocol's own features add sharp edges. Push-notification webhooks are a classic SSRF vector — a workspace-supplied callback URL pointed at 169.254.169.254 or an internal service is a data-exfiltration or metadata-theft primitive if the proxy forwards it blindly. And Agent Cards over-advertise by default: an agent that lists every skill it can do, to every caller, is a discovery-time information leak.

This is the same lesson the industry already learned with tools and MCP — an agent will send an SSN to a tool that didn't need it, because models aren't trained to think about data minimization — except now the "tool" is another autonomous agent that will take that data and act on it.

What it means to govern agent-to-agent calls​

Governing A2A means putting a control point on the delegation hop — the same move that an LLM Gateway makes for model calls and an MCP Gateway makes for tool calls. Every hop passes through one place that can authenticate it, decide it, scan it, and record it, before the message reaches the peer.

Concretely, agent-to-agent governance is a set of controls on that hop:

• Caller identity on every hop. Resolve the calling agent to a real workspace (and, via the delegated on-behalf-of credential, a real end user) before anything else runs. Delegation without identity is the root cause of most of the risks above.
• A peer-agent registry with per-workspace scope. Register each peer agent once — its endpoint, auth scheme, and credentials — then scope which agents and which skills each workspace is allowed to reach. Default-deny, not default-reach.
• Policy on every hop. Allow or deny a hop by caller, target agent, or requested skill, expressed as policy-as-code with dry-run, versioning, and rollback — so "who may delegate what to whom" is a reviewable, testable artifact, not a hard-coded assumption inside an agent.
• PII scanning on the message. Inspect the outbound message — and streamed events — for PII, then block or redact before it crosses to the peer. Data minimization enforced at the boundary, not hoped for.
• Human approval gates. For sensitive hops — a particular skill, a particular target agent — pause and require a human to approve before execution, resolvable from a console or portal.
• Delegation-loop detection. Watch the chain of hops and kill a session that starts to spiral before it runs away.
• Governed discovery. Serve each peer's Agent Card through the control point with a per-workspace skills allow-list, so agents advertise only the skills you intend — closing the over-advertisement leak.
• Centralized credentials. Hold peer-agent credentials centrally — stored, or forwarded on-behalf-of — so calling agents never see raw secrets.
• A tamper-evident audit trail. Sign and hash-chain every hop into an append-only record, so "which agent acted on whose behalf, and what data crossed" is answerable — and provably un-altered.

That last one is the point where A2A governance meets compliance. A delegation you can't prove happened is a delegation you can't defend in an audit; a tamper-evident AI audit log of every hop is what turns "our agents talk to each other" into "here is exactly what they did."

How DVARA governs A2A: a separate, independent plane​

DVARA treats A2A as a first-class plane — a peer to the LLM plane and the MCP plane, not a bolt-on. It has its own governance engines and its own audit chain, so your model and tool traffic are unaffected by what your agents delegate.

Updated for 1.7.0. This post originally said the A2A plane ran as its own proxy on :8075. In 1.7.0 the A2A and MCP planes moved in-process — the gateway serves them on its own port, so there is no second service to deploy. What the word "plane" was pointing at is unchanged: separate governance engines, a separate audit chain, and activation only when you licence it. Current documentation calls this component the DVARA A2A Gateway.

Updated for 1.8.2. This post and the note above originally said the A2A plane activates only under a licence. In 1.8.2 every feature, the MCP and A2A planes included, runs without a licence for non-production use, within 3 workspaces and 100,000 governed calls a month. A licence removes both limits and gives production rights and support. An expired licence refuses no traffic.

The hop flow is deliberately simple: DVARA governs exactly one hop, returns the peer's reply unchanged, and makes no onward call of its own.

1. Agent issues a hop. Your agent sends its A2A message to a peer through the DVARA A2A Gateway instead of calling the peer directly.
2. Identity & registry. The proxy authenticates the caller — API key to workspace — and resolves the target agent from that workspace's registry.
3. Policy, PII & approval. The hop is checked against A2A policy, the message is scanned for PII, sensitive hops pause for human approval, and loop detection guards the delegation chain.
4. Forward & audit. The message reaches the peer, the reply is returned unchanged, and every step is signed into the A2A audit chain and metered per hop.

Coverage spans the whole protocol — message/send, streaming message/stream, the task lifecycle, and push-notification config — with the workspace-supplied webhook URLs SSRF-validated before they're ever forwarded. The A2A audit chain is its own append-only, HMAC-signed, hash-chained trail, separate from the LLM/MCP chain but unioned into the same SIEM exports and compliance reports.

Where A2A fits: the three planes of AI governance​

The shape here is not new — it's the third instance of the same pattern. Governance belongs at the boundary every request already crosses:

• The LLM plane governs your app-to-model calls — routing, policy-as-code, PII, cost, and audit on every completion.
• The MCP plane governs your agent-to-tool calls — MCP tool-call governance with per-tool policy, PII scanning, approval gates, and audit.
• The A2A plane governs your agent-to-agent calls — agent-to-agent governance with identity, per-hop policy, PII, approval, loop detection, and its own audit chain.

Model, tool, agent. Three call types, one governance model, one tamper-evident audit story across all three. If your agents already call models and tools through a governed boundary, extending that boundary to the hops they make to each other is the natural next step — and it's the one that most teams haven't taken yet.

Frequently asked questions​

What does A2A stand for? Agent-to-agent — a protocol for one AI agent to delegate work to another, so specialist agents can call each other instead of one monolithic agent doing everything.

How is A2A different from MCP? MCP (Model Context Protocol) governs an agent calling a tool — a database, an API, a file store. A2A governs an agent calling another agent. They're complementary: an agent might use MCP to read a database and A2A to ask a billing agent for a customer's status. The risk profile differs too — a tool executes and returns, but a peer agent may itself make more calls, so A2A carries delegation-chain risk that tools don't.

Why is agent-to-agent a security risk? The hop between two agents is usually invisible: no reliable caller identity, no policy on what may be delegated, no scan of the data crossing to the peer, and no record it happened. Add runaway delegation loops, SSRF-prone push webhooks, and over-advertised Agent Cards, and the delegation hop becomes the least-instrumented, highest-blast-radius surface in the stack.

What does it take to govern A2A? A control point on the hop that authenticates the caller, resolves the peer from a scoped registry, evaluates the hop against policy, scans the message for PII, optionally requires human approval, detects delegation loops, and signs every hop into a tamper-evident audit trail — before the message reaches the peer.

Do I need A2A governance if I already govern LLM and MCP calls? If your agents delegate to one another, yes — it's the third of the same three call types (model, tool, agent) and the one most teams haven't instrumented yet. The controls are cheapest to add before the delegation graph gets complex.

Getting started​

A2A is early, which is exactly why it's worth governing now: the controls are cheapest to put in before the delegation graph gets complicated, not after. If your agents are starting to delegate to one another, the fastest path is to read how the A2A governance plane works, or go straight to the A2A Governance Plane documentation for the concrete registry, policy, and audit model.

The delegation hop between two agents is invisible today. It doesn't have to stay that way.

Everything below ships in every Enterprise install — the same governance core across all three planes. 1.3.0 is a drop-in upgrade from 1.2.x.

Govern the hop, not the orchestration​

An agent calling another agent is a delegation: agent A asks agent B to do something on a user's behalf. Left ungoverned, that hop is invisible — no identity on the caller, no policy on what may be delegated, no scan of what data crosses, no record that it happened.

DVARA's A2A plane governs exactly one hop at a time. A request comes in for agent B, the plane runs the full governance chain, forwards it to B, and returns B's reply. It makes no onward call and runs no agent loop — orchestration stays where it belongs, in your framework. DVARA is the checkpoint each hop passes through, not the conductor.

The plane is fully independent of the LLM and MCP planes — its own proxy (dvara-a2a-gateway, port 8075), its own governance engines, its own audit chain. It ships as an opt-in component, off by default, so existing three-plane deployments are untouched until you turn it on.

What a governed hop checks​

Every A2A hop runs through a chain of operator-controlled checks:

• Caller identity — the calling agent presents a DVARA API key, resolved to a workspace. No anonymous hops.
• Registered peers — agent B must be a peer you registered, with its endpoint, auth scheme, and credentials under your control. Callers reference agents by id, never by raw URL.
• A2A policy — an operator-authored policy DSL decides whether A may call B, or a given skill on B. Versioned, with lifecycle and rollback, like the LLM and MCP policy engines — but its own rules, in its own store.
• PII — the outbound message is scanned; PII is redacted in flight or the hop is blocked, per your action.
• Loop detection — runaway delegation chains are detected and the session killed before they spiral.
• Human approval — hops matching a workspace's rules (by skill or by target agent) pause on a durable, cross-pod approval gate; a reviewer approves or denies from the Console or the workspace Portal, and the paused hop releases on whichever pod is holding it.

Governed across the full A2A surface​

1.3.0 governs the whole protocol surface, not just the basic send:

• message:send — the governed synchronous hop.
• message:stream — streaming hops with in-flight PII enforcement over the SSE events.
• tasks:get · list · cancel · resubscribe — the task lifecycle, with PII-scanned task reads.
• pushNotificationConfig CRUD — push-webhook config, with the workspace-supplied callback URL SSRF-validated before it's ever forwarded.
• Agent Card discovery — /.well-known/agent-card.json, served through DVARA with a per-workspace skills allow-list so you advertise only the skills you intend to expose.

A separate plane means a separate record​

The A2A plane writes its own tamper-evident, HMAC hash-chained audit trail — every hop intent and result, every policy denial, PII action, loop kill, and approval decision — plus per-hop metering. Because it uses the same signing primitive as the LLM and MCP chains, that A2A trail folds straight into what you already run: it exports to Splunk, CloudWatch, and Kafka alongside the rest, and it shows up in the SOC 2, HIPAA, GDPR, and India RBI/SEBI compliance reports.

A note on scope. In 1.3.0, DVARA authenticates the caller, enforces the agents and policy you registered, and records every delegation tamper-evidently — including on-behalf-of tokens, which are forwarded to the peer for it to enforce. Cryptographic verification of the authority to delegate — signed, portable delegation with scope attenuation — is the next step on the roadmap, not a claim we make today.

Console review — pagination, navigation, and sessions​

1.3.0 also lands a Console review sweep:

• Server-side pagination across the high-volume views — token usage, audit (both the shared and the A2A chains), MCP tool calls, the cost dashboard, and the agent-session viewer — so large installs page instead of loading everything.
• A reorganized navigation grouped into task-oriented sections, with the audit viewer and the approval queue unified behind a plane selector so LLM, MCP, and A2A share one surface.
• A longer, verified session — the Console session lifetime is now four hours, backed by durable JDBC-stored sessions, ending the frequent re-logins.

Upgrade​

1.3.0 is a drop-in upgrade over 1.2.x. Pull the 1.3.0 images (or the oci://ghcr.io/dvarahq/charts/dvara:1.3.0 chart), keep your existing license and configuration, and roll — the A2A schema migrations apply automatically on first boot.

The A2A plane is opt-in. Turn it on by enabling the dvara-a2a-gateway component (chart) or adding the service (Compose); leave it off and nothing changes. See the A2A Governance Plane docs for the full surface — the governance chain, the agent registry, and the documented limits.

If you are new to DVARA: it is self-managed, activated by a signed license key, with bring-your-own-key credentials so provider keys and request data never leave your perimeter. Start with a 30-day, all-features trial license — every feature unlocked, including native MCP and the A2A plane, no credit card. Get the trial or, for production licensing or a managed deployment in your region, talk to us. The release is live now.

This is a framework, not a leaderboard. And it's deliberately governance-first, because that's the axis where products differ most and demo videos reveal least. Routing and provider coverage are easy to compare and roughly commoditized; whether a platform can enforce a policy, prove what it did, and keep sensitive data in your perimeter is where the real divergence — and the real buying decision — lives. DVARA is one option in the field; the point of this post is to help you score the field, including us, on criteria that matter.

The categories on the market​

Broadly, the options fall into a few shapes, each with a different center of gravity:

• Open-source proxies / libraries (e.g. LiteLLM, Bifrost). Fast to start, free to run, huge provider coverage. You operate and secure them yourself, and governance depth — Policy-as-Code, tamper-evident audit, PII enforcement — is usually thin or DIY. Great when routing and provider abstraction are the whole ask.
• Cloud-vendor gateways (e.g. Cloudflare AI Gateway, Kong's AI Gateway add-on). If you're all-in on one ecosystem, the native integration is tight. The trade-off is gravity toward that vendor and weaker neutrality — and governance that's often coarse.
• Developer/observability-first platforms (e.g. Portkey, Helicone). Strong on tracing, evals, and prompt management, with a gateway attached. (Since this was written, Helicone was acquired by Mintlify in March 2026 and has moved to maintenance mode, with its experiments feature withdrawn — read it as a description of the category rather than a current recommendation.) A good fit if your primary pain is visibility; typically lighter on hard enforcement (blocking budget caps, policy denial, PII blocking) and on self-hosting for regulated data.
• Governance-native platforms. Built around Policy-as-Code, PII/guardrails, tamper-evident audit, agentic (MCP) governance, and self-hosting from the start, with routing and cost as table stakes. The fit when compliance and control are the point. DVARA sits here — and the honest framing is that most of the field started as a gateway and added governance, while a governance-native platform starts from the control requirements and ships the gateway as the mechanism.

No category is "best." They're answers to different questions.

The criteria that matter — governance first​

Score any candidate on these, weighted by what your team is actually optimizing for. Governance rows lead deliberately:

A few deserve emphasis because they're where products quietly differ:

• Enforcement vs. observation. "We show you your spend / your prompts / your traces" is not "we can stop a disallowed call." Hard caps that block and policies that deny are a different capability class than dashboards that inform. → Cost Management
• Where it runs. SaaS is fastest; self-hosting keeps prompts, policy enforcement, and audit in your perimeter. For regulated data this is often the first filter, not the last. → Self-Hosted LLM Gateway
• Model-aware governance. A repurposed API gateway can't see models, tokens, or PII. If governance matters, that gap is disqualifying. → Why an AI Gateway, Not an API Gateway
• Capability-aware failover. Blind failover can retry on a model that can't serve the request. Ask specifically. → Fallback & Failover

A note on MCP and agents​

By 2026, governing agent traffic — not just model calls but the tool calls agents make via the Model Context Protocol — is a real axis of comparison, and most serious gateways now offer some MCP support. The differentiator isn't presence, it's depth: whether MCP is governed (approval gates, loop detection, tool-poisoning defense, per-call audit) or just proxied. If you're building agents, weight this heavily; if you're not yet, treat it as forward-looking rather than deciding.

How to actually choose​

1. Write down your top three constraints. Regulated data? Multi-cloud? Tiny ops team? These pick your category before you compare products.
2. Disqualify on the hard requirements. SaaS-only is a non-starter for some; that filter shrinks the list fast.
3. Trial two candidates on real traffic. Routing and coverage demo easily; enforcement, PII blocking, audit integrity, and self-hosting only reveal themselves under load.
4. Weight enforcement over dashboards if control is why you're here.

Where DVARA fits​

DVARA is an AI gateway that governs model, tool and agent-to-agent calls and deploys anywhere: Policy-as-Code that denies, tamper-evident HMAC-signed audit, PII redaction and guardrails, governed MCP for agent traffic, and data residency by self-hosting — with capability-aware routing and failover, real-time cost attribution and hard budget caps, and $0 PII-safe semantic caching as table stakes. It's the right fit when compliance, control, and sovereignty lead your criteria; if your top constraint is "free and OSS with the widest provider list," an open-source proxy may serve you better. Score it against your own weighted criteria using the pillar guide, What Is an LLM Gateway?, as the reference.

Observability isn't a nice-to-have next to governance; it's the layer underneath it. Policy enforcement, cost attribution, and audit are only as trustworthy as the telemetry that proves they ran. "We block disallowed models" needs a metric that shows the blocks; "we attribute every dollar" needs token counts per workspace; "we can prove what happened" needs traces and logs. In DVARA, the DVARA LLM Gateway emits that evidence on every request, which is what turns governance claims into governance you can audit.

What LLM observability is​

LLM observability is the practice of instrumenting your model traffic so you can measure and diagnose it: request rate, latency distribution, token consumption, cost, cache behavior, and errors — broken down by the dimensions that matter (model, provider, workspace, status). It's ordinary observability applied to a new kind of traffic, plus a few AI-specific signals (tokens, cache hits, provider failovers) that generic app monitoring doesn't capture.

Because a governed control point sits on the path of every call, it's the natural place to emit this telemetry consistently — every request, every provider, one schema.

The three pillars, applied to LLMs​

Metrics​

The counters and histograms you actually watch on a dashboard:

• Request rate and latency — requests/sec and p50/p95/p99 latency, tagged by model, provider, and workspace.
• Token usage — input and output tokens, per model and workspace. This is the raw material for cost governance.
• Errors — provider errors by code, so you see which provider is failing and how.
• Resilience signals — retries, fallbacks (from-provider → to-provider), and rate-limit sheds, which tell you whether your failover is a quiet safety net or a daily tax.

Exposed in a standard format (Prometheus), these drop straight into the monitoring stack you already run.

Traces​

A single request often fans out — filter chain, routing, an upstream call, maybe a failover to a second provider. Distributed tracing stitches those into one timeline so you can see where the milliseconds and the failures actually went, with trace context propagated to the upstream call.

Logs​

One structured log line per request — trace ID, model, provider, status, latency, token counts, workspace, masked API key, cache status, error code — is the record you grep when something specific went wrong. Structured (JSON), not prose, so it's queryable.

The GenAI semantic conventions​

Here's the piece that's new and worth understanding. OpenTelemetry has standardized a set of GenAI semantic conventions — a common vocabulary for model telemetry, with attributes like gen_ai.system (which provider), gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.model.

By default, the DVARA LLM Gateway sets these ten on the span for every provider call (management.tracing.genai-semconv.enabled turns them off):

Which observability platforms support gen_ai span attributes? Any OpenTelemetry backend stores them as ordinary span attributes. GenAI-aware backends go further and read them as model calls, with tokens and models broken out.

Why it matters: if your platform emits these standard attributes on its spans, your traces are portable. GenAI-aware backends — Langfuse, Arize Phoenix, and others — understand them without custom parsing. You're not locked into one vendor's bespoke schema; you're speaking the ecosystem's shared language. The DVARA LLM Gateway emits gen_ai.* alongside its own span attributes, so your governance telemetry travels to whatever backend you standardize on.

What to actually put on the dashboard​

• Traffic: requests/sec by model and workspace.
• Latency: p95 by provider (where failover decisions get validated).
• Tokens: input/output per workspace per day (the cost driver).
• Errors: provider error rate by code.
• Resilience: fallback and retry rates by provider pair.
• Cache: hit rate (ties to cost).

For the request-level view — streaming, structured outputs, and per-call telemetry together — see the deep dive on streaming, structured outputs & observability.

An evaluation checklist​

• Are Prometheus metrics exposed for rate, latency, tokens, and errors — tagged by model/provider/workspace?
• Is there distributed tracing with context propagated to upstream calls?
• Does it emit the OpenTelemetry GenAI conventions (gen_ai.*), so traces are portable?
• Is there one structured log line per request with the fields you'd need to debug?
• Are resilience and cache signals (retries, fallbacks, hit rate) observable, not just request/latency?

Where DVARA fits​

DVARA is an AI gateway that governs model, tool and agent-to-agent calls, and observability is the evidence layer beneath it: its LLM Gateway exposes Prometheus metrics for request rate, latency percentiles, token usage, cost, and errors — tagged by workspace, model, and provider — and emits OpenTelemetry traces carrying the standard GenAI semantic-convention attributes (gen_ai.*) so they're portable, alongside one structured JSON log line per request. That telemetry is what lets you prove policy, cost, and audit ran. Read the pillar guide, What Is an LLM Gateway?, for how observability fits the rest.

Semantic caching keys on meaning instead of bytes. But a cache that stores prompts and responses is also a governance surface: it can quietly hold raw PII, and it can quietly keep charging you for calls that never happened. Both are failure modes a governed gateway has to close by default — which is why, in DVARA, the semantic cache lives inside the same governed path as Policy-as-Code and PII redaction, not off to the side.

What semantic caching is​

Semantic caching stores prompt–response pairs and serves a cached response when a new prompt is semantically similar to a stored one — close in meaning, not identical in text. Instead of hashing the prompt, it embeds the prompt into a vector and compares that vector against cached prompts by cosine similarity. If the closest match is above a similarity threshold, it's a hit and you return the stored answer. Below the threshold, it's a miss and the call goes to the model.

How it works, step by step​

1. Embed the incoming prompt into a vector.
2. Compare it against the vectors of cached prompts (cosine similarity).
3. Threshold — if the best match scores above your similarity threshold (say 0.9), serve the cached response.
4. On a miss, call the model, then store the new prompt vector and response for next time.

The similarity threshold is the dial that matters. Too low and you serve near-misses that aren't really the same question (false hits). Too high and you behave like an exact-match cache and rarely hit. It's tunable per pattern and per workspace — an FAQ bot can be looser than a code assistant.

PII-safe by default​

Here's the governance failure mode a naive cache walks into: it becomes a store of raw sensitive data. If prompts or responses can contain PII, and you cache them as-is, your cache is now an unaudited copy of the very data your policies exist to protect.

In a governed gateway, PII is stripped before anything is written to the cache — the redaction that runs on the request path also runs before the cache write, so the cache never holds raw sensitive data. Caching is a performance feature that must not become a compliance hole; closing that gap by default is the difference between a cache and a governed cache.

Why a cache hit must cost $0​

The other quiet failure: billing. When you serve a response from cache, no upstream API call happened — so that request should cost you nothing. A cache that still books the original call's cost against your budget on every hit erases the savings the cache exists to create.

Done right, a cache hit is recorded (you want the token volume for reporting) but billed at $0, because there was no upstream spend. That distinction — served volume vs. actual upstream cost — is what makes caching a real cost governance lever and not just a latency trick. On a workload with repetitive prompts, it's one of the largest cost reductions available.

The latency win​

Cost is the headline, but hits are also fast. A cached response skips the round trip to the provider entirely — sub-hundred-millisecond instead of multi-second. For high-repetition surfaces (support bots, autocomplete, templated queries), that's a materially better user experience on top of the savings.

Trade-offs to respect​

• Staleness. A cached answer can go out of date. TTLs and per-pattern scoping keep it fresh.
• False hits. Two prompts can be close without being the same. The similarity threshold, tuned per use case, is your control.
• Personalization and correctness. Don't cache responses that are user-specific or must be freshly computed. Scope caching to surfaces where a shared answer is genuinely correct.

An evaluation checklist​

• Does it match on meaning (embeddings + similarity), or only exact text?
• Is the similarity threshold tunable per pattern/workspace?
• Is PII stripped before caching, so the cache isn't a sensitive-data store?
• Does a cache hit book $0 cost, or does it re-charge the original call?
• Are there TTLs and per-pattern scoping to control staleness?

Where DVARA fits​

DVARA does per-workspace caching inside its governed path, with TTLs, PII stripped before anything is cached, and a cache hit that records served token volume but books $0 cost — no upstream call, no charge. Note what is off by default: DVARA uses vectors as a prefilter but serves only on an exact cache-key match unless you set similarity-serving: true. That is a measured decision rather than caution — over a 20-pair corpus the three highest-scoring pairs were all ones that must not be served (two negations and a word-order swap), and precision never exceeded 0.38 at any threshold. Turn it on where a topically-similar answer is acceptable; leave it off where a wrong answer is a correctness problem. It's a cost-and-latency win that stays a governance-safe surface. See how it ties into spend in the LLM cost management deep dive, or the pillar, What Is an LLM Gateway?.

Self-hosting keeps the enforcement point inside your environment. Policy evaluation and PII handling happen before a request reaches a model provider, and a denied request never reaches one.

What "self-hosted" means here​

A self-hosted deployment is one you run inside your own environment — your VPC, your Kubernetes cluster, your on-prem data center — rather than consuming it as a vendor-operated SaaS. Requests from your applications hit your instance. The Gateway still calls out to model providers (unless you're running fully local models), but it does so from inside your perimeter, under your network controls, with your keys, after your governance has run.

Self-hosted and Open Source describe different choices. DVARA Open Source is Apache-2.0 and runs as a file-configured LLM Gateway without a database. The Enterprise platform is also self-hosted, but adds Flightdeck, central persistence, fleet operations, advanced LLM controls, and the MCP and A2A planes. In either case, the deployment and request path stay in your environment.

What self-hosting actually buys you​

Ordered governance-first, because that's the reason to do it:

• Governance runs before the provider sees the prompt. The most sensitive thing you send a model is often the prompt itself. Self-hosting means the request is policy-checked, PII-scanned, and logged before it reaches a model provider — and a blocked request never reaches one at all.
• Audit stays in your control. Open Source can write a local tamper-evident audit file. Enterprise stores fleet evidence in your database for search, export, and reporting.
• Data residency you control. You decide which region the platform runs in and which providers it may reach — how you keep EU traffic on EU infrastructure. (This is the bridge to the broader AI compliance and sovereignty story.)
• Your keys, your vault. Enterprise provider credentials can resolve from your secret store rather than a vendor-operated service.
• Air-gap-friendly. Paired with local models, it can run with no outbound internet at all.

BYOK: bring your own keys​

A serious self-hosted platform is BYOK — bring your own keys. Each workspace's provider credentials are theirs; the Gateway resolves the right key per request and never forces everyone through one shared secret. Credentials can be stored encrypted at rest, or — better — as references into a vault (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault), so the secret material never lands in the platform's own database at all. Rotate a key, and in-flight resolution picks up the new one without a redeploy.

For the strictest setups, strict BYOK refuses to let a workspace borrow a shared platform key at all — no own credential, no call — so you can guarantee cross-workspace key isolation as a governed policy.

What self-hosting doesn't do for free​

Self-hosting is a governance-and-data-path guarantee, not a magic wand:

• You operate it. Open Source can run as one process with no database. Enterprise adds PostgreSQL and the services needed for central operations.
• You still call external providers unless you run local models; self-hosting the platform doesn't self-host OpenAI.
• You still own the operating work. Self-hosting trades a vendor-managed request path for your own infrastructure, upgrades, and monitoring.

For most regulated buyers, those are easy trades against the alternative of not being allowed to use the product at all.

An evaluation checklist​

• Does it deploy into your VPC / cluster cleanly (containers, a Helm chart)?
• Does governance run before the provider sees the request — policy and PII enforced on every call?
• Does the audit trail stay in your database, outside any vendor's scope?
• Is it BYOK, with per-workspace credential isolation (and a strict mode)?
• Can credentials live in your vault as references, not just encrypted in its DB?
• Can you pin it to a region and restrict which providers it may reach?

Where DVARA fits​

DVARA is an AI gateway that governs model, tool and agent-to-agent calls and runs in your environment. Start with the file-configured Open Source LLM Gateway, or use Enterprise when you need Flightdeck, central evidence, fleet operations, per-workspace credentials, and vault-backed secrets. For the wider regulatory picture, read the AI Audit & Compliance guide.

Most tools answer this with a dashboard — visibility after the fact. That's the difference between observation and governance: a dashboard tells you what you spent; a governed control point can stop you from spending it. Real cost governance moves control upstream, to the moment each call is made — attributing it, and enforcing a budget on it, on every call. That enforcement lives in the DVARA LLM Gateway, the part of DVARA that handles model calls.

Why LLM cost is hard to control​

Three properties make model spend uniquely slippery:

1. It's usage-priced and unpredictable. Cost scales with tokens in and out, and a single verbose prompt or a runaway agent loop can cost more than a thousand ordinary calls.
2. It's invisible until it's an invoice. Without instrumentation on the request path, you find out what you spent after you've spent it.
3. It's un-attributed. "Which team, which feature, which customer drove this?" has no answer if every call looks the same.

A governed control point sits on the one path every call takes, which makes it the only place you can price, attribute, and cap spend as it happens.

Real-time cost attribution​

Every request through the Gateway is priced the moment it completes — token counts × the per-model rate — and written as a cost record tagged with the workspace, API key, model, and provider. That turns the unanswerable question into a query: cost by team, by model, by customer, by arbitrary tag, over any window. Attribution is the accountability half of cost governance.

Budget caps: enforcement with teeth​

Attribution tells you what happened; budgets control what's allowed to happen — the enforcement half. A budget cap sets a spend ceiling for a scope, and it has two teeth:

• Soft limit — crossing it fires a warning (an alert, a webhook) but lets traffic continue. Your early warning.
• Hard limit — crossing it blocks further spend with a clear "budget exceeded" response, so a runaway process can't drain the month in an afternoon.

Budgets nest — a global cap, a per-workspace cap, a per-key cap — and the tightest applicable limit wins, so platform-wide guardrails and team-level budgets coexist without stepping on each other. Every block is an audited governance event, not a silent drop.

Model downgrade under pressure​

Blocking isn't the only lever. When a workspace approaches its budget, the Gateway can automatically downgrade the model — routing gpt-4o to gpt-4o-mini, say — so the workload keeps running at a fraction of the cost instead of stopping outright. A softer governed response: degrade the expensive path before you close it.

Per-call ceilings​

Accumulated-spend budgets don't catch a single pathological call — one request with a huge max_tokens that alone costs more than it should. An optional per-call cost ceiling estimates a request's worst-case cost before dispatch and rejects the outliers, complementing (not replacing) the running budget.

Caching and routing are cost levers too​

The cheapest call is the one you do not make. Serving a repeat prompt from cache costs nothing upstream because no provider call happened. In DVARA, exact cache-key matching is the default; matching a merely similar prompt is opt-in. That makes semantic caching a direct cost-control tool. Routing matters too: sending suitable work to a cheaper model is a routing decision with a measurable effect on spend.

Chargeback and forecasting​

For finance, raw cost records aren't enough — they need chargeback: per-workspace reports (PDF or CSV) that turn platform spend into numbers a business unit can be billed for. Add forecasting (project the month from the trailing run rate) and anomaly detection (flag a workspace spending 2× its baseline), and cost management becomes proactive governance instead of a monthly autopsy.

An evaluation checklist​

• Is cost attributed in real time, per workspace/key/model — or only in aggregate, after the fact?
• Are there hard budget caps that block, or only dashboards that inform?
• Does it downgrade under budget pressure, or only stop?
• Can it catch a single runaway call, not just accumulated spend?
• Does it produce chargeback reports and forecasts finance can use?

Where DVARA fits​

DVARA is an AI gateway that governs model, tool and agent-to-agent calls: its LLM Gateway prices every call in real time and attributes it by workspace, key, model, provider, and custom tag; enforces nested soft/hard budget caps (tightest wins); auto-downgrades models under budget pressure; catches runaway single calls with an optional per-call ceiling; and generates chargeback reports, forecasts, and anomaly alerts. Cache hits book $0. Cost isn't a dashboard bolted on afterward — it's enforced on the governed path. Read the pillar guide, What Is an LLM Gateway?, for how cost governance fits the rest.

Everything below ships in every Enterprise install — the same governance core, the same MCP Proxy, the same agentic controls. 1.1.0 is a drop-in upgrade from 1.0.x.

Native MCP — speak the protocol, keep the governance​

In 1.0.0 the MCP Proxy governed tool calls over a REST surface. In 1.1.0 the gateway speaks native MCP (protocol revision 2025-06-18) on both sides, behind the same transport-agnostic governance filter chain.

Northbound, DVARA exposes a native /mcp Streamable-HTTP endpoint. Point Claude Desktop, Cursor, or any MCP-compatible agent straight at your gateway and it gets a per-workspace tools/list — namespaced as {serverId}__{tool}, scanned for tool poisoning — and a tools/call that runs through the full chain: policy evaluation, argument-level tool policies, human approval gates enforced at execution, PII scanning, and signed, chained audit. The client sees a standard MCP server; you get governance and a complete audit trail transparently, on call one.

A note on transports. The MCP spec defines two standard transports — stdio and Streamable HTTP, the latter having superseded the older HTTP+SSE transport. Northbound, DVARA's /mcp endpoint is Streamable HTTP by design — the one network-native transport that carries per-workspace auth and scales across pods; stdio- or SSE-only clients reach it through a local bridge such as mcp-remote. Southbound, DVARA reaches upstream MCP servers through a pooled native client, dispatched per server on its configured transport: Streamable HTTP, stdio, and the legacy HTTP+SSE transport for older servers — while DVARA's existing REST integration remains the default and is unchanged. Native is opt-in per upstream.

The agent governance you already had — loop detection with auto-kill, the unified LLM + MCP session timeline, approval queues — now extends across native MCP traffic without a protocol translation layer in the middle. See Native MCP Support in the docs for the full protocol surface — transports, namespacing, and the documented limits.

Drop-in SSO — Keycloak and Auth0 presets​

Wiring an external identity provider into the DVARA Console used to mean hand-mapping OIDC claims. 1.1.0 ships verified configuration presets for Keycloak and Auth0 — tested against real IdPs, with step-by-step setup guides. Activate a preset with a Spring profile, point it at your realm or workspace, and SSO to the Console works, including Auth0's namespaced claim URIs out of the box. SAML 2.0 and generic OIDC remain supported for everything else.

Turnkey Kubernetes — verified GKE and DOKS​

Running DVARA on managed Kubernetes is now a documented, validated path rather than an exercise. We brought the stack up end-to-end on two clouds and wrote the recipes down:

• GKE — Autopilot or Standard, Cloud SQL over private IP, Workload Identity for secrets, GCE Ingress with managed TLS.
• DigitalOcean Kubernetes (DOKS) — DO Managed PostgreSQL, a plain Kubernetes Secret, Traefik v3 ingress with Let's Encrypt.

The Helm chart now injects the signed license into every component from a single platform Secret, so all three services boot from one source of truth. Per-cloud guides are in the docs, with reference recipes you can adapt to your own cluster.

Usage visibility​

Every token through the LLM gateway and every MCP call is metered per workspace, with a monthly usage report you can export — for cost attribution, chargeback, and capacity planning. Universal on every install, as it has been since 1.0.

Upgrade​

1.1.0 is a drop-in upgrade. Pull the 1.1.0 images (or the oci://ghcr.io/dvarahq/charts/dvara:1.1.0 chart), keep your existing license and configuration, and roll. The Spring Boot 4.1.0 platform bump and a round of base-image and dependency CVE fixes come along for free.

If you are new to DVARA: it is self-managed, activated by a signed license key, with bring-your-own-key credentials so provider keys and request data never leave your perimeter. Start with a 30-day, all-features trial license — every feature unlocked, including native MCP and agentic governance, no credit card. Get the trial or, for production licensing or a managed deployment in your region, talk to us. The release is live now.

Resilience belongs in the same layer as governance, and for the same reason: it can't be trusted if it's scattered and invisible. When failover runs at a single governed control point — the DVARA LLM Gateway, the part of DVARA that handles model calls — every retry and provider switch is policy-checked and written to the audit trail, so "we failed over to a different provider" is a recorded, reviewable event, not a silent mystery. LLM fallback and failover move resilience out of application code and onto the governed path.

Fallback vs. failover​

The terms get used interchangeably; the distinction is useful:

• Failover — the primary provider fails (an error, a 429, a timeout, an open circuit) and the request is retried against an alternative that can serve it.
• Fallback — the ordered list of alternatives to try. Your fallback chain is the plan; failover is executing it.

Both live naturally at a governed control point, because it's the one place that already knows every provider, their health, and their capabilities — and governs the switch.

The circuit breaker​

Retrying a provider that's clearly down just adds latency and load. A circuit breaker tracks failures per provider and, once they cross a threshold, "opens" — the Gateway stops sending traffic there for a cooldown window and fails straight over to an alternative. When the cooldown passes, it lets a probe through; if that succeeds, the circuit closes and normal traffic resumes.

This protects both sides: your users don't wait on a doomed call, and you don't hammer a struggling provider while it's trying to recover.

Capability-aware failover — the part that's easy to get wrong​

Here's the failure mode naive failover walks into. A request asks for response_format: json_schema. The primary — which supports structured outputs — is down. A naive gateway fails over to the next provider in the list… which doesn't support structured outputs. Now the request fails anyway, with a confusing error, and your "resilience" made things worse.

Capability-aware failover filters the fallback candidates by what the request actually needs before retrying. If the request needs structured output, only structured-output-capable providers are considered. If no capable fallback exists, the Gateway says so explicitly — a clear "no capable provider for this request" error — instead of failing over into a wall. This is the same capability filtering that governs normal multi-provider routing.

Upstream rate limits: shed before you fail​

The best failover is the one you never trigger. Providers publish rate-limit headers on every response — how much of your token and request budget remains. A control point that reads those can proactively shed or shift traffic as you approach a limit, rather than waiting for the 429. Under pressure it can honor workspace priority — protecting premium traffic and letting bulk traffic absorb the throttling first. You degrade deliberately instead of hitting a wall, and the shed is attributed and audited like any other governed decision.

Resilience you can prove​

Failover only earns trust if you can observe and account for it. Which providers are failing over, how often, to whom? What's the retry rate, the circuit-open rate, the fallback rate by provider pair? Those are Prometheus metrics and audit events, not guesswork — which is why resilience and observability are two sides of the same coin. A governed gateway doesn't just recover from a provider outage; it can show you exactly what it did.

A checklist for evaluating failover​

• Does it use a circuit breaker, or just blind retries?
• Is failover capability-aware, or will it retry on a model that can't serve the request?
• Does it react to upstream rate-limit headers before the 429, or only after?
• Does it honor workspace priority when a shared credential is near its limit?
• Are failover events observable and audited — metrics and a trail per provider pair, not just a log line?

Where DVARA fits​

DVARA is an AI gateway that governs model, tool and agent-to-agent calls. Its LLM Gateway wraps each provider in a circuit breaker, fails over capability-aware (a request that needs structured output only retries on a provider that supports it, with a clear error if none can), and reads upstream rate-limit headers to shed or shift traffic — honoring workspace priority — before a 429 fires. Every retry, fallback, and shed is a governed, audited, observable event. Read the pillar guide, What Is an LLM Gateway?, for how resilience fits with routing and cost control.

DVARA is an AI gateway that governs model, tool and agent-to-agent calls; the DVARA LLM Gateway is the component that routes model traffic. The application keeps one OpenAI-compatible endpoint, while the Gateway chooses a provider only after governance checks and capability filtering.

Updated September 24, 2026 for DVARA 1.8.0.

What is multi-provider LLM routing?​

Multi-provider LLM routing is the controlled selection of an upstream model provider for each request. A route matches the requested model, removes providers that cannot satisfy the request, then applies a strategy such as round-robin or weighted distribution to the remaining pool.

That definition has three important boundaries:

• Routing is not model rewriting. The selected provider receives the model identifier from the request. Every provider in a shared route must accept that identifier.
• Routing is not failover. Routing selects the primary provider before dispatch; failover is a recovery step after an eligible provider failure.
• Routing is not governance by itself. Provider selection still needs policy enforcement, PII handling, rate limits, and an evidence trail around it.

Centralizing the decision gives platform teams one place to review routing behavior without moving the decision into every application repository.

How does an LLM routing decision work?​

A governed request follows five distinct stages:

1. Evaluate governance. Resolve the API key and workspace, then apply policy, guardrails, PII handling, and configured limits.
2. Match a route. Compare the request's model with configured route patterns. If no route matches, use the built-in model-prefix mapping.
3. Filter by capability. Remove providers that cannot satisfy requirements such as structured output, JSON mode, or tool calling.
4. Apply the routing strategy. Select one provider from the eligible pool.
5. Recover separately. If the upstream call fails and resilience permits recovery, try a compatible registered fallback.

This order prevents a routing decision from bypassing governance. It also prevents failover from sending a structured-output or tool-calling request to a provider that cannot honor it. If no compatible provider exists, the Gateway returns an explicit capability-mismatch error instead of silently degrading the request.

Which routing strategy should you use?​

Choose the simplest strategy that expresses the operational decision you need to make.

DVARA Open Source 1.8.0 does not provide latency-aware, cost-aware, geo-aware, or intelligent routing. Configuring one of those strategies in the Open Source build is refused at startup rather than silently falling back to another strategy.

Configure weighted routing without changing application code​

This gateway.yaml registers OpenAI and Azure OpenAI, then sends roughly 80% of matching traffic to OpenAI and 20% to Azure OpenAI:

providers:
  - type: openai
    api_key: ${OPENAI_API_KEY}
  - type: azure-openai
    api_key: ${AZURE_OPENAI_API_KEY}
    base_url: ${AZURE_OPENAI_BASE_URL}

routes:
  - id: weighted-gpt
    model: "gpt*"
    strategy: weighted
    providers:
      - provider: openai
        weight: 80
      - provider: azure-openai
        weight: 20

Both providers receive the same requested model name. The Azure deployment therefore needs to use the model name the application sends; routing does not translate gpt-4o-mini into a different deployment identifier.

After restarting DVARA Open Source with the updated file, the application continues to call the same endpoint:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $DVARA_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Summarize this incident in two bullets."}]
  }'

The response stays OpenAI-compatible. To verify the routing effect, use the response trace ID to inspect the structured access log; its provider field records which upstream handled the request. Over a meaningful request sample, the observed distribution should approach the configured weights. A handful of calls is not enough to validate a probabilistic split.

Open Source reads routes from gateway.yaml at startup, so changing a route requires a restart. DVARA Enterprise stores routes centrally and distributes updates through Flightdeck without requiring an application redeploy.

Keep capability filtering separate from provider preference​

A provider can be healthy and still be wrong for a particular request. Structured outputs, JSON mode, tool calls, streamed tool calls, vision, embeddings, and batch operations differ across providers.

Capability filtering narrows the pool before the strategy runs. This means an 80/20 weighted route is really an 80/20 split across the providers still eligible for that request. If one configured member is unavailable or lacks a required capability, it does not receive traffic merely to preserve the nominal percentage.

This distinction matters when interpreting routing metrics. A changed distribution can reflect health or capability filtering, not a broken weight calculation.

Do not confuse routing with failover​

Routing and failover answer different questions:

Retries, circuit breakers, timeouts, and fallback are enabled by default in DVARA Open Source, with configuration available per provider. They should still be tested with realistic failures. A fallback is only safe when the alternative provider supports the request and accepts the same model identifier.

For the full recovery path, read LLM fallback and failover.

Put governance before optimization​

Provider optimization is useful only if it preserves the controls applied to the request. A centrally routed call should still answer:

• Which workspace and API key initiated it?
• Which policy decision applied before dispatch?
• Was PII logged, blocked, or redacted under the configured action?
• Which provider handled the call, and did a retry or fallback occur?
• What usage and cost were attributed to the request?

DVARA Open Source puts routing, file-configured policy, PII controls, guardrails, rate limits, and an optional local tamper-evident audit file on the same LLM request path. DVARA Enterprise adds Flightdeck, central persistence and fleet operations, advanced Enterprise controls, and the MCP and A2A Gateways.

Common multi-provider routing mistakes​

Treating provider names as model mappings. A route selects a provider; it does not translate model identifiers between vendors.

Calling weights cost optimization. A static 80/20 split does not inspect price. Cost-aware selection is a separate Enterprise strategy.

Assuming configuration is live in Open Source. gateway.yaml is read at startup. Restart the Gateway after changing routes or policies.

Ignoring capabilities during failover tests. Test structured output, tool calling, streaming, and vision separately. A successful text fallback does not prove every request shape can fail over.

Letting applications bypass the governed endpoint. Direct provider calls create a second path without the same routing policy or evidence. Restrict provider credentials and network paths so production calls traverse the intended control point.

Start with one route you can verify​

Begin with one model family and one explicit operational goal: an even split, a controlled canary, or a static weight. Record the baseline provider distribution, test an unavailable provider, and verify that policy and audit behavior remain intact.

Use the Open Source quickstart to run DVARA locally, then follow the routing and load-balancing reference for every strategy, field, and distribution boundary. For how routing fits the wider governance path, see the DVARA LLM Gateway.

Guardrails as code applies the same discipline that policy as code brought to access rules — declarative, version-controlled, centrally enforced — to the content of AI traffic: the prompts going in and the responses coming out.

Guardrails vs. policy — two halves of AI governance​

It's worth being precise, because the terms get blurred. In a governed gateway they're complementary layers:

• Policy governs access: which model or tool a caller may use, under which conditions (region, budget, time). That's the subject of Policy-as-Code for LLM and agent traffic and the LLM Policy as Code guide.
• Guardrails govern content: is this prompt a jailbreak attempt? does it contain PII? is the response toxic, off-topic, or malformed? Policy decides if the call is allowed; guardrails inspect what's actually in it.

Both run at the same control point, on every request. Both should be code, not tribal knowledge. (If you're new to the "as code" idea, start with What Is Policy as Code? — the principles carry over directly.)

What "as code" means for a guardrail​

A guardrail as code has the same four properties that define policy as code:

• Declarative — you configure what to detect and what to do about it (block, flag, redact, log), not a bespoke scanner you maintain per service.
• Version-controlled — thresholds and rules have a history you can diff and roll back.
• Centrally enforced — one layer scans every LLM and agent call, so a new service can't quietly skip the PII check.
• Tested and audited — you can tune a threshold and see what it would catch, and every detection is recorded on the call it fired for.

The last point matters most for guardrails: a blocked prompt-injection attempt or a redacted SSN is exactly the kind of event a security team needs a durable record of.

What belongs in the guardrail layer​

A practical guardrail layer covers a handful of categories, mapped closely to the OWASP Top 10 for LLM Applications:

• Prompt injection & jailbreak detection (LLM01) — pattern- and classifier-based detection of attempts to override instructions or exfiltrate the system prompt, on the way in.
• PII / sensitive-data detection — layered detection: deterministic, checksum-validated patterns for structured data (cards, SSNs, IBANs) plus NER for unstructured identifiers (names, locations). Each hit can log, block, or redact permanently. Commercial tokenization is the separate request-side action for values that must be recoverable later.
• Content safety — toxicity, violence, sexual content, plus your own denylists: banned topics, competitor mentions, whatever your domain requires.
• Output validation — when you ask for structured output, enforce it: a response that does not match the declared JSON schema is refused with HTTP 422. DVARA does not retry schema-validation failures automatically.
• Resource limits (LLM10) — caps on input tokens, message count, and message length so a single request can't blow up cost or context.
• Grounding / hallucination checks (optional) — compare response claims against provided source documents and flag ungrounded statements.

The point isn't that every workload needs all of them — it's that they live in one configurable layer you can reason about, not scattered across prompts and middleware.

Inputs and outputs, including streams​

A prompt-only guardrail only sees half the traffic. A real guardrail layer scans both directions — the request before it reaches the model, and the response before it reaches the user — because a model can emit PII or policy-violating content the input never contained. That includes streaming responses. Observational actions can release events as they arrive. A blocking or redacting action holds the complete response, evaluates it once, and then releases or refuses it so unsafe text cannot escape between chunks.

Actions and per-workspace tuning​

Detection is only useful if you can decide what happens next. Each category resolves to an action — block the call, flag it (record and continue), redact the offending span, or log — and a risk-score threshold below which weak detections are ignored. In a multi-tenant setup, those settings tune per workspace: a regulated customer might block on any PII while an internal sandbox only logs, without either one touching application code.

Why this is the same move as policy as code​

Guardrails hard-coded into prompts and per-service middleware recreate exactly the problem access rules had before policy as code: unreviewable, untestable, unauditable, and inconsistent across code paths. Lifting them into a declarative layer at the gateway — the one point every model and agent call passes through — makes AI safety a configurable, versioned, provable property of the platform instead of a habit each team hopes it remembered.

That's the guardrail half of Policy-as-Code for LLM and agent traffic. It applies to model calls today; the same idea extends to agent tool calls — the subject of DVARA's MCP governance, where a guardrail can inspect a tool's arguments and an approval gate can pause a risky action for a human.

Part of the Policy-as-Code cluster: start with the pillar guide, then What Is Policy as Code? and LLM Policy as Code.

Policy as code is the practice of writing those rules in a declarative, version-controlled language that a system enforces automatically — so a policy becomes a reviewable, testable, auditable artifact instead of a paragraph in a wiki nobody reads.

Policy as code, defined​

Policy as code (PaC) is the practice of defining governance and compliance rules in machine-readable, version-controlled files, and having software evaluate and enforce them automatically at the point a decision is made — rather than documenting the rules in prose and relying on humans to remember and apply them.

Three things fall out of that definition, and together they're what make it "policy as code":

• It's declarative. You describe what is allowed or denied, not how to check it. deny any model not on this allowlist — not a script you maintain.
• It's version-controlled. The policy is a file with a history. You can diff it, review it in a pull request, roll it back, and answer "who changed this rule, when?"
• It's enforced by a system, automatically. The rule isn't advisory. A control point evaluates every relevant request against it and acts — allow, deny, warn — without a human in the loop.

Take away any one of these and it stops being policy as code. A YAML file no system reads is just documentation. A script that enforces rules but lives untracked in one service is just more app code.

Where it came from​

Policy as code isn't new — AI is just the newest place it's landing. The pattern has a decade of lineage:

• Infrastructure as code (Terraform, CloudFormation) made server and network configuration a reviewable artifact instead of a runbook of manual clicks.
• Open Policy Agent (OPA) and its Rego language generalized the idea to authorization — a single engine that answers "is this action allowed?" across services.
• Kubernetes admission control and CEL (Common Expression Language) brought policy evaluation into the request path itself: every API call is checked against policy before it's admitted. Envoy, Cilium, and gRPC use the same CEL engine for the same reason — it's sandboxed, deterministic, and type-checked.

Each step moved rules out of scattered, imperative code and into a declarative layer a system enforces. AI governance is the same move, applied to a new kind of traffic.

Policy as code vs. a policy document​

The contrast is the whole point. A policy document and policy as code can say the same words — "only approved models, no PII in prompts, log every call" — but only one of them can act on them:

A document is a statement of intent. Policy as code is intent that runs.

The lifecycle that makes it safe​

Writing rules as code is only half the value; the other half is being able to change them without fear. Mature policy as code moves a change through stages, the same way you'd ship any production change:

1. Author — write the rule; it starts as a draft, affecting nothing.
2. Test — dry-run it against real inputs and see the exact decision before it touches live traffic.
3. Shadow — run the candidate in production in parallel, recording what it would have done, while changing nothing.
4. Promote — activate it, with conflict detection to catch rules that contradict each other.
5. Roll back — every change is versioned; revert instantly if it misbehaves.

Without this, "policy as code" is just a config file you're afraid to touch. With it, governance changes become routine and low-risk.

Policy as code for LLM and agent traffic​

For AI, the rules are about model and tool calls rather than servers or Kubernetes objects. Typical LLM policies answer questions like these:

A rule is a few lines of YAML. This one blocks an unapproved model and tells the caller what to use instead:

rules:
  - id: deny-legacy-model
    priority: 10
    conditions:
      model:
        denylist: [mock/legacy]
    action: DENY
    deny_message: "This model is not approved. Use mock/approved instead."

The gateway checks every request against the rule before it reaches the provider, and records the decision in the audit log. Content checks such as PII detection or prompt-injection screening are guardrails, not access policy: they inspect the text rather than the request's facts, so they are configured separately.

LLM Policy as Code walks through writing, testing and rolling out rules like this one.

Why AI teams need it now​

AI workloads broke the old assumption that governance can live in application code. Model calls and agent (MCP) tool calls carry decisions — which model, how many tokens, which tools, which region, whose budget — that a general-purpose API gateway or IAM system can't even see, let alone govern. Hard-coding those rules across every service that calls a model recreates exactly the ungovernable mess that infrastructure as code was invented to solve.

The answer is the same: express the rules once, in a declarative language, and enforce them at one control point every LLM and agent request already passes through. That's what Policy-as-Code for LLM and agent traffic is — and if you want the concrete rule model (YAML conditions, CEL expressions, the allow/deny/warn actions), the companion guide LLM Policy as Code walks through it end to end.

Start with the Policy-as-Code pillar guide for the full picture, then go deep on the rule model in LLM Policy as Code.

DVARA is an AI gateway that governs model, tool and agent-to-agent calls; the DVARA LLM Gateway is the component that enforces LLM policy on the request path. The policy is useful because enforcement and evidence travel together—not merely because the rules are written in YAML.

Updated September 24, 2026 for DVARA 1.8.0.

What is LLM policy as code?​

LLM policy as code is a way to define AI-traffic violations as machine-readable rules, keep those rules under change control, and evaluate them consistently before model dispatch. A matching rule can deny the call or warn the caller while allowing it to continue.

The implementation has three parts:

1. A policy artifact that names the rule, its priority, matching conditions, action, and an operator-written message.
2. A central enforcement point that sees AI-specific request attributes such as the model, requested output tokens, and offered tools.
3. An evidence record that identifies the policy and rule involved in each decision.

Without central enforcement, a YAML file is only documentation. Without a reviewable artifact, central enforcement becomes opaque application logic. Without evidence, neither one proves what happened on a particular request.

What should policy govern?​

Use policy for deterministic access and usage decisions that can be evaluated from request context.

Keep probabilistic content inspection separate. Prompt-injection detection, content filtering, and PII detection are guardrail or data-protection controls; they should not be disguised as deterministic access-policy rules.

Write one policy that fails clearly​

This DVARA Open Source policy denies one unapproved model. Put it in gateway.yaml alongside the workspace, route, and API-key fingerprint from the Open Source quickstart:

policies:
  - id: approved-models-only
    workspace: default
    status: ACTIVE
    dsl: |
      version: "1"
      rules:
        - id: deny-legacy-model
          priority: 10
          conditions:
            model:
              denylist: [mock/legacy]
          action: DENY
          deny_message: "This model is not approved. Use mock/approved instead."

Restart DVARA Open Source after changing gateway.yaml; the file is read at startup. Then send a request for the denied model:

curl -s -w '\nHTTP %{http_code}\n' \
  http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $DVARA_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mock/legacy",
    "messages": [{"role": "user", "content": "Draft a customer reply."}]
  }'

The Gateway stops the request before provider dispatch and returns the safe alternative written into the rule:

{
  "error": {
    "message": "This model is not approved. Use mock/approved instead.",
    "type": "policy_violation",
    "code": "policy_denied",
    "trace_id": "01K4V5R9X21MDN8W5A6E7ZQ3TC"
  }
}
HTTP 403

When the local audit file is configured, the same decision is recorded as POLICY_DENIED with the policy ID, rule ID, workspace, reason, and chain-integrity fields. The trace ID changes on every request; use the value from the actual response when investigating a call.

Read conditions as violations​

DVARA policy conditions describe when a rule should fire, not the desired steady state. For example, a model allowlist matches when the requested model is outside the list; the DENY action then blocks the violation.

Four rules prevent most policy mistakes:

• Multiple conditions inside one rule use AND. Put independent denial reasons in separate rules.
• Model names are exact and case-sensitive, not prefixes or globs. List every approved dated model name explicitly.
• A max_tokens rule evaluates an explicit request value. If the request omits max_tokens, that condition does not match or inject a default.
• Empty conditions are refused rather than accepted as policies that appear active but check nothing.

Write denial messages for the caller, not the policy author. “Use mock/approved instead” is more useful than “rule 10 failed.”

Choose between deny and warn​

DENY stops the matching request before provider dispatch. Use it for a control whose violation must not reach the model provider.

WARN_AGENT allows the call to continue and attaches a governance warning to the result and audit trail. Use it when the calling application is expected to react but blocking would be premature, such as a preview-model migration period.

A warning is not a softer spelling of enforcement. If the caller ignores warnings, the control is observational. Define who consumes the warning and what that system does before relying on it.

Separate Open Source enforcement from Enterprise lifecycle​

The YAML language is shared, but the available conditions and operating workflow differ.

DVARA Open Source 1.8.0 evaluates ACTIVE policies from gateway.yaml. It does not evaluate Shadow policies or provide CEL, data-residency, time-of-day, budget-utilization, or MCP conditions. Unsupported conditions are refused rather than silently ignored.

DVARA Enterprise adds the managed lifecycle: author a Draft, dry-run the fields the test surface can represent, observe representative traffic in Shadow, promote to Active, and roll back by restoring an earlier version as a new revision. Enterprise production rights and support require a licence.

Updated for 1.8.2. This post originally said the licence activates the MCP and A2A planes. In 1.8.2 every feature, the MCP and A2A planes included, runs without a licence for non-production use, within 3 workspaces and 100,000 governed calls a month. A licence removes both limits and gives production rights and support. An expired licence refuses no traffic.

Roll out policy without creating an outage​

A reliable rollout is operational, not merely syntactic:

1. Name the violation. Write one rule for one independently actionable reason.
2. Use representative requests. Include allowed, denied, missing-field, tool-calling, and streaming cases relevant to the rule.
3. Check the error contract. The caller needs a stable status, code, trace ID, and useful remediation message.
4. Verify evidence. Confirm that the decision names the expected policy, rule, workspace, and reason.
5. Bound the rollout. In Enterprise, use Draft, dry-run, and Shadow before Active. In Open Source, test the file against a non-production Gateway before restarting production.
6. Plan reversal. Keep the last known-good file or Enterprise version and rehearse the rollback path.

Dry-run and Shadow are not equivalent. Dry-run evaluates a synthetic input and only the fields the test surface supplies. Shadow observes real request context without changing the live decision. Conditions involving tools, time, region, budgets, or MCP context need representative governed traffic rather than assumptions from a simple model-only dry-run.

Policy as code is not application authorization​

IAM still determines who can authenticate to applications and administrative surfaces. Application authorization still decides which business operation a user may perform. LLM policy governs the AI request at the shared enforcement point.

These layers should reinforce one another, not substitute for one another. A tool policy can prevent an application from offering cancel_order to a model; it does not decide whether the signed-in human is authorized to cancel order ORD-1048 inside the order service.

Common LLM policy-as-code mistakes​

Writing broad rules with unclear messages. Separate model, token, and tool violations so each response tells the caller what to fix.

Assuming an allowlist matches allowed values. It matches the violation—values outside the list—so the associated action can deny them.

Describing Enterprise lifecycle as an Open Source feature. Open Source reads active file-based policy at startup. Draft, Shadow, dry-run, version history, rollback, CEL, and advanced conditions belong to Enterprise.

Forgetting the bypass path. A centrally enforced rule governs only requests that traverse the Gateway. Restrict direct provider credentials and network paths accordingly.

Treating an audit record as proof that storage cannot change. The Open Source audit file is tamper-evident through HMAC signing and hash chaining. Someone with direct filesystem access can still delete data; the mechanism makes changes detectable rather than impossible.

Start with one model rule and one observable result​

The fastest useful policy is small: deny one model, send one allowed request and one denied request, then verify the error and audit record. Expand only after the team can explain the behavior and reversal path.

Use the Open Source quickstart for a local governed request, the Policy-as-Code guide for the complete rollout, and the policy condition reference for exact matching behavior. For product-level evaluation, see Policy-as-Code for LLM traffic.