Your AI Spend Has No Owner. Cortex AI Gateway Gives It One.

A hands-on production deployment of Snowflake Cortex AI Gateway, covering access governance, live inference, end-to-end tracing, and cost attribution — including the lessons the documentation doesn’t tell you.

Most organizations adopted LLMs the way they once adopted SaaS: one team at a time. One group has an OpenAI key in a CI secret. Another runs Claude Code on personal tokens. A third calls Cortex from a notebook. Finance gets one bill and can’t say which team drove it. Security can’t say which prompts left the building. Platform engineering gets told to “govern AI” with no control point to govern from.

The friction isn’t the models. It’s that there’s no single path traffic has to take, so identity, logging, and limits have nowhere to attach.

On Nov 3, 2026, Snowflake made Cortex AI Gateway generally available: one governed endpoint per account, speaking the OpenAI and Anthropic formats, with every request traced and every credit attributed to a user. I deployed it in a live account and verified every claim here against what came back. The gateway isn’t a new way to call models — it turns AI governance from a policy document into enforced architecture, if you close three gaps the happy path hides.

TL;DR

• Cortex AI Gateway ships open (USAGE to PUBLIC) — revoke it first.
• Manage the gateway spec as code; ALTER ... FROM SPECIFICATION replaces it wholesale.
• PATs need a network policy, or you get a silent 401 with no trace.
• AI_GATEWAY_USAGE_HISTORY is model-keyed OBJECTs — flatten it for showback.
• Budgets notify; quotas block. Wire a stored proc for team cutoffs.

What you’ll learn: how to lock down the gateway, apply a spec safely, onboard SDK clients (with joined client + gateway traces), query traces and billing, enforce spend, and alert on anomalies — all from a live deployment. Full code: cortex-ai-gateway-starter on GitHub.

1. What changed

AI Gateway, Nov 2026: three capabilities GA, two in preview.

Three properties shape the architecture:

• Nothing to provision. One gateway per account, named SNOWFLAKE. Get the endpoint from DESCRIBE AI GATEWAY — private-connectivity accounts use a different host.
• Open by default. USAGE goes to PUBLIC. Change that first.
• Path, not entitlement. Gateway USAGE doesn't grant model access; requests still need SNOWFLAKE.CORTEX_USER. That's what makes one chokepoint safe.

2. Before and after

Before: many unmanaged paths

Before — fragmented access, no attribution

After: one governed path

After — one boundary, full attribution, full governance

Removed: provider keys in secret stores, logging-only LLM proxies, chargeback spreadsheets. Added: a spec managed as code, a sensitive trace table, and one dependency for all AI traffic. (Not to be confused with the Snowflake ML GATEWAY object for traffic splitting.)

3. Production implementation

Everything below ships in the companion repo (RBAC, specs, SQL, client configs, CI, runbooks) and was executed in a live account.

Step 1: Lock down access (default-deny)

USE ROLE ACCOUNTADMIN;

REVOKE USAGE ON AI GATEWAY SNOWFLAKE FROM ROLE PUBLIC;  -- it ships open
CREATE ROLE IF NOT EXISTS AI_GW_USER;     -- may send inference
CREATE ROLE IF NOT EXISTS AI_GW_MONITOR;  -- reads all usage + traces
GRANT USAGE   ON AI GATEWAY SNOWFLAKE TO ROLE AI_GW_USER;
GRANT MONITOR ON AI GATEWAY SNOWFLAKE TO ROLE AI_GW_MONITOR;
-- Model entitlement is separate from the gateway path
GRANT DATABASE ROLE SNOWFLAKE.CORTEX_USER TO ROLE AI_GW_USER;

Verified via SHOW GRANTS: only OWNERSHIP, USAGE→AI_GW_USER, and MONITOR→AI_GW_MONITOR remain. FinOps reads everyone's spend and traces without being able to generate any; a USAGE-only role sees only its own spans.

Step 2: Payload-exclusion tag

CREATE TAG GOVERNANCE.TAGS.PAYLOAD_CAPTURE_EXEMPT;
ALTER USER EXEC_USER SET TAG GOVERNANCE.TAGS.PAYLOAD_CAPTURE_EXEMPT = 'true';

Only tags set directly on users count, the value is irrelevant, and exclusion fails closed. Anyone who can apply this tag can disable prompt auditing — keep APPLY with security.

Step 3: Manage the specification as code

ALTER AI GATEWAY ... FROM SPECIFICATION replaces the whole spec — an omitted field reverts to its default. So: spec in Git, apply in full, verify with DESCRIBE. I confirmed the round-trip live — what you apply is exactly what DESCRIBE returns:

ALTER AI GATEWAY SNOWFLAKE FROM SPECIFICATION $$
schema_version: 1
models:
  - name: claude-sonnet-4-6
  - name: claude-haiku-4-5
  - name: claude-opus-5
logging:
  enabled: true
  enable_client_telemetry: true
  capture_payload:
    request_response: true
    exclude_tags:
      - GOVERNANCE.TAGS.PAYLOAD_CAPTURE_EXEMPT
$$;

Three deliberate decisions:

• Allow-list, never '*', in production. '*' exposes every model the account can reach — including ones added tomorrow — with no review.
• capture_payload.request_response: true is a data-classification decision. It stores full prompts and responses in an event table in your account. Classify it, restrict it, set retention — before enabling.
• enable_client_telemetry: true lets clients export OTLP spans to <gateway-endpoint>/telemetry/v1/traces, joining client-side agent steps to gateway spans under one trace_id (via W3C traceparent).

Step 4: Point clients at the gateway — and the PAT gotcha the docs mention once

The endpoint is https://<account-host>/api/v2/aigateways/snowflake plus a path per API:

| API                  | Path                    | Models            | SDKs              |
|----------------------|-------------------------|-------------------|-------------------|
| Chat Completions     | `/v1/chat/completions`  | All except Claude | OpenAI Python/JS  |
| Messages             | `/v1/messages`          | Claude only       | Anthropic Python  |
| Responses (preview)  | `/v1/responses`         | OpenAI only       | OpenAI Python/JS  |

Auth is a Snowflake bearer token (OAuth, key-pair JWT, or PAT). Live finding: a PAT works only for a user subject to a network policy. Without one you get a bare HTTP/2 401 with an empty body and no trace. The fix:

CREATE NETWORK POLICY IF NOT EXISTS AI_GW_PAT_NP ALLOWED_IP_LIST = ('<client-cidr>');
ALTER USER <user> SET NETWORK_POLICY = AI_GW_PAT_NP;

And one more syntax catch: PAT creation takes DAYS_TO_EXPIRY, not EXPIRES_IN_DAYS — the latter is rejected.

ALTER USER svc_agent ADD PROGRAMMATIC ACCESS TOKEN ai_gw DAYS_TO_EXPIRY = 1;

Once that was in place, the first real call came back clean:

curl -sS -X POST "$GW/v1/messages" \
  -H "Authorization: Bearer $SNOWFLAKE_TOKEN" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model":"claude-haiku-4-5","max_tokens":32,"messages":[{"role":"user","content":"Reply with exactly: pong 1"}]}'

Response: "pong 1", 15 input tokens, 7 output tokens — plus an x-snowflake-request-id header. Keep that header; it's the join key to your billing record (Step 6).

Step 5: Observability that answers real questions

AGENT_TRACE_TABLE is a table function over a standard event table. The live row for that test call:

| Span                    | Status           | User   | Model             | In/Out Tokens | HTTP | Duration |
|-------------------------|------------------|--------|-------------------|---------------|------|----------|
| `chat claude-haiku-4-5` | `STATUS_CODE_OK` | SATISH | `claude-haiku-4-5`| 15 / 7        | —    | —        |

Token counts match the API response exactly. The production reliability query, verified against the live table:

SELECT
  record_attributes:"gen_ai.request.model"::string        AS model,
  COUNT(*)                                                AS requests,
  COUNT_IF(record:"status":"code"::string = 'STATUS_CODE_ERROR'
           OR TRY_TO_NUMBER(record_attributes:"http.status_code"::string) >= 400)
        / NULLIF(COUNT(*), 0)                             AS failure_rate,
  APPROX_PERCENTILE(
    DATEDIFF('millisecond', start_timestamp, timestamp), 0.95) AS p95_ms
FROM TABLE(AGENT_TRACE_TABLE('SNOWFLAKE'))
WHERE timestamp >= DATEADD(hour, -24, SYSDATE())
  AND record_attributes:"gen_ai.operation.name"::string = 'chat'
GROUP BY model
ORDER BY failure_rate DESC;

Two accounting traps and one UI trap:

• Cache tokens are subsets. input_tokens already includes cache reads/writes — adding them double-counts spend.
• Payload attributes are opt-in and never populate for exclude-tagged users.
• Empty Snowsight pages ≠ broken gateway. Before traffic, Overview and Cost show “No usage”. Analytics charts also exclude client OTLP spans and are approximate — alert on SQL, explore in the UI.

For alerting I consolidated to two native alert+procedure pairs (no third-party APM, no scheduled-agent fire, no read-only tradeoff):

• Hourly AI_GW_WATCHER — spec drift, failure-rate spikes, 429 bursts. Logs every run; emails only on anomaly, naming the failing models inline.
• Daily AI_GW_DAILY_DIGEST — HTML dashboard email: health pill, KPI cards, 7-day trend chart, per-model latency, top errors, showback. Both headers carry account, region, user, role, and warehouse.

Step 6: Cost management — and the real usage-view schema

Snowsight → AI Gateway → Cost answers the day-to-day questions, filterable by date range and user:

• Summary cards: gateway spend, median (p50) user spend, model spend, and tool spend (shows “Not available” until MCP tools are in use).
• Charts: spend trend, spend by model, spend by MCP server, and per-user p50/p90/p99 with a distribution histogram.
• Tables: model breakdown (credits, credits/request), top 20 users, and a detailed breakdown grouped by model with a period-over-period Δ.

The Snowsight Cost page packages the day-to-day questions (spend cards, trends per model and MCP server, p50/p90/p99 user distribution, top-20 spenders, period deltas). For anything feeding chargeback or audit, the system of record is AI_GATEWAY_USAGE_HISTORY It actually contain:

• OPERATION_DETAILS and CREDITS_GRANULAR are keyed by model name({"claude-haiku-4-5": {"input_tokens": 15, ...}}) — use LATERAL FLATTEN.
• USER_ID joins to ACCOUNT_USAGE.USERS; METADATA carries role_id.
• REQUEST_ID equals the x-snowflake-request-id response header — verified client-to-billing correlation (CREDITS = 0.000025).

One request ID bridges your client logs, trace spans, and billing rows.

The working showback pattern (validated on live rows):

SELECT
  DATE_TRUNC('day', h.START_TIME)  AS usage_day,
  u.NAME                           AS user_name,
  f.KEY                            AS model,          -- model name is the object key
  f.VALUE:"input_tokens"::number   AS input_tokens,
  f.VALUE:"output_tokens"::number  AS output_tokens,
  h.REQUEST_ID,
  h.CREDITS
FROM SNOWFLAKE.ACCOUNT_USAGE.AI_GATEWAY_USAGE_HISTORY h
LEFT JOIN SNOWFLAKE.ACCOUNT_USAGE.USERS u ON u.USER_ID = h.USER_ID
, LATERAL FLATTEN(input => h.OPERATION_DETAILS) f
WHERE h.START_TIME >= DATEADD(day, -30, CURRENT_TIMESTAMP());

One subtlety: CREDITS is per-request, so if a request fans out across models (agent turns), it repeats per flattened row — dedupe by REQUEST_ID when summing credits; sum tokens per flattened row.

Get Satish Kumar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

For enforcement, choose by what actually stops spend:

Three API paths, three SDKs, three model families.

The budget → stored proc → REVOKE cutoff works. I drilled it end to end: the revoke applied, a non-allow-listed role was refused, and the re-grant was logged.

• Validate the role against an allow-list before any dynamic SQL (IDENTIFIER() inside EXECUTE IMMEDIATE doesn't bind).
• Quotas are the backstop. Budgets can overshoot for hours; quotas block within minutes.
• Edition caveat: CREATE BUDGET was unsupported in my account. Budgets and quotas live in Snowsight → AI Gateway → Cost.

Step 7: MCP Tools governance (plan, don’t bet on it yet)

Tools connects agents to GitHub, Slack, Salesforce, Google Workspace and 100+ other systems over MCP, through the same governed path. In Snowsight → MCP & Tools, every server starts disabled — but enabling one without configuring tools exposes all of them.

• Allowlist specific tools, read-only where possible. “All current and future tools” grows your attack surface with every provider update.
• Per-user OAuth: the agent can do only what the authorizing user can.
• Discovery ≠ authorization: clients see servers only with the SNOWFLAKE$OPS.MCP$USER database role.
• AWS commercial regions only, administered by ACCOUNTADMIN.

Live Demo:

https://youtu.be/E5z-qyXP640?si=x3FhDp1ciLlkVpjh

4. Production reality

Beyond the gotchas above, my own deploy skipped a CREATE TABLE — drills catch what reviews don't.

What I learned by deliberately breaking it (a four-case failure drill):

1. Denied ≠ traced. A request outside the allow-list returns a bare 401 with no span and no billing row — denial happens at the edge. Trace-based failure alerts only see served failures.
2. max_tokens: 0 succeeds. 200 with stop_reason: max_tokens and zero output — client bugs that zero out output look healthy in reliability metrics.
3. Malformed requests DO trace. A 400 landed as STATUS_CODE_ERROR with the parse error in record.status.message and, at volume, tripped the watcher's anomaly email.
4. Failed spans record NULL tokens, and rejections are free. Coalesce the NULLs or they drop out of token sums; the billing view showed only the 2 served requests against 4 trace spans.

Post-drill: p50 323 ms / p95 449 ms for claude-haiku-4-5 through the gateway — the proxy hop is measurable noise at this scale.

What can fail at scale: single-path dependency (add retry/backoff), spec drift (Git + PR + scheduled check), and 429s when a new agent fleet arrives.

Security boundaries: once payload capture is on, the trace table holds customer content, MONITOR reads all of it, and the exclusion tag's APPLY is an audit override.

Migration path: gateway on (metadata-only, explicit models) → one team via snow ai → team by team → revoke old keys → payload capture last.

5. Trade-offs

Six dimensions, six benefits, six costs. No net-zero row.

• Use it when multiple teams or agents call LLMs, finance needs chargeback, or security needs evidence of what was sent.
• Wait if you need a model Cortex doesn’t serve.
• Migrate now if either of these is true today:
• Shared provider API keys live in CI secrets (e.g. an OpenAI key in GitHub Actions). Every call bills to one key, so spend can’t be traced to a person or pipeline, and anyone with repo access can copy it.
• Engineers run coding agents on personal provider accounts. That traffic has no company logs, no limits, and no record of what code or data left.

Through the gateway, each call carries a Snowflake identity, leaves a trace, and is billed to the user who made it.

6. Architect’s takeaways

• Revoke PUBLIC USAGE first. The gateway ships open; production is default-deny.
• Spec as code, applied in full, verified with DESCRIBE. Full-replacement APIs without CI are governance regressions waiting to happen.
• Path ≠ entitlement ≠ oversight. USAGE, CORTEX_USER, MONITOR — three grants, three owners.
• PATs need a network policy. Without it you get a silent 401 and zero telemetry — the least documented, most likely first failure.
• x-snowflake-request-id is your reconciliation key from client response to AI_GATEWAY_USAGE_HISTORY.REQUEST_ID. Log it in your clients.
• Quotas backstop budgets. Budgets evaluate on a schedule (up to ~6.5h); quotas block in minutes.

The important change isn’t that Snowflake can now proxy LLM calls. It’s that every AI request now carries an identity, leaves a trace, and counts against a budget — verified end to end, from curl to credit — without a sidecar proxy or a spreadsheet.

Found this useful? Give it a clap and share it with others.

Follow Snowflake Chronicles for practical Snowflake engineering and architecture insights.

Read more: Snowflake Chronicles
Connect: Satish Kumar

Reuse & Attribution: Feel free to share, adapt, and build on this work. For public redistribution or substantial adaptations, please credit the original article and author with links.

Disclaimer: Shared for educational purposes. Test and validate before production use. Views are my own and do not represent any current or former employer.