Gateway inference tables and usage tracking

What it is

Unity Gateway keeps three separate records of a model call, and they answer three different questions.

Inference tables hold the payloads: the request and the response as JSON, in a Delta table in a schema you choose. This is the record you read when the answer was wrong. The usage system table, system.ai_gateway.usage, holds one row per request with tokens, latency and identity, and no payload. This is the record you read when you want to know how much, by whom. Cost lives in the billing tables, in DBUs for Databricks-hosted models and as an estimate for external providers.

Unity Gateway (formerly AI Gateway) covers what the gateway is and Model services on Unity Gateway covers the endpoint object. This page is about reading back what passed through it.

Why it exists

An LLM call is the one part of an application whose output you cannot reconstruct from its input. Rerunning the same prompt gives a different answer, the model behind the service may have changed since, and the retrieved context that shaped the response is gone. Without the payload written down at the time, a complaint about a wrong answer is unfalsifiable.

The usage table exists for a different reason: the bill. Token spend is generated by whoever writes the prompt, which in a shared service means a number nobody owns. One row per request with a requester on it turns that into an allocation, and request tags turn it into an allocation somebody in finance recognises.

How it works

Inference tables: the payloads

You turn logging on per model service by naming a catalog and a schema, from the gateway page or the service configuration. The table is created after the first request arrives, not when you save the setting, so an empty schema after configuration is expected rather than broken.

The schema is 17 columns. The ones you use most: request_id and invocation_id to correlate, event_time, latency_ms and time_to_first_byte_ms for timing, request and response for the raw JSON, requester and request_tags for attribution, status_code, destination_type, destination_name and destination_model for where the call actually went, sampling_fraction if you are not logging everything, and logging_error_codes and schema_version for the health of the logging itself.

Five constraints decide whether you can rely on it:

Constraint	Consequence
Requests and responses over 10 MiB are not logged	a long-context call can succeed and leave no payload
Logs are typically available within minutes	not a live view, so do not build an alert on the absence of a row
External storage catalogs only	a default storage catalog cannot host the table
Rows may not appear for 401, 403, 429 and 500 responses	the failures you most want to inspect are the least reliably captured
Renaming, dropping or altering the table schema breaks logging	treat it as owned by the gateway, not as a table to refactor

Setting it up needs MANAGE on the model service, CREATE TABLE in the target schema, and the usual USE CATALOG and USE SCHEMA above it.

The usage table: tokens, latency, identity

system.ai_gateway.usage is written for every request, with around 30 columns. Beyond the identifiers (account_id, workspace_id, request_id, invocation_id, endpoint_id, endpoint_name) it carries input_tokens, output_tokens and total_tokens with a token_details breakdown for cached and reasoning tokens, latency_ms and time_to_first_byte_ms, requester and requester_type, ip_address, url, user_agent, api_type, status_code, destination_type, destination_name, destination_model, endpoint_tags, request_tags, and routing_information showing any fallback attempts.

Account and metastore admins can read it by default and can grant it onwards to teams who need their own numbers. One gap to know: token usage is not tracked for non-streaming, non-embedding responses larger than 1 MiB, so a table of token totals can undercount a workload built on long single-shot answers.

Because status_code is there, a throttled call is visible: a caller over a queries-per-minute or tokens-per-minute limit gets 429, and the row lands like any other. Counting 429 by requester is how you tell a rate limit from an application bug. The limits themselves and their precedence are in Model services on Unity Gateway.

Tagging requests so attribution survives

requester tells you which user or service principal made the call, which is rarely the question. The question is which project, which environment, which customer. Send a Databricks-Ai-Gateway-Request-Tags header holding a JSON object of string keys and values, and those pairs land in request_tags in both the usage table and the inference table.

Tags are the difference between “the support service spent 40 million tokens” and “31 million of those were the nightly backfill”. Add them before the service has users: a tag you start sending in month three does not retrofit onto month one.

Where cost shows up

Two places, depending on who runs the model.

Databricks-hosted models bill through system.billing.usage, where the MODEL_SERVING records are enriched with gateway metadata: usage_metadata.ai_gateway.endpoint_name as the fully qualified service name, usage_metadata.ai_gateway.destination_model for the model that handled it, identity_metadata.run_by for the requester, custom_tags, and usage_quantity in DBUs.

External providers bill you directly, so Databricks estimates: system.ai_gateway.external_model_spend, aggregated hourly, with usage_quantity in USD derived from the provider’s published prices. It is informational by design, and it excludes the custom provider type. See External models and model provider services.

Account admins can also generate a prebuilt dashboard from Govern and Create Usage Dashboard, with tabs for overview, performance, usage, cost observability, external MCP servers and coding agents.

Example: the two questions, in order

How much, by whom, over the last week:

SELECT request_tags['project']      AS project,
       requester,
       destination_model,
       count(*)                     AS requests,
       count_if(status_code = 429)  AS throttled,
       sum(total_tokens)            AS tokens,
       percentile(latency_ms, 0.95) AS p95_latency_ms
FROM system.ai_gateway.usage
WHERE endpoint_name = 'main.ai.support_llm'
  AND event_time >= current_date() - INTERVAL 7 DAYS
GROUP BY ALL
ORDER BY tokens DESC;

Then the payload for one of those requests, from the inference table:

SELECT event_time, requester, latency_ms, request, response
FROM main.observability.support_llm_payloads
WHERE request_id = '<request_id from the query above>';

Sending the tag that makes the first query possible:

import json, os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["DATABRICKS_TOKEN"],
    base_url="https://<workspace-url>/ai-gateway/mlflow/v1",
)

reply = client.chat.completions.create(
    model="main.ai.support_llm",
    messages=[{"role": "user", "content": "Summarise ticket 44812 in two sentences."}],
    extra_headers={
        "Databricks-Ai-Gateway-Request-Tags": json.dumps(
            {"project": "support-triage", "env": "prod"}
        )
    },
)

Common mistakes

• Turning on inference tables and assuming everything is captured. Payloads over 10 MiB are dropped, and rows may be missing for 401, 403, 429 and 500, which is exactly the set you wanted.
• Managing the inference table like a normal Delta table. Renaming it, dropping it or altering its schema breaks logging, and nothing else tells you.
• Never sending request tags. The usage table will tell you which service principal spent the tokens, which is the identity of a job, not the name of a cost centre.
• Alerting on the usage table as if it were live. Rows arrive within minutes, so a five-minute window produces false alarms about a service that is fine.
• Reading the external spend estimate as the invoice. It is computed from published list prices for informational use; reconcile it with the provider’s own bill before anyone budgets from it.