Coding agents on Databricks, and how to keep them safe

What it is

A coding agent — Claude Code, Cursor, GitHub Copilot, Codex — writes and runs code from a terminal or an editor. On its own it knows Databricks only as well as its training data does, which means old product names, flags that moved, and APIs that changed. Databricks ships three things to close that gap:

• Agent skills: Markdown instruction files (SKILL.md plus reference notes) that the agent loads when a task matches, one per product area — Unity Catalog, Lakeflow Jobs and pipelines, bundles, Model Serving, Genie, AI Search, Databricks Apps and more. They are knowledge: patterns, the right CLI commands, the mistakes to avoid. They live in databricks/databricks-agent-skills and are installed by the Databricks CLI.
• The AI Dev Kit: a Field Engineering toolkit (databricks-solutions/ai-dev-kit) whose installer sets up the skills through the same CLI and adds a standalone MCP server exposing more than forty Databricks tools, a Python library behind it, and a builder web app with Claude Code built in. The skills are maintained by Databricks engineering; the MCP server is maintained on a best-effort basis.
• Managed MCP servers: tools Databricks hosts for you — Genie One, a single Genie Agent, AI Search, Databricks SQL, Unity Catalog functions — which an agent calls over HTTPS with your OAuth identity (see Model Context Protocol on Databricks). They are in Public Preview.

Skills tell the agent how; MCP servers and the CLI are what it acts with.

Why it exists

Without skills, an agent asked to “schedule this notebook every night” writes a plausible job definition from memory: a deprecated field here, a cluster spec that the workspace policy rejects there, a Workflows API that has since been renamed. With skills, the same request loads the jobs skill, which says to define the job in a bundle, validate it, deploy to a development target, and choose a profile deliberately. The productivity gain is less about speed than about the agent doing the Databricks thing the current way the first time.

The same capability is the risk. An agent that can deploy a bundle can also destroy one; an agent that can run a query can run DROP TABLE. The skills themselves insist on least privilege and on never picking a profile without asking, but instructions are not permissions.

How it works

Installing the skills

The Databricks CLI detects the coding agents on the machine and installs for each: agents with plugin support (Claude Code, Codex CLI, GitHub Copilot) get a databricks plugin, the others get skill files linked from ~/.databricks/aitools/skills/.

databricks aitools install                                  # every detected agent, global scope
databricks aitools install --agents claude-code --scope project
databricks aitools install --skills bundles,sql             # only some skills
databricks aitools list                                     # what is available and installed
databricks aitools update                                   # keep them current
databricks aitools uninstall

--experimental adds skills that are explicitly not officially supported. In Claude Code, the same skills are also available from the plugin marketplace (/plugin marketplace add databricks/databricks-agent-skills, then /plugin install databricks@databricks-agent-skills); the CLI route is the one Databricks recommends because it tracks the stable versions.

The AI Dev Kit installer (install.sh, or install.ps1 on Windows) delegates the skills to databricks aitools and additionally installs its MCP server, for the current project by default or with --global. Its --uninstall flag removes the skill folders, the MCP runtime in ~/.ai-dev-kit, and the databricks entry from each editor’s MCP configuration; --dry-run shows what it would do first.

How the agent reaches the workspace

Skills drive the CLI, so the agent acts with whatever the CLI is authenticated as: a profile in ~/.databrickscfg (The CLI and the SDKs). Managed MCP servers act on behalf of the signed-in user, and Unity Catalog decides what that user may see and change. In both cases the agent has exactly your access — which is why the safe setup starts with the identity, not with the agent.

Working safely: three layers

1. The identity (the real boundary). Give the agent a profile that cannot do lasting damage. The simplest version is a development workspace, or a user or service principal with read access to production catalogs and write access only to a sandbox schema (Privileges: GRANT, REVOKE, and DENY). Anything the Databricks permissions forbid, no prompt injection, typo or overeager plan can do. Keep production credentials off the machine the agent runs on, and let Bundles in a CI/CD pipeline deploy to production with a service principal.

2. Where it deploys. Point the agent at a bundle target with mode: development (Bundles: variables, targets, and per-environment overrides): deployed resources are prefixed with [dev <your name>], and every schedule and trigger is paused, so a job the agent deploys does not start running on its own. mode: production validates the opposite — pipelines not in development, run_as and permissions set explicitly — and belongs to CI.

3. What the agent may run without asking. Claude Code evaluates permission rules in the order deny, then ask, then allow, and a deny rule wins over any allow. Put destructive commands in deny, deployments and job runs in ask, and start exploratory sessions in plan mode, where the agent reads and runs read-only commands but changes nothing. These rules match the command as the agent writes it — not the same program called through sh -c or an absolute path — so they catch mistakes, while the boundary against anything worse stays layer 1.

A SQL statement is the case layer 3 cannot see: DROP TABLE inside a query looks, to a shell rule, like any other query. Only Unity Catalog privileges stop it.

Example

A project-level .claude/settings.json for a repository that holds a bundle:

{
  "permissions": {
    "defaultMode": "plan",
    "deny": [
      "Bash(databricks * delete *)",
      "Bash(databricks * permanent-delete *)",
      "Bash(databricks bundle destroy *)",
      "Bash(databricks * --profile prod*)",
      "Read(~/.databrickscfg)"
    ],
    "ask": [
      "Bash(databricks bundle deploy *)",
      "Bash(databricks bundle run *)",
      "Bash(databricks jobs run-now *)",
      "Bash(databricks * create *)",
      "mcp__databricks__*"
    ],
    "allow": [
      "Bash(databricks bundle validate *)",
      "Bash(databricks auth profiles)",
      "Bash(databricks catalogs list *)",
      "Bash(databricks tables get *)"
    ]
  }
}

The identity the agent’s profile signs in as, granted just enough in Unity Catalog:

-- Read production, write only the sandbox.
GRANT USE CATALOG ON CATALOG prod TO `agent-dev@example.com`;
GRANT USE SCHEMA, SELECT ON SCHEMA prod.sales TO `agent-dev@example.com`;
GRANT USE CATALOG ON CATALOG sandbox TO `agent-dev@example.com`;
GRANT ALL PRIVILEGES ON SCHEMA sandbox.agent TO `agent-dev@example.com`;

And the bundle target the agent deploys to:

targets:
  dev:
    mode: development
    default: true
    workspace:
      profile: agent-dev

With this, “build me a nightly job that cleans prod.sales.orders into a sandbox table” gets a plan first, a validated bundle, a paused [dev …] job after you approve the deploy — and no path to changing prod.sales at all.

Common mistakes

• Letting the agent use DEFAULT, and DEFAULT points at production. The skills tell the agent to ask which profile to use; keep DEFAULT on a development workspace anyway, so a command that forgets --profile lands somewhere harmless.
• Treating deny rules as the security boundary. They stop the commands an agent normally writes. The permissions of the identity stop everything else.
• Relying on shell rules to catch SQL. DROP, DELETE and TRUNCATE travel inside a query string; grant SELECT, not MODIFY, where the data matters.
• Installing --experimental skills on a production-facing setup. They are best-effort and not officially supported; install them where a wrong suggestion costs nothing.
• Forgetting the MCP tools. The AI Dev Kit’s MCP server and the managed MCP servers are tools like any other: add an ask rule for them (mcp__<server name>__*), or approve each call.
• Deploying straight to a production target from a laptop. Development mode for the agent, CI with a service principal for production.