# Lakenaut > Databricks theory written out in full and in English, organised three ways: by product area (mirroring the workspace panel), by learning path, and by certification roadmap. One concept, one page. Independent project; Databricks product names are used as nomenclature only. ## Learning paths - [SQL & Python Foundations](https://lakemap.fernandodenitto.workers.dev/paths/foundations/): The SQL and PySpark you need before any other path: the Spark dialect, types, joins, windows, MERGE, and the DataFrame API as it differs from pandas. (15/15 concepts written) - [Lakehouse Foundations](https://lakemap.fernandodenitto.workers.dev/paths/lakehouse-foundations/): How the Databricks platform is put together: control plane and compute, Delta Lake as the storage format, Unity Catalog as the governance layer, and the medallion pattern everything else builds on. (9/9 concepts written) - [Data Engineering](https://lakemap.fernandodenitto.workers.dev/paths/data-engineering/): Build production pipelines: ingest with Auto Loader, COPY INTO and Lakeflow Connect, transform with PySpark and SQL, orchestrate with Lakeflow Jobs, and ship it with bundles. (52/52 concepts written) - [Governance & Security](https://lakemap.fernandodenitto.workers.dev/paths/governance-security/): Unity Catalog end to end: the three-level namespace, managed and external tables, privileges, row filters, column masks and attribute-based policies. (16/16 concepts written) - [Platform & Administration](https://lakemap.fernandodenitto.workers.dev/paths/platform-administration/): Choose the right compute, read the Spark UI, tune the handful of settings that matter, and diagnose clusters that will not start or run out of memory. (16/16 concepts written) - [SQL & Analytics](https://lakemap.fernandodenitto.workers.dev/paths/sql-analytics/): Query the lakehouse from the SQL editor, model gold tables and views for BI, and understand what a SQL warehouse costs. (30/30 concepts written) - [Machine Learning](https://lakemap.fernandodenitto.workers.dev/paths/machine-learning/): Track experiments with MLflow, build features, register models in Unity Catalog and serve them behind an endpoint. (17/17 concepts written) - [Generative AI](https://lakemap.fernandodenitto.workers.dev/paths/generative-ai/): Foundation models on Databricks, vector search for retrieval, agents built with Agent Bricks and the Agent Framework, and how to evaluate them. (26/26 concepts written) ## Certifications - [Data Analyst Associate](https://lakemap.fernandodenitto.workers.dev/roadmaps/data-analyst-associate/): The entry-level certification for data analysis on Databricks. Querying and modeling data with Databricks SQL, building AI/BI dashboards and Genie Agents (the guide says Genie spaces), and applying basic data governance and security. Exam guide 2025-10. - [Data Engineer Associate](https://lakemap.fernandodenitto.workers.dev/roadmaps/de-associate/): The entry-level certification for data engineering on Databricks. Ingestion, transformation, Lakeflow Jobs, CI/CD with bundles, troubleshooting, and governance with Unity Catalog. Exam guide 2026-05. - [Data Engineer Professional](https://lakemap.fernandodenitto.workers.dev/roadmaps/de-professional/): The advanced certification for production data engineering on Databricks. ETL pipeline design, streaming, governance, security, and cost and performance optimization, plus CI/CD with Declarative Automation Bundles. Exam guide 2026-07. - [Generative AI Engineer Associate](https://lakemap.fernandodenitto.workers.dev/roadmaps/genai-engineer-associate/): The associate-level certification for designing and implementing LLM-enabled solutions on Databricks, covering prompt design, RAG pipelines, agent frameworks, AI Search (formerly Vector Search), Model Serving, and MLflow-based evaluation. Exam guide 2026-03. - [Machine Learning Associate](https://lakemap.fernandodenitto.workers.dev/roadmaps/ml-associate/): The associate-level certification for using Databricks to perform core machine learning tasks, from feature engineering and AutoML to MLflow tracking, model registration in Unity Catalog, and endpoint deployment. Exam guide 2025-03. ## SQL the Databricks Way - [Spark SQL, the dialect](https://lakemap.fernandodenitto.workers.dev/concepts/spark-sql-basics/): How Spark SQL differs from a textbook SQL dialect - three-level namespace, ANSI mode on by default, and what a warehouse deliberately leaves out. - [Types and casting](https://lakemap.fernandodenitto.workers.dev/concepts/sql-data-types/): The Spark SQL type system, CAST versus TRY_CAST under ANSI mode, DECIMAL precision limits, and how timestamps carry a time zone. - [Joins and set operations](https://lakemap.fernandodenitto.workers.dev/concepts/sql-joins-and-sets/): Join types including LEFT SEMI and LEFT ANTI, USING versus ON, join hints, and why a Spark SQL join means a shuffle across machines, not a local scan. - [Window functions](https://lakemap.fernandodenitto.workers.dev/concepts/sql-window-functions/): OVER, PARTITION BY, ranking and lag/lead functions, frame clauses, and QUALIFY - a Databricks convenience Postgres does not have. - [MERGE, UPDATE, DELETE on Delta](https://lakemap.fernandodenitto.workers.dev/concepts/sql-merge-and-dml/): MERGE INTO upserts and SCD-1, INSERT OVERWRITE versus REPLACE WHERE, and why Delta rewrites files instead of updating rows in place. - [Metric views](https://lakemap.fernandodenitto.workers.dev/concepts/metric-views/): A metric view is a Unity Catalog object whose body is YAML. It defines measures once, and every query picks its own grouping and reads the measures with MEASURE(). - [AI functions in SQL](https://lakemap.fernandodenitto.workers.dev/concepts/ai-functions-sql/): The ai_* family: task-specific functions for parsing, extraction, classification and text work, the general-purpose ai_query, and which of them are actually generally available. - [Pipe syntax for queries](https://lakemap.fernandodenitto.workers.dev/concepts/sql-pipe-syntax/): The |> operator chains query operators in the order the engine applies them, so a query reads top to bottom and a second aggregation is one more line instead of a subquery. - [SQL scripting and stored procedures](https://lakemap.fernandodenitto.workers.dev/concepts/sql-scripting/): BEGIN ... END compound blocks with local variables, loops, EXECUTE IMMEDIATE, condition handlers and cursors, and how to persist a working script as a Unity Catalog procedure. ## Playground - [AI Playground](https://lakemap.fernandodenitto.workers.dev/concepts/ai-playground/): AI Playground is a chat UI in the workspace for trying foundation models, comparing them side by side, and prototyping tool-calling agents without writing code. - [Foundation Model APIs](https://lakemap.fernandodenitto.workers.dev/concepts/foundation-model-apis/): Foundation Model APIs serve chat and embedding models as Databricks-hosted endpoints, pay-per-token or with provisioned throughput, callable from Python, SQL, or an OpenAI-compatible client. ## Runs - [Monitoring runs: states, run history, trends](https://lakemap.fernandodenitto.workers.dev/concepts/runs-monitoring/): The Runs page and a job's run history show states, durations, and the task graph. Comparing a run against the historical baseline is the first step in telling whether a job is getting worse. ## SQL Editor - [The SQL editor](https://lakemap.fernandodenitto.workers.dev/concepts/sql-editor-basics/): The SQL editor runs ad-hoc queries against a warehouse, with saved queries, parameters, snippets, and scheduled refreshes. - [Genie Code](https://lakemap.fernandodenitto.workers.dev/concepts/genie-code/): The assistant embedded across the workspace, governed by your own Unity Catalog permissions, with an agent mode that plans and runs work and asks before using a tool. - [SQL warehouse sessions](https://lakemap.fernandodenitto.workers.dev/concepts/sql-warehouse-sessions/): A session keeps variables, temporary views and tables, the current catalog and schema and session settings across statements, and it belongs to the query object and the warehouse rather than to you. - [Query parameters and session variables](https://lakemap.fernandodenitto.workers.dev/concepts/sql-parameters-and-variables/): Named parameter markers (:name) and the widgets they raise in the editor, notebooks, dashboards and Genie, the IDENTIFIER clause for dynamic names, session variables, and migrating off mustache. ## Workspace - [Architecture of the Data Intelligence Platform](https://lakemap.fernandodenitto.workers.dev/concepts/platform-architecture/): Databricks separates a vendor-managed control plane from a compute plane that processes the data, with storage in the customer's cloud, Delta Lake as the format, and Unity Catalog for governance. - [Notebooks](https://lakemap.fernandodenitto.workers.dev/concepts/notebooks-basics/): A notebook mixes SQL, Python, Scala, and Markdown cells with live results, widgets, and version history, and runs interactively or as a job task. - [Workspace files and volumes](https://lakemap.fernandodenitto.workers.dev/concepts/workspace-files-volumes/): Workspace files live with your code under /Workspace, Unity Catalog volumes govern non-tabular files under /Volumes, and DBFS is the deprecated predecessor. - [Secrets and credentials](https://lakemap.fernandodenitto.workers.dev/concepts/secrets-management/): Secret scopes store credentials outside your code, dbutils.secrets.get and the SQL secret() function redact them on read, and automation should use OAuth, not tokens. - [Git folders: branches, commits, pull requests](https://lakemap.fernandodenitto.workers.dev/concepts/git-folders/): A Git folder is a clone of a repository inside the workspace. From the UI you create branches, commit and push, resolve conflicts, and open the pull request on the provider. - [The CLI and the SDKs](https://lakemap.fernandodenitto.workers.dev/concepts/cli-and-sdk/): The Databricks CLI, the language SDKs, and Databricks Connect all share one authentication order and wrap the same REST API for scripting and automation. - [Declarative Automation Bundles and the Databricks CLI](https://lakemap.fernandodenitto.workers.dev/concepts/bundles-overview/): A bundle describes jobs, pipelines, and other assets in YAML alongside the code. With the Databricks CLI you validate, deploy, and run it, locally or from a CI/CD pipeline. - [Bundles: variables, targets, and per-environment overrides](https://lakemap.fernandodenitto.workers.dev/concepts/bundles-variables-targets/): The same bundle is promoted across dev, test, and prod thanks to variables with defaults, target overrides, ${…} substitutions, and the development and production modes. - [Bundles in a CI/CD pipeline](https://lakemap.fernandodenitto.workers.dev/concepts/bundles-ci-cd/): The documented flow for deploying a bundle from a build server: compile and test, upload a versioned artifact, validate, deploy. Separate dev, staging and production workspaces, and OIDC instead of tokens. - [Cost attribution and budgets](https://lakemap.fernandodenitto.workers.dev/concepts/cost-attribution-and-budgets/): Default tags, custom tags and where each one propagates, serverless usage policies, account budgets, and the billing system table that is the only record of what was actually spent. ## Agents - [Agents on Databricks](https://lakemap.fernandodenitto.workers.dev/concepts/agent-framework/): The four ways to build an agent here, from a no-code assistant to your own Python, and which page covers each. Also the bridge between the old Agent Framework name and the current one. - [Agent tools as Unity Catalog functions](https://lakemap.fernandodenitto.workers.dev/concepts/agent-tools-uc-functions/): Registering a Python or SQL function in Unity Catalog turns it into a governed agent tool, where the docstring becomes the schema the model reads and EXECUTE decides who may call it. - [Evaluating agents](https://lakemap.fernandodenitto.workers.dev/concepts/agent-evaluation/): mlflow.genai.evaluate() scores agent traces with built-in and custom judges, and the same scorers can run continuously in production. - [Agent memory](https://lakemap.fernandodenitto.workers.dev/concepts/agent-memory/): Long-term memory for an agent as a Unity Catalog securable, with entries scoped and pathed, governed and audited like a table rather than kept in a side database. - [Agent Bricks: Knowledge Assistant and Supervisor Agent](https://lakemap.fernandodenitto.workers.dev/concepts/agent-bricks/): Two declarative agent builders: Knowledge Assistant answers questions over your documents with citations, and Supervisor Agent routes a request across up to 50 subagents and tools. - [Deploy an agent on Databricks Apps](https://lakemap.fernandodenitto.workers.dev/concepts/agent-deployment-apps/): The documented way to ship a custom agent: the MLflow ResponsesAgent interface, an AgentServer inside a Databricks App, a bundle to deploy it, and Model Serving as the legacy path. - [Model Context Protocol on Databricks](https://lakemap.fernandodenitto.workers.dev/concepts/mcp-on-databricks/): Three places an agent's MCP servers come from: Databricks-managed servers, external servers registered in Unity Catalog as MCP Services, and custom servers hosted as apps. ## Catalog - [Unity Catalog, the governance layer](https://lakemap.fernandodenitto.workers.dev/concepts/unity-catalog-overview/): Unity Catalog is the central metastore of Databricks. Three-level namespace, securable objects, credentials to storage, lineage and audit shared by every workspace in a region. - [The metastore and how a workspace gets Unity Catalog](https://lakemap.fernandodenitto.workers.dev/concepts/uc-metastore-and-setup/): One metastore per cloud region holds the catalogs; the account console attaches it to workspaces; workspaces created after 8 November 2023 arrive already enabled with their own workspace catalog. - [Managed and external tables](https://lakemap.fernandodenitto.workers.dev/concepts/managed-vs-external-tables/): In a managed table Unity Catalog governs both metadata and files and deletes them on DROP; in an external table it governs only the metadata, and the files stay in the path you specified with LOCATION. - [External locations and storage credentials](https://lakemap.fernandodenitto.workers.dev/concepts/external-locations-and-storage-credentials/): A storage credential holds the cloud identity, an external location binds that credential to a path, and the privileges on the location decide who may read, write or create tables there. - [Privileges: GRANT, REVOKE, and DENY](https://lakemap.fernandodenitto.workers.dev/concepts/privileges-grant-revoke/): Unity Catalog privileges are granted to users, groups and service principals and inherit from the catalog down. USE CATALOG and USE SCHEMA are the entry door. - [Row filters and column masks](https://lakemap.fernandodenitto.workers.dev/concepts/row-filters-column-masks/): A row filter is a SQL UDF deciding which rows a user sees; a column mask transforms a value. Both attach with ALTER TABLE and tell groups apart with is_account_group_member. - [ABAC policies in Unity Catalog](https://lakemap.fernandodenitto.workers.dev/concepts/abac-policies/): ABAC policies apply row filters and column masks at the catalog or schema level based on governed tags. A rule written once with CREATE POLICY covers every tagged table, present and future. - [Data lineage in Unity Catalog](https://lakemap.fernandodenitto.workers.dev/concepts/unity-catalog-lineage/): Unity Catalog captures table and column lineage automatically for queries on governed objects, filters the graph by your privileges, and exposes it as system tables and an external lineage API. - [System tables](https://lakemap.fernandodenitto.workers.dev/concepts/system-tables/): The system catalog holds read-only tables that record cost, job runs, audit events, lineage, compute and query history for every workspace in a cloud region. - [Lakehouse Federation](https://lakemap.fernandodenitto.workers.dev/concepts/lakehouse-federation/): Query MySQL, PostgreSQL, Snowflake, Glue and others from Unity Catalog without moving the data, through connections and foreign catalogs, read-only and with pushdown. - [Workspace-catalog binding](https://lakemap.fernandodenitto.workers.dev/concepts/catalog-workspace-binding/): Every catalog in a metastore is reachable from every attached workspace until you bind it. Binding restricts a catalog to named workspaces, optionally read-only, and overrides individual grants. - [Managed and external volumes](https://lakemap.fernandodenitto.workers.dev/concepts/volumes-managed-vs-external/): A managed volume lives in the schema's managed storage and its files are purged after a 7-day window when dropped; an external volume registers a path you own and leaves the files behind. - [The information schema](https://lakemap.fernandodenitto.workers.dev/concepts/information-schema/): The SQL standard metadata views in every Unity Catalog catalog, filtered by what you are allowed to see, and the fastest way to answer questions about your own data estate. - [Governed tags](https://lakemap.fernandodenitto.workers.dev/concepts/governed-tags/): Governed tags are account-level tag keys with a fixed list of allowed values and their own assign permission, which is what makes them safe to write access policies against. - [Data Classification in Unity Catalog](https://lakemap.fernandodenitto.workers.dev/concepts/data-classification/): Data Classification scans table columns for sensitive values, writes system class tags onto the ones that match, and those tags are what an ABAC policy masks on. ## Dashboards - [AI/BI dashboards](https://lakemap.fernandodenitto.workers.dev/concepts/dashboards-overview/): AI/BI dashboards turn datasets built on governed tables into shareable visualizations, refreshed on a schedule or as a job task. - [Modelling data inside a dashboard](https://lakemap.fernandodenitto.workers.dev/concepts/dashboard-data-modeling/): Four ways to shape data inside an AI/BI dashboard: datasets, custom calculations, local metric views and relationships, and when the logic should move to Unity Catalog. - [Dashboard schedules and subscriptions](https://lakemap.fernandodenitto.workers.dev/concepts/dashboard-schedules-and-subscriptions/): A schedule reruns a published dashboard's dataset queries on a cadence and warms the query result cache; subscriptions deliver the resulting snapshot to email, Slack or Teams. - [Dashboard filters, parameters and variables](https://lakemap.fernandodenitto.workers.dev/concepts/dashboard-filters-and-variables/): The four ways a dashboard becomes interactive: field filters, query parameters, dashboard variables, and click-driven cross-filtering and drill-through. ## Data Ingestion - [Ingestion patterns: batch, streaming, incremental](https://lakemap.fernandodenitto.workers.dev/concepts/ingestion-patterns/): Batch, streaming and incremental are the three ways into the lakehouse, served by UI uploads, standard connectors and Lakeflow Connect. Choosing between them is an exam question. - [COPY INTO](https://lakemap.fernandodenitto.workers.dev/concepts/copy-into/): COPY INTO is the idempotent SQL command that loads files from object storage into a Delta table, remembering what it already loaded and running from a SQL warehouse. - [Auto Loader](https://lakemap.fernandodenitto.workers.dev/concepts/auto-loader/): Auto Loader is the cloudFiles streaming source that incrementally loads new files from object storage, with schema inference, schema evolution and a _rescued_data column. - [Lakeflow Connect: managed connectors](https://lakemap.fernandodenitto.workers.dev/concepts/lakeflow-connect/): Lakeflow Connect managed connectors ingest SaaS applications and databases into Unity Catalog streaming tables, with authentication, change data capture and scheduling handled for you. - [Ingesting from JDBC and REST APIs in notebooks](https://lakemap.fernandodenitto.workers.dev/concepts/ingestion-jdbc-rest/): With no managed connector, a notebook reads a source over JDBC or REST, writes to a Unity Catalog table and runs as a job task. Credentials live in a secret scope. - [Semi-structured data: JSON, nested data, VARIANT](https://lakemap.fernandodenitto.workers.dev/concepts/semi-structured-data/): JSON, struct, array and VARIANT represent nested data in a Delta table. Colon notation queries JSON, from_json builds structs, explode flattens arrays, VARIANT stores them binary. - [Zerobus Ingest](https://lakemap.fernandodenitto.workers.dev/concepts/zerobus-ingest/): Writing records straight into a Unity Catalog table from an application, over gRPC for throughput or REST for edge fleets, with no message bus in between. ## Python / PySpark - [PySpark versus pandas](https://lakemap.fernandodenitto.workers.dev/concepts/pyspark-vs-pandas/): Why PySpark code looks like pandas but behaves completely differently — lazy, distributed, and built around a DAG instead of an in-memory object. - [Python in notebooks: dbutils, widgets, modules](https://lakemap.fernandodenitto.workers.dev/concepts/python-in-notebooks/): How dbutils, widgets, and %pip fit into a Databricks notebook, and how to structure code so it survives the move to a proper module. - [Reading and writing DataFrames](https://lakemap.fernandodenitto.workers.dev/concepts/dataframe-io/): How spark.read and DataFrameWriter load and persist data on Databricks, and why saving to a Unity Catalog table beats saving to a path. - [Columns, rows, and DataFrame structure](https://lakemap.fernandodenitto.workers.dev/concepts/dataframe-columns-rows/): The PySpark operations for adding, renaming, dropping, and transforming columns, filtering rows, and exploding arrays, with their Spark SQL equivalents. - [Joins and unions between DataFrames](https://lakemap.fernandodenitto.workers.dev/concepts/dataframe-joins-unions/): How to combine DataFrames in PySpark and Spark SQL. Join types, multiple keys, broadcast joins, and the differences between union, unionByName, UNION ALL, and UNION. - [Deduplication and aggregations](https://lakemap.fernandodenitto.workers.dev/concepts/dataframe-dedup-aggregations/): Removing duplicates with distinct, dropDuplicates, or a window function, and aggregating with groupBy/agg, count, approx_count_distinct, avg, describe, and summary. - [UDFs and when not to write one](https://lakemap.fernandodenitto.workers.dev/concepts/udfs-and-alternatives/): The real cost of a Python UDF versus built-in functions, pandas UDFs, and Unity Catalog functions, ranked from cheapest to most expensive. ## Unity Gateway - [Unity Gateway (formerly AI Gateway)](https://lakemap.fernandodenitto.workers.dev/concepts/ai-gateway-basics/): The governance layer in front of every model call: model services as Unity Catalog securables, rate limits, usage attribution, service policies and one address per provider dialect. - [Model services on Unity Gateway](https://lakemap.fernandodenitto.workers.dev/concepts/model-services/): A model service is a governed LLM endpoint that lives in a Unity Catalog schema, with grants, routing, rate limits and usage tracking attached to the object itself. - [Guardrails for generative applications](https://lakemap.fernandodenitto.workers.dev/concepts/guardrails-and-service-policies/): Policies evaluated on the way in and on the way out of a model call, returning allow, deny or ask, plus the data-side masking that stops a prompt carrying what it should not. - [Gateway inference tables and usage tracking](https://lakemap.fernandodenitto.workers.dev/concepts/gateway-usage-and-inference-tables/): Three records of what went through Unity Gateway: payload logging to a Unity Catalog table, the usage system table, and where spend lands, plus the request tags that make attribution possible. - [Agent and MCP services as Unity Catalog securables](https://lakemap.fernandodenitto.workers.dev/concepts/agent-and-mcp-services/): Registering agents and MCP servers as Unity Catalog objects, so every team's agents are discoverable in one place and every tool a server exposes is chosen, policed and counted. ## Delta Lake - [Delta Lake, the lakehouse table format](https://lakemap.fernandodenitto.workers.dev/concepts/delta-lake-overview/): Delta Lake is Parquet plus a transaction log. The log provides ACID transactions, time travel, schema enforcement, and a history you can inspect with DESCRIBE HISTORY. - [Time travel and table history](https://lakemap.fernandodenitto.workers.dev/concepts/delta-time-travel/): DESCRIBE HISTORY, VERSION AS OF, and RESTORE let you inspect and recover earlier states of a Delta table, within the limits of log and file retention. - [Medallion architecture: bronze, silver, gold](https://lakemap.fernandodenitto.workers.dev/concepts/medallion-architecture/): Three layers of Delta tables with increasing quality. Bronze keeps the raw data, silver cleans and types it, gold aggregates it for the business. - [Gold objects: tables, views, materialized views, streaming tables](https://lakemap.fernandodenitto.workers.dev/concepts/gold-layer-objects/): The four objects you use to expose the gold layer in Unity Catalog. What they store, how they refresh, what they cost, and when to pick one over another. - [Upsert with MERGE INTO](https://lakemap.fernandodenitto.workers.dev/concepts/merge-upsert/): MERGE INTO applies inserts, updates and deletes to a Delta table in one atomic commit. Clause semantics, the single-match rule, deduplicating the source, and schema evolution. - [Liquid clustering](https://lakemap.fernandodenitto.workers.dev/concepts/liquid-clustering/): Liquid Clustering replaces partitioning and Z-ORDER with mutable clustering keys and incremental OPTIMIZE; predictive optimization runs OPTIMIZE, VACUUM, and statistics on its own on managed tables. - [OPTIMIZE, VACUUM, and file layout](https://lakemap.fernandodenitto.workers.dev/concepts/delta-optimize-vacuum/): OPTIMIZE compacts small files, VACUUM removes ones no version needs after a 7-day default retention, and predictive optimization now runs both for you. - [Change Data Feed](https://lakemap.fernandodenitto.workers.dev/concepts/change-data-feed/): Change Data Feed records every row-level insert, update, and delete on a Delta table so downstream jobs can propagate just the change, not the whole table. - [Predictive optimization](https://lakemap.fernandodenitto.workers.dev/concepts/predictive-optimization/): Predictive optimization decides on its own when to run OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables, on serverless compute, with no maintenance job to schedule. - [Partitioning, Z-order, and data skipping](https://lakemap.fernandodenitto.workers.dev/concepts/data-layout-partitioning-zorder/): Partitioning, ZORDER BY, file-skipping statistics and file size tuning are the layout toolkit that predates liquid clustering, which replaces the first two and cannot be combined with either. - [Deletion vectors](https://lakemap.fernandodenitto.workers.dev/concepts/deletion-vectors/): Deletion vectors record deleted and updated rows in metadata instead of rewriting whole Parquet files, and every reader applies them at scan time to work out which rows still count. - [Iceberg on Databricks](https://lakemap.fernandodenitto.workers.dev/concepts/iceberg-interoperability/): Managed Iceberg tables, foreign Iceberg tables, Iceberg reads on Delta, and the REST catalog that lets an engine outside Databricks read the same rows. - [Table history and transaction log checkpoints](https://lakemap.fernandodenitto.workers.dev/concepts/table-history-and-checkpoints/): DESCRIBE HISTORY returns one row per modifying operation with its parameters and metrics, and the log behind it is folded into Parquet checkpoints so readers never replay every JSON commit. ## Genie Agents - [Genie Agents](https://lakemap.fernandodenitto.workers.dev/concepts/genie-agents/): A Genie Agent (formerly a Genie space) answers natural-language questions over a small, curated set of governed tables, writing read-only SQL that runs with each user's own Unity Catalog permissions. - [The Genie knowledge store](https://lakemap.fernandodenitto.workers.dev/concepts/genie-knowledge-store/): How authors make a Genie Agent accurate: metadata first, then the knowledge store, then trusted example SQL and functions, with free text last. - [The Genie Ontology](https://lakemap.fernandodenitto.workers.dev/concepts/genie-ontology/): The context layer shared by Genie One and Genie Code. Half of it is Unity Catalog semantics that a human governs, half is ranked snippets Genie infers from existing assets. - [Genie benchmarks, feedback and monitoring](https://lakemap.fernandodenitto.workers.dev/concepts/genie-benchmarks-monitoring/): Benchmarks measure a Genie Agent against questions with known answers; the Monitoring tab and user feedback show what real users ask and where it fails. Together they drive every change to the agent. - [Using Genie outside the UI: API, embedding and agents](https://lakemap.fernandodenitto.workers.dev/concepts/genie-conversation-api/): The Conversation API, embedding, Slack and Teams, MCP and the Supervisor Agent let a Genie Agent answer from anywhere, always as an identity whose permissions apply. - [Domains and Pages](https://lakemap.fernandodenitto.workers.dev/concepts/uc-domains-and-pages/): The human half of the Genie Ontology. Domains group assets by business purpose, Pages define what a business term actually means, and Genie prefers both over what it infers. - [Genie One](https://lakemap.fernandodenitto.workers.dev/concepts/genie-one/): Genie One is the simplified Databricks surface for business users: dashboards, Genie Agents and Databricks Apps in one place, reachable with the Consumer access entitlement alone. - [Tuning a Genie Agent for correct answers](https://lakemap.fernandodenitto.workers.dev/concepts/genie-agent-tuning/): The three instruction surfaces of a Genie Agent: example SQL queries, Unity Catalog SQL functions and plain-text instructions, which of them produce verified answers, and the order to reach for each. ## Jobs & Pipelines - [Lakeflow Jobs, what a job is](https://lakemap.fernandodenitto.workers.dev/concepts/jobs-overview/): A job is the unit of orchestration in Databricks, a graph of tasks that runs on a compute of your choice, with triggers, parameters, and notifications. - [Tasks, dependencies, and the job graph](https://lakemap.fernandodenitto.workers.dev/concepts/jobs-task-dependencies/): The tasks of a job form a DAG. Dependencies set the order; the run-if condition decides whether a task starts based on the outcome of the tasks upstream. - [Control flow: retries, if/else, for each, run job](https://lakemap.fernandodenitto.workers.dev/concepts/jobs-control-flow/): Per-task retries and timeouts, If/else tasks for conditional branches, For each for loops, and Run job for composing jobs: the control logic lives in the graph, not in the code. - [Triggers: schedule, file arrival, table update, continuous](https://lakemap.fernandodenitto.workers.dev/concepts/jobs-triggers/): A job can start on a schedule (Quartz cron), when files land in a volume or external location, on a Unity Catalog table commit, or run continuously. Time-based vs. data-driven is a favorite exam question. - [Serverless compute for jobs](https://lakemap.fernandodenitto.workers.dev/concepts/jobs-serverless/): Serverless is the default compute for most Lakeflow Jobs tasks. Which task types take it, how environments and performance modes are declared, and when a job cluster still wins. - [Job and task parameters, dynamic values, and task values](https://lakemap.fernandodenitto.workers.dev/concepts/jobs-parameters/): Job parameters apply to every task, task parameters to just one; dynamic values like {{job.start_time.iso_date}} carry context; task values pass results from one task to another. - [Repair runs, retries, and notifications](https://lakemap.fernandodenitto.workers.dev/concepts/jobs-repair-runs/): A repair reruns only the failed tasks and their downstream tasks within the same run; notifications via email or system destinations, duration-based health rules, and queueing round out error handling. - [Lakeflow pipelines](https://lakemap.fernandodenitto.workers.dev/concepts/pipelines-overview/): A declarative pipeline describes streaming tables and materialized views in SQL or Python; the engine works out the graph, ordering and incremental updates. Formerly Delta Live Tables. - [Data quality: expectations and constraints](https://lakemap.fernandodenitto.workers.dev/concepts/pipelines-expectations/): Declarative pipeline expectations (warn, drop, fail) and Delta NOT NULL and CHECK constraints. Where each one is declared, what happens on a violation, and how to read the metrics in the event log. - [Change data capture with AUTO CDC](https://lakemap.fernandodenitto.workers.dev/concepts/pipelines-auto-cdc/): AUTO CDC and AUTO CDC FROM SNAPSHOT apply a change feed or a sequence of snapshots to a streaming table as SCD Type 1 or Type 2, handling out-of-order events for you. - [Choosing SQL or Python for a pipeline](https://lakemap.fernandodenitto.workers.dev/concepts/pipelines-sql-vs-python/): Both pipeline interfaces build the same dataflow graph, so most of the choice is taste. The asymmetries are few and they all run one way: Python covers the whole feature set, SQL does not. - [Concurrent runs, queueing, and the limits behind them](https://lakemap.fernandodenitto.workers.dev/concepts/jobs-queue-and-concurrency/): max_concurrent_runs defaults to 1 and caps at 1000. With queueing on, a run that hits one of three limits waits up to 48 hours instead of being skipped. The exact numbers are what the exam asks about. - [Sinks: writing out of a pipeline](https://lakemap.fernandodenitto.workers.dev/concepts/pipelines-sinks/): A sink lets a pipeline flow write somewhere that is not a pipeline-managed table, such as a Kafka topic or an external Delta table. Declared with create_sink, fed by an append flow, Python only. - [The pipeline event log](https://lakemap.fernandodenitto.workers.dev/concepts/pipelines-event-log/): Every pipeline records its own history in a Delta table: updates, flows, expectation counts and lineage. You read it with the event_log() function, or publish it to Unity Catalog and treat it as a table. - [Continuous jobs](https://lakemap.fernandodenitto.workers.dev/concepts/jobs-continuous/): A continuous job keeps exactly one run alive, restarting it in under a minute when it ends and backing off exponentially when it keeps failing. On serverless, only bounded triggers work. ## Alerts - [SQL alerts](https://lakemap.fernandodenitto.workers.dev/concepts/alerts-overview/): A SQL alert re-runs a query on a schedule and notifies a destination when a condition on the result is met. - [What an alert costs to run](https://lakemap.fernandodenitto.workers.dev/concepts/alert-compute-and-cost/): An alert runs on a warehouse you choose, and the schedule decides the bill. Serverless with a short auto-stop, alerts grouped on one warehouse, and the startup delay counted in. ## Compute - [Choosing compute: all-purpose, job cluster, serverless, SQL warehouse](https://lakemap.fernandodenitto.workers.dev/concepts/compute-options/): Databricks offers serverless compute, all-purpose clusters, job clusters, and SQL warehouses. Each has its own DBU-based cost model, its own limits, and a use case where it's the right choice. - [Serverless compute](https://lakemap.fernandodenitto.workers.dev/concepts/serverless-compute/): Serverless compute runs notebooks, jobs, and pipelines on Databricks-managed infrastructure with no cluster to configure, at the cost of some Spark control. - [Databricks Runtime and Photon](https://lakemap.fernandodenitto.workers.dev/concepts/runtime-and-photon/): Databricks Runtime is the versioned Spark-plus-libraries bundle a classic cluster runs; Photon is its optional vectorized C++ engine for SQL and DataFrame work. - [Cluster policies](https://lakemap.fernandodenitto.workers.dev/concepts/cluster-policies/): A cluster policy is an admin-defined JSON rule set that locks down what a user can configure on a cluster, enforcing cost, security, and tagging limits. - [Instance pools and autoscaling](https://lakemap.fernandodenitto.workers.dev/concepts/instance-pools/): A pool keeps idle VMs on standby so clusters attach to them instead of waiting on the cloud provider, and autoscaling adjusts worker count once a cluster is up. - [Basic Spark tuning parameters](https://lakemap.fernandodenitto.workers.dev/concepts/spark-tuning-basics/): The four Spark parameters the exam expects you to know, what AQE already does for you on Databricks, how to set them, how to measure the effect, and what's not available on serverless. - [Spark UI: skew, shuffle, and spill](https://lakemap.fernandodenitto.workers.dev/concepts/spark-ui-bottlenecks/): A stage's summary metrics in the Spark UI (min, median, max for duration, shuffle, and spill) tell you whether a job is slow because of skew, too much shuffle, or disk spill, and point to the fix. - [Diagnosing clusters: startup failures, libraries, out of memory](https://lakemap.fernandodenitto.workers.dev/concepts/cluster-troubleshooting/): The event log and driver logs tell you why a cluster didn't start; precedence rules explain library conflicts; telling driver OOM apart from executor OOM points you to the right fix. - [Access modes: standard and dedicated](https://lakemap.fernandodenitto.workers.dev/concepts/compute-access-modes/): Standard access mode shares one compute resource between isolated users; dedicated assigns it to one user or group. What each allows, which Unity Catalog features need which, and the old names. ## Experiments - [MLflow tracking on Databricks](https://lakemap.fernandodenitto.workers.dev/concepts/mlflow-tracking/): MLflow tracking records the parameters, metrics, and artifacts of every training run so you can compare runs and reproduce the best one. - [MLflow 3 for models](https://lakemap.fernandodenitto.workers.dev/concepts/mlflow-3-models/): MLflow 3 makes the model a first-class object with its own id, metrics and artifacts, defaults the registry to Unity Catalog, and renames enough of the API to break MLflow 2 code. - [MLflow Tracing for GenAI applications](https://lakemap.fernandodenitto.workers.dev/concepts/mlflow-tracing/): MLflow Tracing records each GenAI request as a tree of spans carrying inputs, outputs, latency and token counts, stored in an MLflow experiment or in Unity Catalog. - [Evaluation datasets for generative AI](https://lakemap.fernandodenitto.workers.dev/concepts/evaluation-datasets/): An evaluation dataset is a governed Unity Catalog table of inputs and expectations, curated from traces, expert labels, synthetic generation or by hand, that an agent is scored against on every change. - [Human feedback on generative AI output](https://lakemap.fernandodenitto.workers.dev/concepts/human-feedback/): Human judgement reaches MLflow as assessments on a trace, from developers annotating in the UI, from experts working a review queue, and from end users pressing thumbs up or down. - [Prompt registry](https://lakemap.fernandodenitto.workers.dev/concepts/prompt-registry/): The MLflow prompt registry stores prompt templates as versioned Unity Catalog objects with aliases for production, so a prompt change can be evaluated and rolled back like a model version. - [Production monitoring for GenAI apps](https://lakemap.fernandodenitto.workers.dev/concepts/genai-production-monitoring/): Registered scorers run continuously against a sampled fraction of live traces and attach their verdicts to each trace as feedback, so quality drift shows up without a scheduled evaluation. - [Training and tuning a classic model](https://lakemap.fernandodenitto.workers.dev/concepts/model-training-and-tuning/): Where a model trains on Databricks, the estimator and transformer vocabulary, and the tuning libraries to use now that Hyperopt is gone from the machine learning runtime. - [AutoML](https://lakemap.fernandodenitto.workers.dev/concepts/automl/): Automatic model search over classification, regression and forecasting that hands back a notebook per trial, which is the part that makes it useful rather than magic. - [AI Runtime and serverless GPU compute](https://lakemap.fernandodenitto.workers.dev/concepts/ai-runtime/): Serverless GPU compute for training and fine-tuning: a @distributed decorator in notebooks, the air CLI with a workload YAML, and the successor to Foundation Model Fine-tuning. ## Streaming - [Structured Streaming on Databricks](https://lakemap.fernandodenitto.workers.dev/concepts/structured-streaming-basics/): Structured Streaming treats a data stream as a table that keeps growing, processed in repeated micro-batches with readStream/writeStream, triggers, and checkpoints. - [Trigger intervals in Structured Streaming](https://lakemap.fernandodenitto.workers.dev/concepts/streaming-triggers/): The trigger decides when a streaming query looks for new data. Default, processingTime, availableNow and realTime, and what each one costs you. - [Reading and writing Apache Kafka](https://lakemap.fernandodenitto.workers.dev/concepts/kafka-streaming/): The kafka format as a Structured Streaming source and sink: record schema, offsets, checkpoints, authentication, and the end-to-end Kafka to Delta pattern. - [Arbitrary sinks with foreachBatch](https://lakemap.fernandodenitto.workers.dev/concepts/foreachbatch/): foreachBatch hands each micro-batch to your own function as a batch DataFrame. It guarantees at-least-once, so exactly-once is something you build on batchId. - [Watermarks and stateful streaming](https://lakemap.fernandodenitto.workers.dev/concepts/streaming-watermarks-state/): A watermark bounds how long Structured Streaming waits for late data, so windowed aggregations, stream-stream joins, and deduplication can drop old state instead of growing forever. ## Data Quality - [Data quality on Databricks, layer by layer](https://lakemap.fernandodenitto.workers.dev/concepts/data-quality-overview/): Constraints, expectations, DQX, data profiling and anomaly detection each catch a different failure. What each layer sees, what it costs, and how to choose. - [Data profiling and anomaly detection](https://lakemap.fernandodenitto.workers.dev/concepts/data-quality-monitoring/): Unity Catalog's own quality monitoring, with two halves that answer different questions, two metric tables you can query, and a schedule that costs serverless compute. - [DQX: data quality checks for PySpark](https://lakemap.fernandodenitto.workers.dev/concepts/dqx-framework/): The Databricks Labs framework that validates PySpark DataFrames and tables, splits the good rows from the bad ones, and explains every failure row by row. ## Features - [Feature engineering and the feature store](https://lakemap.fernandodenitto.workers.dev/concepts/feature-engineering/): Feature tables in Unity Catalog let a team compute a feature once and reuse the exact same values for training and for real-time serving. - [Training sets and point-in-time joins](https://lakemap.fernandodenitto.workers.dev/concepts/training-sets-and-point-in-time/): How a training set is assembled from feature lookups, why the logged model re-resolves features at scoring time, and how a timestamp key keeps future facts out of training. - [Online Feature Store](https://lakemap.fernandodenitto.workers.dev/concepts/online-feature-store/): The low-latency half of the feature store, a Lakebase-backed copy of a feature table that a serving endpoint reads by primary key on every request. - [Feature Views](https://lakemap.fernandodenitto.workers.dev/concepts/feature-views/): Declarative features, defined as a source plus an aggregation over a time window, registered in Unity Catalog and materialised by managed pipelines rather than by a table you build yourself. ## Discover / Marketplace - [Marketplace and Clean Rooms](https://lakemap.fernandodenitto.workers.dev/concepts/marketplace-delta-sharing/): Marketplace is the public catalogue of data, models and notebooks built on sharing. Clean Rooms are the opposite trade, a joint computation where neither side sees the other's rows. - [Sharing data with OpenSharing](https://lakemap.fernandodenitto.workers.dev/concepts/opensharing-overview/): How a share, a provider and a recipient actually work: the two sharing protocols, bearer tokens against OIDC federation, sharing with history, and what cannot be shared. ## Query History - [Reading the query profile](https://lakemap.fernandodenitto.workers.dev/concepts/query-profile/): The query profile turns a finished statement into a graph of operators and metrics for spotting scans, spills, and bad joins. - [Query tags](https://lakemap.fernandodenitto.workers.dev/concepts/query-tags/): Key-value tags attached to a session or a single statement, surfacing in query history and the system tables, which is how warehouse spend gets an owner. - [Query caching layers](https://lakemap.fernandodenitto.workers.dev/concepts/sql-query-caching/): Databricks SQL has five caches, not one: the UI cache, the local and remote result caches, the disk cache and the AI/BI dashboard cache. Each has its own lifetime and its own invalidation rule. - [Query performance insights](https://lakemap.fernandodenitto.workers.dev/concepts/query-performance-insights/): Databricks analyses every finished statement and returns named insights, each with a recommendation, plus a record of the accelerations it already applied on your behalf. ## Ecosystem & Tools - [Databricks Labs, the tools around the platform](https://lakemap.fernandodenitto.workers.dev/concepts/databricks-labs-tools/): Labs projects are open-source, unsupported, and often the fastest answer to migration, data quality, test data and scaffolding. What each one does and how much to lean on it. ## Models - [Models in Unity Catalog](https://lakemap.fernandodenitto.workers.dev/concepts/models-in-uc/): Models in Unity Catalog register a trained model under a three-level name and mark its deployment status with aliases instead of stages. - [MLflow deployment jobs](https://lakemap.fernandodenitto.workers.dev/concepts/mlflow-deployment-jobs/): A deployment job binds a registered model to a Lakeflow job whose evaluation, approval and deployment tasks fire whenever a new model version appears, with a human gate in the middle. - [Monitoring a deployed model](https://lakemap.fernandodenitto.workers.dev/concepts/model-monitoring/): Attaching a data profile to a model's inference table, so prediction quality, drift and fairness become Delta tables you can query and alert on. ## SQL Warehouses - [Sizing a SQL warehouse](https://lakemap.fernandodenitto.workers.dev/concepts/sql-warehouse-sizing/): How warehouse type, t-shirt size, and cluster scaling combine to set query latency, concurrency, and cost. - [Standalone materialized views in Databricks SQL](https://lakemap.fernandodenitto.workers.dev/concepts/materialized-views-sql/): A materialized view created from Databricks SQL gets its own serverless pipeline. How to schedule the refresh, when it is incremental, and why the bill lands on pipelines rather than on your warehouse. - [Streaming tables from Databricks SQL](https://lakemap.fernandodenitto.workers.dev/concepts/streaming-tables-sql/): A streaming table declared in the SQL editor, refreshed incrementally by a serverless pipeline the system creates for you, without opening a pipeline editor. - [SQL warehouse types and channels](https://lakemap.fernandodenitto.workers.dev/concepts/sql-warehouse-types-and-channels/): The three generally available warehouse types and the Beta fourth one, which acceleration features each has, how fast each starts, and what the Preview channel is for. ## Serving - [Model serving endpoints](https://lakemap.fernandodenitto.workers.dev/concepts/model-serving-endpoints/): A serving endpoint puts a custom or foundation model behind a REST API with autoscaling, traffic splitting, and built-in request logging. - [Batch inference with ai_query](https://lakemap.fernandodenitto.workers.dev/concepts/batch-inference-ai-query/): Running a model over a whole table from SQL, with the platform handling parallelism, retries and scale, and failOnError deciding whether one bad row ruins the job. - [Structured outputs](https://lakemap.fernandodenitto.workers.dev/concepts/structured-outputs/): response_format constrains a chat model's answer to valid JSON or to a JSON schema, with a 64-key ceiling and a deliberately reduced subset of JSON Schema. - [Serving compute and scaling](https://lakemap.fernandodenitto.workers.dev/concepts/serving-compute-and-scaling/): Sizing a custom model endpoint: workload size and type including the GPU options, scale to zero and its cold start, provisioned concurrency from target QPS and latency, and route optimisation. - [Paying for a foundation model: tokens, units and reservations](https://lakemap.fernandodenitto.workers.dev/concepts/provisioned-throughput/): The four capacity modes for a Databricks-hosted foundation model: pay-per-token, priority, on-demand provisioned throughput and reserved provisioned throughput, and how to size model units. - [External models and model provider services](https://lakemap.fernandodenitto.workers.dev/concepts/external-models/): Reaching a model Databricks does not host, either as an external model on a serving endpoint or as a model provider service whose credentials and spend live in Unity Catalog. ## AI Search - [Databricks AI Search (formerly Vector Search)](https://lakemap.fernandodenitto.workers.dev/concepts/vector-search-basics/): AI Search (formerly Mosaic AI Vector Search) turns Delta tables into governed, queryable embedding indexes with hybrid keyword-vector search and filters. - [AI Search index types and sync modes](https://lakemap.fernandodenitto.workers.dev/concepts/ai-search-indexes/): The four AI Search index options, continuous against triggered sync, and standard against storage-optimized endpoints, with the choices you cannot undo later. - [Building a RAG pipeline](https://lakemap.fernandodenitto.workers.dev/concepts/rag-pipeline/): A RAG pipeline parses, chunks, embeds, indexes, retrieves, and generates — and its quality is decided mostly by the chunking step. ## Optional - [Concept graph](https://lakemap.fernandodenitto.workers.dev/graph/): every concept and how they connect - [News](https://lakemap.fernandodenitto.workers.dev/news/): platform updates - [RSS](https://lakemap.fernandodenitto.workers.dev/rss.xml)