Product area
SQL the Databricks Way
The SQL fundamentals that actually matter on Databricks: Spark SQL, differences from Postgres, types, functions, and the patterns that keep showing up in exams.
Concepts
- 1Spark SQL, the dialectSpark SQL is the dialect you write everywhere on Databricks: in the SQL editor, inside a notebook cell, behind spark.sql(...) in PySpark, and inside a pipeline. It looks close enough to Postgres or MySQL that a query someone learned on psql will often just…Beginner
- 2Types and castingSpark SQL's type system covers the usual numeric, string, and temporal types, plus complex types (ARRAY, MAP, STRUCT) and VARIANT for semi-structured values (see semi structured data). Types matter more here than in a row-store: Delta stores data…Beginner
- 3Joins and set operationsJoins combine rows from two table references on a matching condition; set operators combine the results of two queries with the same shape. Spark SQL's SQL-level syntax reads like any relational database, but the plan it compiles to is distributed - matching…Beginner
- 4Window functionsA window function computes a value per row using a set of "peer" rows - defined by PARTITION BY and ORDER BY - without collapsing them into one row the way GROUP BY does. Ranking a row within its group, comparing it to the previous row, or running a…Intermediate
- 5MERGE, UPDATE, DELETE on DeltaMERGE INTO, UPDATE, DELETE, and INSERT OVERWRITE are the statements that change data already sitting in a Delta table instead of just appending to it. They read like ordinary DML from any relational database, but Delta has no in-place row storage: every one…Intermediate
- 6AI functions in SQLAI functions are built-in SQL functions, all named ai, that apply a model to a column. They run from the SQL editor, from notebooks, from Lakeflow pipelines and from jobs, and they need no endpoint of your own: the query runs on the compute you submit it…Intermediate
- 7Pipe syntax for queriesAny query on Databricks can be followed by a chain of pipe operators, each separated by the token |> and each consuming the result of the one before it. A pipeline normally starts with FROM main.silver.orders or TABLE main.silver.orders, but any query can…Intermediate
- 8SQL scripting and stored proceduresSQL scripting is the procedural half of Databricks SQL. Everything lives inside a compound statement: a BEGIN ... END block that first declares local variables, conditions, cursors and error handlers, then runs a sequence of queries, DML, DDL, GRANT, loops…Advanced
Also covered here
- Batch inference with ai_queryaiquery calls a model from SQL. Point it at an endpoint, give it a prompt built from your columns, and it returns the model's answer as a column. Run it over a table and you have batch inference, with no loop, no notebook and no serving client.
- Metric viewsA metric view is a Unity Catalog securable whose definition is a YAML document rather than a SELECT. The YAML names a source, optional joins and a filter, then two lists: fields (scalar expressions you group and filter by) and measures (aggregate expressions…
News
Resources
Nothing of that kind for this area. Try the full list.