Product area
Python / PySpark
Python and PySpark in the Databricks context: the DataFrame API, differences from pandas, notebooks, and transformation patterns.
Concepts
- 1PySpark versus pandasPySpark's DataFrame and pandas' DataFrame share a name and a lot of method names (select, filter, groupby...), but they are built on opposite execution models. A pandas DataFrame lives entirely in the memory of one process and every line of code runs…Beginner
- 2Python in notebooks: dbutils, widgets, modulesA Databricks notebook is a Python (or SQL, Scala, R) session attached to a cluster, split into cells, plus a set of platform-specific helpers that don't exist in a plain .py script: dbutils, widgets, and magic commands like %pip and %run. None of this is…Beginner
- 3Reading and writing DataFramesspark.read builds a DataFrame from files or an external source; df.write persists a DataFrame somewhere. Databricks defaults everything to Delta Lake (see delta lake overview), so spark.read.parquet(...) and friends exist mostly for reading data that arrived…Beginner
- 4Columns, rows, and DataFrame structureA PySpark DataFrame is a distributed, immutable table: every operation returns a new DataFrame without touching the original, and nothing actually runs until you ask for a result (display, write, count). This is the first stumbling block for anyone coming…Beginner
- 5Joins and unions between DataFramesA join combines two DataFrames by pairing rows that satisfy a condition; a union stacks them on top of each other. These are the two operations used in silver and gold to enrich facts with dimensions and to bring together data from different sources.Intermediate
- 6Deduplication and aggregationsDeduplication removes repeated rows, either entirely or with respect to a subset of columns; aggregation summarizes groups of rows into a single value (count, average, distinct count). These are the operations that turn silver into a trustworthy dataset and…Beginner
- 7UDFs and when not to write oneA user-defined function (UDF) is custom logic you register so Spark can call it inside a query, for cases the built-in functions don't cover. The mistake most people coming from pandas or plain Python make is reaching for a UDF as the first option, because…Intermediate
News
No news for this area.