Product area
Workspace
The Databricks workspace: platform architecture, notebooks, files, Git folders, bundles, and the development workflow.
Concepts
- 1Architecture of the Data Intelligence PlatformThe Data Intelligence Platform is the name Databricks gives to the set of services it offers on top of a public cloud (AWS, Azure, GCP). The underlying idea is the lakehouse: data stays in cheap, open object storage, but on top of it you get transactions…Beginner
- 2NotebooksA notebook is the default place to write and run code in the workspace: an ordered list of cells, each holding either code or formatted text, that you execute one at a time or all together against attached compute. One notebook can carry Python, SQL, Scala…Beginner
- 3Workspace files and volumesFour different places end up holding "files" (as opposed to tables) on Databricks, and it's easy to reach for the wrong one:Beginner
- 4Secrets and credentialsA secret is a key-value pair stored outside your notebooks and job definitions, so a database password or an API key never appears as a literal string in code that gets committed, shared, or logged. Secrets live inside a secret scope, a named container you…Intermediate
- 5Git folders: branches, commits, pull requestsA Git folder is a folder in the workspace that is also a clone of a remote Git repository. Inside it you work as in any other folder (notebooks, .py, .sql, YAML files), but on top of that you get a visual Git client: branch, commit, push, pull, merge, rebase…Beginner
- 6The CLI and the SDKsThe Databricks CLI is a single binary that turns every workspace and account operation into a terminal command; the Python SDK (databricks-sdk) exposes the same operations as a typed WorkspaceClient object for use inside scripts; Databricks Connect goes one…Intermediate
- 7Declarative Automation Bundles and the Databricks CLIA bundle is a project that keeps, in a single Git-versioned folder, both the code (notebooks, Python files, SQL, wheels) and the declarative definition of the Databricks resources that run it: Lakeflow Jobs, Lakeflow pipelines, dashboards, MLflow experiments…Intermediate
- 8Bundles: variables, targets, and per-environment overridesIn a bundle (see bundles overview), variables are named values that the YAML references with ${var.name}; targets are the deployment environments (dev, test, prod), each with its own workspace, its own mode, and the ability to override variables and pieces of…Intermediate
- 9Bundles in a CI/CD pipelineThis is a Declarative Automation Bundle, the project described in bundles overview, driven by a build server instead of by a person at a terminal. Databricks documents a four-stage flow for it: compile and test the code, upload the compiled file under a…Intermediate
- 10Cost attribution and budgetsCost attribution is the practice of making every DBU answerable to a team, a project or a cost centre. On Databricks it is three mechanisms, and they are not interchangeable. Tags put a key:value label on a compute resource, and that label follows the usage…Intermediate
News
No news for this area.