Product area
Data Ingestion
Getting data into the lakehouse: batch, streaming, and incremental patterns; COPY INTO, Auto Loader, Lakeflow Connect, JDBC/REST, semi-structured data.
Concepts
- 1Ingestion patterns: batch, streaming, incrementalIngestion is the first step of every pipeline: bringing data from an external source (files, databases, SaaS applications, message queues) into a Delta table governed by Unity Catalog, usually in the bronze layer (see medallion architecture). Databricks…Beginner
- 2COPY INTOCOPY INTO is a SQL command that reads files from cloud object storage (S3, ADLS, GCS) or from a Unity Catalog volume and appends them to a Delta table. Its key property: it is idempotent. It keeps track of the files it has already loaded and, when you rerun…Intermediate
- 3Auto LoaderAuto Loader is a Structured Streaming source, identified by the cloudFiles format, that watches a directory in S3, ADLS, GCS, or a Unity Catalog volume and processes files as they arrive. It reads JSON, CSV, XML, Parquet, Avro, ORC, text, and binary files. It…Intermediate
- 4Lakeflow Connect: managed connectorsLakeflow Connect is the umbrella name Databricks uses for every way of ingesting data. This page covers the managed connectors: prebuilt pipelines for specific enterprise sources, where Databricks handles authentication, incremental reads, schema evolution…Intermediate
- 5Ingesting from JDBC and REST APIs in notebooksNot every source has a managed connector. A legacy database, an internal API, a niche SaaS service: in these cases the notebook becomes the connector. Two tools:Intermediate
- 6Semi-structured data: JSON, nested data, VARIANTSemi-structured data has a structure, but not a fixed one: an event JSON has nested fields, arrays of variable length, and keys that only show up sometimes. Unstructured data (PDFs, images, audio) has no tabular structure at all. Databricks handles them like…Intermediate
- 7Zerobus IngestZerobus Ingest is a write API. An application calls it and the records land in a Unity Catalog Delta table, queryable within seconds. There is no topic, no connector, no landing zone and no file to pick up afterwards.Intermediate
News
No news for this area.