Product area
Data Quality
Keeping bad rows out of silver and gold: pipeline expectations, Delta constraints, the DQX framework, and the monitoring that tells you when quality drifts.
Concepts
- 1Data quality on Databricks, layer by layer"Data quality" on Databricks is not a single product you turn on. It is four different mechanisms, each watching a different moment in the life of a row:Intermediate
- 2Data profiling and anomaly detectionUnity Catalog groups two features under data quality monitoring, and they answer opposite questions.Intermediate
- 3DQX: data quality checks for PySparkDQX is a data quality framework from Databricks Labs. You describe rules, it applies them to a PySpark DataFrame or a Unity Catalog table, and it hands back the same rows plus two columns explaining what failed: errors and warnings. From there you either keep…Intermediate
News
No news for this area.