Product area
Data Quality
Keeping bad rows out of silver and gold: pipeline expectations, Delta constraints, the DQX framework, and the monitoring that tells you when quality drifts.
Concepts
- 1Data quality on Databricks, layer by layer"Data quality" on Databricks is not a single product you turn on. It is four different mechanisms, each watching a different moment in the life of a row:Intermediate
- 2Data profiling and anomaly detectionUnity Catalog groups two features under data quality monitoring, and they answer opposite questions.Intermediate
- 3DQX: data quality checks for PySparkDQX is a data quality framework from Databricks Labs. You describe rules, it applies them to a PySpark DataFrame or a Unity Catalog table, and it hands back the same rows plus two columns explaining what failed: errors and warnings. From there you either keep…Intermediate
Also covered here
- Data quality: expectations and constraintsDatabricks offers two mechanisms for saying "this data must satisfy a rule":
- Monitoring a deployed modelMonitoring a deployed model on Databricks is not a separate product. It is the same data profiling feature that watches tables, described in data quality monitoring, pointed at a table whose rows happen to be model requests. You choose the InferenceLog…
News
Resources
Nothing of that kind for this area. Try the full list.