Experiments
PreviewWorth evaluating
UCDataset streams Unity Catalog tables into PyTorch on AI Runtime (Beta)
From Databricks release notes (AWS)
UCDataset reads rows from Unity Catalog tables straight into PyTorch training loops on AI Runtime, with no Spark or SQL warehouse. It applies filter expressions and deletion vectors while reading and splits matching rows across GPUs and data loader workers. It is in Beta.
What to re-read
More from Experiments
- PreviewUCDataset in Beta streams Unity Catalog tables straight into PyTorch GPU training
- PreviewMLflow managed warehouse lets you view Unity Catalog traces without picking a warehouse
- ReleaseAI Runtime, serverless GPU training on Databricks, announced as generally available
- ReleaseMLflow run Logs tab for AI Runtime gains search, filters and download