UCDataset in Beta streams Unity Catalog tables straight into PyTorch GPU training

UCDataset is a PyTorch data loader, in Beta on AI Runtime, that reads a Unity Catalog Delta table's files directly from cloud storage and splits the matching rows across all GPUs and loader workers, so no Parquet export or custom sharding code is needed. Jobs read as the identity that runs them and Unity Catalog permissions apply, but in the Beta it cannot read tables that have row filters or column masks. Databricks reports about a 10x faster training time against exported Parquet on a simulated recommendation workload.

Read the original