datasets library: 100x More Efficient Streaming Datasets

AI Impact Summary

The datasets library has significantly improved streaming performance, achieving 100x fewer requests and a 10x faster data resolution, primarily through persistent data file caching and optimized resolution logic. This addresses a key bottleneck in training large models, particularly with datasets like FineVisionMax, reducing startup requests and improving overall throughput. This change allows for training on multi-TB datasets without the complexities of traditional data downloads and avoids common issues like 429 errors or disk space limitations.