Introducing Zephon: an open-source data loader for text and multimodal AI training that makes online data loading fast & deterministic (even if the number of GPUs changes across training runs!).
Blog: datologyai.com/blog/zephon
Great work by @MaxiBoether and @datologyai.
Blog: datologyai.com/blog/zephon
Great work by @MaxiBoether and @datologyai.
www.datologyai.com Zephon: Fast, Flexible, Elastically Deterministic Dataloader Fast, flexible, elastically deterministic data loading for reliable large-scale AI model training. New from DatologyAI: Zephon, an open-source data loader for text and multimodal training.
Resume a run on a different number of GPUs and most loaders quietly change the data. In our tests, GPU count alone moved eval scores by up to 0.82 points, about 4x the gap used to choose between data curation methods.
tl;dr: same training batches on 8, 16, 32 or 64 GPUs, even while tokenizing, packing and mixing on the fly.View quoted post on X
