Hot take: Git was the wrong abstraction for 90% of ML data.
Checkpoints, optimizer states, training logs, agent traces - none of this needs version control. It needs fast, cheap, mutable storage.
So we built Buckets. S3-like storage on the @huggingface Hub with Xet dedup and zero egress.
Train in a bucket. Publish to a repo. One platform. 🤗🤗🤗
Shit, we did everything wrong then 😂
Git works well for code and small artifacts, but it becomes painful once you are dealing with multi gigabyte checkpoints, optimizer states, and repeated training outputs. For most ML workflows, content addressable object storage is a much more natural abstraction than pretending
DVC did close the gap a little bit. docs.lakefs.io/v1.80/ buckets for data, logs etc. metadata checkpoints in git.Join the conversation
