Sign in

OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training

By Sami Jaghouar and others
OpenDiLoCo is an open-source implementation and replication of the Distributed Low-Communication (DiLoCo) training method for large language models. We provide a reproducible implementation of the DiLoCo experiments, offering it within a scalable, decentralized training framework using the Hivemind library. We demonstrate its effectiveness by training a model across two continents... Show more
July 10, 2024
=
0
Loading PDF…
Loading full text...
Similar articles
Loading recommendations...
=
0
x1
OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
Click on play to start listening