# LLM Training ## General - The Llama 3 Herd of Models [[paper]](https://arxiv.org/abs/2407.21783) - TorchScale - A Library for Transformers at (Any) Scale [[GitHub]](https://github.com/microsoft/torchscale) - DLRover: An Automatic Distributed Deep Learning System [[GitHub]](https://github.com/intelligent-machine-learning/dlrover) ## 2024 - FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision [[paper]](https://tridao.me/blog/2024/flash3/) - MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs [[paper]](https://arxiv.org/abs/2402.15627) - ByteCheckpoint: A Unified Checkpointing System for LLM Development [[paper]](https://arxiv.org/abs/2407.20143)