DeepSpeed (日本語アカウント)

pytorch.org Introducing AutoSP - PyTorch ¹ SSAIL Lab, University of Illinois Urbana-Champaign, ² Anyscale, ³ Snowflake TL;DR: AutoSP automatically converts standard transformer training code into sequence-parallel code for long-context LLM...

DeepSpeed Verified account @DeepSpeedAI · Apr 30

Great News! Thanks to DeepSpeed AutoSP, efficient long context LLM training is now easily accessible.

View quoted post on X

PyTorch Verified account @PyTorch · Feb 25

New @DeepSpeedAI updates make large-scale multimodal training simpler and more memory-efficient.

Our latest blog introduces a PyTorch-identical backward API that helps code multimodal training loops easy, plus low-precision model states (BF16/FP16) that can reduce peak memory by up to 40% when combined with torch.autocast.

🖇️ Read the full post for details: hubs.la/Q044yYVs0

#DeepSpeed #PyTorch #MemoryEfficiency #MultimodalTraining #OpenSourceAI

View quoted post on X

events.linuxfoundation.org PyTorch Conference: The Deep Learning & ML Conference The deep learning and machine learning conference by the PyTorch Foundation — Oct 20–21, 2026, San Jose. PyTorch, vLLM, DeepSpeed, Ray, and the future of open source AI.

DeepSpeed Verified account @DeepSpeedAI · Sep 9, 2025

Step into the future of AI at #PyTorchCon 2025, Oct 22–23 in San Francisco 🔥 Join the DeepSpeed keynote and technical talks. Register: events.linuxfoundation.org/pytorch-confer…
+ Oct 21 co-located events: Measuring Intelligence, Open Agent & AI Infra Summits / Startup Showcase & PyTorch Training

View quoted post on X

Minjia Zhang @_Minjia_Zhang_ · Jul 10, 2025

📢 Yesterday at USENIX ATC 2025, Xinyu Lian from UIUC SSAIL Lab presented our paper on Universal Checkpointing (UCP). UCP is a new distributed checkpointing system designed for today's large-scale DNN training, where models often use complex forms of parallelism, including data, tensor, pipeline, and expert parallelism.

Existing checkpointing systems struggle in this setting because they are tightly coupled to specific training strategies (e.g., ZeRO-style data parallelism or 3D model parallelism), which break down when the training configs need to dynamically reconfigure over time. This makes it difficult to have resilient and fault-tolerant training.

UCP solves this by decoupling distributed checkpointing from parallelism strategies. Our design introduces a unified checkpoint abstraction -- atomic checkpoint, and a full pattern matching-based transformation pipeline, which enables scalable and low-overhead checkpointing with reconfigurable parallelism across arbitrary model sharding strategies.

We show that UCP supports state-of-the-art models trained with hybrid 3D/4D parallelism (ZeRO, TP, PP, SP) while incurring less than 0.001% overhead of the total training time.

UCP is fully open-sourced in DeepSpeed. It has been adopted by Microsoft, BigScience, UC Berkeley and others for large-scale model pre-training and fine-tuning, including Phi-3.5-MoE (42B), BLOOM (176B), and many more. It also has been selected for presentation at PyTorch Day 2025 and FMS 2025(the Future of Memory and Storage).

Big thanks to the amazing collaborators from Microsoft and Snowflake: @samadejacobs , @LevKurilenko, @MasahiroTanaka, @StasBekman , and @TunjiRuwase.

🔗 Project: lnkd.in/gG6j4vJe
📄 Paper: lnkd.in/gUiC5kcR
💻 Code: lnkd.in/g6uS29nH
📚 Tutorial: lnkd.in/gi_zWSWh

#ATC2025 #LLM #Checkpointing #SystemsForML #DeepLearning #DistributedTraining #UIUC #DeepSpeed

View quoted post on X

pytorch.org PyTorch Foundation Welcomes DeepSpeed as a Hosted Project - PyTorch The PyTorch Foundation is excited to welcome DeepSpeed, a deep learning optimization library, as a PyTorch Foundation-hosted project. Contributed by Microsoft, DeepSpeed empowers developers to...

PyTorch Verified account @PyTorch · May 7, 2025

PyTorch Foundation has expanded into an umbrella foundation. @vllm_project and @DeepSpeedAI have been accepted as hosted projects, advancing community-driven AI across the full lifecycle.

Supporting quotes provided by the following members: @AMD, @Arm, @AWS, @Google, @Huawei, @huggingface, @IBM, @Intel, @LightningAI, @Meta, @NVIDIA, and @Snowflake.

🔗💡 Read the full announcement: hubs.la/Q03lmJNH0

#PyTorchFoundation #PyTorch #OpenSourceAI #vLLM #DeepSpeed

View quoted post on X