@liger_kernel
🥳🥳 Introducing "CoT-ICL Lab" to #ACL2025
A highly flexible (synthetic) data generation framework to study Chain of Thought learning with "In-Context" examples.
TL;DR: This work bridges the gap between theoretical setups and complex NLP settings to study how transformers learn abstract concepts and their underlying causal dependencies. 🧵
Joint work with: @mamhamed, Maziar SanjabiView quoted post on X
We are open sourcing bytecheckpoint and veomni!View quoted post on X
bytecheckpoint is the Bytedance's production checkpointing system for foundation model training, battle-tested with jobs with 10k+ GPUs. Blazing fast save/load, load-time checkpoint auto-resharding for different parallelism across training stages (pretrain/SFT/RL).
veomni is a open source model training framework for llm and multi-modal training. UI-TARS (the SOTA GUI Agent model prior to OpenAI operator's release) is trained with veomni. Developed with modular design, integrated with sequence/expert/zero-optimizer parallelism, offloading optimizations, @liger_kernel. Trainer-free (let user control the training loop) and easy for researcher to hack! The go-to framework for text/multimodal llm pre-training and post-training, from research to production.
Try them today, your feedback is welcome!
Code:
- github.com/ByteDance-Seed…
- github.com/ByteDance-Seed…
Paper:
- NSDI paper: arxiv.org/abs/2407.20143
glows.ai Glows.ai - GPU Cloud Platform for AI GPU computing cloud for AI developers. Launch NVIDIA GPUs in minutes, scale on demand, and cut costs for training and inference. 
