Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training

Tarun Suresh* Stanford University

Pranshu Chaturvedi* Stanford University

Hangoo Kang* Stanford University

Parth Shroff Stanford University

Ishan S. Khare Stanford University

Hermann Kumbong Stanford University

Azalia Mirhoseini Stanford University

Context-Sharded Block Parallelism (CSBP) accelerates long-context training for block diffusion language models and diffusion-based speculative decoding while preserving the training objective and gradients.