1/N: Best base model for discrete diffusion LM? Masked diffusion is cheap, but it has weak native self-correction ability. Remasking introduces ad-hoc parameters, making the pipeline hard to tune and generalize. Pure uniform diffusion has a stronger correction potential, but it comes with a quadratic state space over the vocabulary, making it unscalable and much more expensive to train.
We therefore revisit the mask+uniform direction previously explored by GIDD and present Self-Correction Discrete Diffusion (SCDD), which can be viewed as a real multi-step BERT. We exploit the absorbing structure not only for the mask state, but also for uniform token-to-token transitions, making our formulation much simpler than GIDD. We do not rely on ReMask for self-correction; instead, everything is learned during pre-/mid-training to maximize generalization potential. Here is how SCDD compares with GIDD and MDLM.