> LLaDA, a diffusion model > forward data masking, and a reverse process .. overall trained via pretraining + SFT > competitive with strong LLAMA3 8b in ICL, and good Instrcut following after SFT > interesting result that it does better against reversal curse against GPT4o

> from the graph looks like it has a better overall non-spiky characterstics.. this is a 8B model > but zeroshot/fewshot defintely signals that nothing can be said about multi-turn > to me it appears that the methods which can't give multi-turn performance via diffusion are just

> what a lucid and clear view of the space.. (ARM = autoregressive) > capabilities of LLMs are result of generative principles not as per say autoregressive modeling (ARM) > toh the limitations of LLMs are more reflective of their autoregression (trivial yeah) > 'While ARMs

> uses masked diffusion model > discrete random masking process and trains a mask predictor to approximate a reverse process (for example, in normal diffusion DDIM, you train to predict noise, and reverse process is needed for continuous refinement of predictions) > approach

> unlike autoregression => it defines the model distribution thru a forward and backward process (typical diffusion) > imagine a process of learning...you can't like learn anything in 'jumps'.. so you incrementally learn stuff... same way here the forward process slowly starts

lemme try to put it more simply prompt = "This is a diffusion process" response = "ldkfjlsfdkgjsdfjglsdkfjgjldfjglsdkfjgsfl" backward process = "idk<MASK>s<MASK>" now again, it will retry from the representation it got from backward process, until it can fill all the masked

its a interesting blend imo, not technically going away with transformers as per se (since mask prediction is still doing that)... seems to me they are simulating a diffusion process over transformers architecture