sft loss is wrong when using grad accumulation 🥶
@QGallouedec aaaa. heres my run: https://huggingface.co/spaces/... maybe somebody could build a really nice library to make this post training stuff super easy.
@ben_burtenshaw It would be so nice
@QGallouedec Don’t tweet. Send a PR
@EmbDim512Layer7 No worry, pr is coming, who do you think I am 😏
@QGallouedec Oh no my PTSD
@TheZachMueller I forgot the trigger warning 🫢
@QGallouedec Andrejs response: It's not a bug, it's a deliberate choice. Doing "the right thing" adds a bunch of added bloat/complexity synchronizing the number. We're only seeing/logging the loss at a single rank but that's ok as just an estimate. The training is all correct otherwise.
@1Sarim no, you're referring to the logging "issue" (which I agree is not an issue). See https://x.com/QGallouedec/stat... Here the loss scaling is wrong. See https://github.com/karpathy/na...
@QGallouedec I saw this and thought it was a great catch, so I went ahead and opened a PR for it. (https://github.com/karpathy/na... Didn't realize you were on it too. Sorry if I jumped the gun! I'm more than happy to collaborate on this one, or if you already have a PR in progress, I can close mine.
@JinguKang_ nice work!
@QGallouedec Is it similar to unsloths GA recent issue fix?
@QGallouedec it's called we do a little submit a PR
@QGallouedec oh no. i hope someone fixes it 🙏
@QGallouedec Is that issue in HF Trainer completely fixed now?
@QGallouedec @grok what does he mean by that