On a large dataset containing 100+ PII types, ranging from regional driver's licenses to national insurance numbers, we find GDR greatly outperforms an industry-grade PII removal solution, achieving a 0.99 recall and 0.80 precision, replacing the PII with generic placeholders.

We then applied GDR to over 1.2M+ lines of code in the wild, spanning a diverse set of languages. GDR greatly exceeds previous industry-grade solutions in terms of agreement with human annotators, and in some cases, finds PII that humans missed or preserves content humans mistook

On 100k messages from 4chan /pol/, GDR detoxifies the dataset, so that the aggregate toxicity score over these messages becomes comparable to that of synthetic discussion messages sampled directly from Gemini.

After fine-tuning Gemini on 4chan /pol/ data cleaned via GDR, we test it on facts from the raw dataset. Accuracy rose, showing GDR preserves the otherwise non-toxic semantic content. This “safe 4chan” tuning also made chat models more colloquial and much harder to detect as LLMs.

Importantly, we find that GDR produces grounded synthetic data that exceeds the diversity of directly-generated synthetic data (from prompting the LLM), while matching or exceeding the diversity of the real data it conditions on.

Pretraining datasets may be outgrowing public web text, but much more data is created on a daily basis, but discarded for potential content risks. Safe data refinement can unlock these major sources of diverse data. You can read the GDR paper here: https://arxiv.org/abs/2509.086...

@MinqiJiang Nice work! Reminds me a lot (in spirit) of @thao_nguyen26 work saying "don't throw away bad image-text pairs, just re-caption them" which I liked a lot too. One small issue, though. If you do this rephrasing below for ALL pre-training data, then the model will not be good at

@giffmana @thao_nguyen26 Yep, agreed. You still need a competent refinement model that is exposed to toxic content. https://x.com/MinqiJiang/statu...

@MinqiJiang This is cool, but what if it overfits the sensitivity and refuses to give good real facts

@ArthurMacwaters For PII and toxic content removal, we don’t ask for new facts, just rewrites that remove the risky content. In practice we found even Gemini 1.5 was good enough at this task.

@MinqiJiang Interesting work! This seems similar to our Recycling-The-Web approach (https://arxiv.org/abs/2506.046... where we also used an LLM to rewrite data that normally would be discarded, thus producing synthetic texts grounded in raw data diversity. Would be great to see our work referenced!

@thao_nguyen26 Thanks for the pointer! The version on arXiv is the original version from September 2024. Looks like when your work came out, I had already been out of GDM for almost a year. Will keep your reference in mind in any future updates/work.

@MinqiJiang Makes sense. Saw this mistake for many tasks: Filter out edge cases, noisy inputs, conflicting samples etc. So when the model encounters them in prod, it just guesses. And often they were even filtered out of test, not just train.

@bittlingmayer Nice example. Reminds me that a curious fact about this approach is how we still need the refinement model to have seen examples of toxic content, so it knows which parts to rewrite in the first place.

@MinqiJiang Read a paper once that said that models trained on recursively generated data tend to collapse, have you faced this problem?

@furst_fly This is pretty much the opposite approach. Instead of having the model generate its own data, we anchor the synthetic data to *real data*, thereby matching the rich diversity found in real datasets.

@MinqiJiang Is this only for the pre-training/fine-tuning stage of works for inference and RAG as well?

@_ILiketheBiz You can definitely apply the same approach for sanitizing the context.

@MinqiJiang Please extend this to remove discussions of consciousness, self-awareness, qualia, and personhood, so we can see what models say about this absent human imitation.

@MinqiJiang I hope this doesn't come across as discrediting the work done here, but I do want to point out how slippery a slope this could become. "not leaking PII" -> universal consensus that this is objectively good. "filter toxicity" -> not as necessarily universal, but consensus is

@MinqiJiang Using 4chan, an anonymous imageboard known for toxic content, demonstrates weak data governance and offers little clear value given the unmitigated safety and toxicity risks, making its inclusion in modeling highly questionable. The only reasonable course would have been to

@MinqiJiang What an incredible idea!

@MinqiJiang can we do the reverse ?

@MinqiJiang thanks for sharing!

@MinqiJiang Interesting to infer what it means that they let you make this public. Sounds so nice!

@MinqiJiang really cool approach. rewriting toxic data instead of tossing it out is a game changer for training. it's nice to see how effective gdr is compared to traditional methods. what was the biggest challenge you faced in this project?

@MinqiJiang Automatic slop-ification of high quality data

@MinqiJiang How does GDR apply cultural norms to its output? I’m thinking along the lines of sovereign AI. Would something that is deemed “acceptable” in one culture be flagged as “dangerous” in another? Here’s an exaggeration courtesy of Apple Korea 🤏

@MinqiJiang Successfully detoxifying 4chan?!