https://x.com/balesni/status/1... Why does this matter? If models mostly relied on memorization, general two-hop reasoning would require externalized chain-of-thought — amazing for our ability to monitor LLM agents for safety by looking at their CoT.

The mystery: Models often succeed at two-hop questions about real facts. But is it genuine compositional reasoning? Or just memorization from seeing "spouse of performer of Imagine" near "Yoko Ono" in training data? It is hard to tell! Previous work had confounds.

https://x.com/soheeyang_/statu... @soheeyang_ et al. tried filtering pretraining data — only asking questions where the two facts never appeared in the same document. Models failed, except when the bridge entity was a country. Why this exception? Maybe the filtering missed something?

https://x.com/feng_jiahai/stat... Then @feng_jiahai et al. had a great idea: teach models "Kevin was born in Paris" then ask "What language do they speak where Kevin was born?" Zero chance of memorization. They found success! But only tested cities as bridge entities. Could models do this

We expanded this semi-synthetic setup to 17 different entity types — programming languages, artworks, chemical elements, video games. The question: is two-hop reasoning limited to specific categories, or is it a general capability? Result: Models succeed across MANY categories.

This proves LLMs can do genuine two-hop reasoning internally. Not just for cities (Feng et al.) or countries (Yang et al.). But here's the weird part: originally we tried teaching models two fictional facts to combine, and they completely failed. 0% accuracy and chance-level

The puzzle: * Synthetic + real fact: ✓ works * Synthetic + synthetic: ✗ fails * Synthetic facts in same training document or in-context: ✓ works

This provides a cautionary tale for studying LLM latent reasoning. Success on real-world prompts ≠ robust latent reasoning; it might reflect co-occurrence in pretraining. Failure on synthetic two-hop ≠ inability to reason; synthetically learned facts can differ natural

https://x.com/emmons_scott/sta... Our honest takeaway for AI oversight: move past multihop QA as a toy model. What matters is whether monitors catch misbehavior in practice. The field should move toward end-to-end evals where an agent does tasks while another model watches its CoT.

Paper: http://arxiv.org/abs/2411.1635... with @tomekkorbak and @OwainEvans_UK Code: http://github.com/mbalesni/syn... An earlier version of this paper was released in November, but since then we’ve updated many experiments and substantially revised our conclusions.

@balesni The setup and findings seem to be quite aligned with our work on Grokked Transformers led by @BoshiWang2 @osunlp. Glad to see larger models like 8B are used in the paper! https://x.com/hhsun1/status/17...

@balesni Well done !