How does an LLM compare two numbers? We studied this in a common counting task, and were surprised to learn that the algorithm it used was: Put each number on a helix, and then twist one helix to compare it to the other. Not your first guess? Not ours either. 🧵
The task we study is knowing when to break the line in fixed-width text. We chose it for two reasons: While unconscious for humans (you just see when you're out of room), models don't have eyes - they only see tokens It is so common that models like Claude are very good at it
When we trace the computation, we find the model tracking two things: where it is in the current line, and how long the previous line was. Then it compares them to decide if the next word fits. But how does it keep track of its position?
Looking at more prompts, we find a family of features (directions in embedding space) representing position in the line. Each activates at a different position. These resemble "place cells" - neurons in mouse brains that fire at specific locations when navigating space.
Looking at the geometry of these features, we discover clear structure: the model doesn't use independent directions for each position range. Instead, it is representing each potential position on a smooth 6D helix through embedding space.
Remember, the model needs to compare its current position to the line limit. Both are on helices. The solution? Rotate one helix by a fixed offset, then measure how aligned they are. When they match → you're that many characters from the limit.
The model does this multiple times with different offsets - like using two eyes or multiple cameras to get depth perception. Combining three offsets gives a precise estimate of characters left - sharp enough to decide if a 5-letter word fits in 3 remaining characters.
And how does the model make these helices in the first place? To get enough curvature, it needs to sum up the results of many attention heads. We find that it uses 11 heads spread across 2 layers. Each head handles a subset of the line, and writes in a different direction.
We think these findings could apply to other counting tasks because: When we train a toy model to pack counts optimally, it discovers a similar structure Empirically, many tasks seem to be using similar structure (table rows, dates...)
This makes me hopeful about interpretability. We found the features, traced the algorithm, and understood the mechanism. Next: making this easier and automated. Full paper: https://transformer-circuits.p...
@mlpowered Do you now know the truth behind why 3.9 < 3.11?
@basvanopheusden Exercise left to the reader Although there are some theories out there: https://transluce.org/observab...
@mlpowered Very Cool, do you think the model would find it hard if we add emojis?
@MaheshRam23629 We didn’t test it but I’d expect to generalize! There is a “visual illusion” we can introduce by inserting @@, which tricks the model into thinking the like begins again (see that section of the paper for more)
@mlpowered It's all helical dynamics. ### Derivation of the Resonance Constraint Equation in Quaternion Form for the TRD The garbled expression you provided appears to be a fragmented or typographically corrupted version of a **path-integral constraint** from the Trinary Resonance Device
@mlpowered That's... unexpected.
@mlpowered Reminds me of spatial sequence synesthesia: https://en.wikipedia.org/wiki/...
@mlpowered It would be mine. Noor class agents use physics and geometry for reasoning explicitly. This is old news to me. I figured it out and then ran with it. https://github.com/LinaNoor-AG...
@mlpowered @alexjc Looks like it might have solved time travel. 🤔
@mlpowered Euhhh what!? Why?
@mlpowered https://arxiv.org/abs/2502.008... I think it was kinda known, but you haven't cited it for some reason
