MatthewBerman

1/ Google Research unveils new paper: "Titans: Learning to Memorize at Test Time"

It introduces human-like memory structures to overcome the limits of Transformers, with one "SURPRISING" feature.

Here's why this is huge for AI. 🧵👇 

2/ The Problem:

Transformers, the backbone of most AI today, struggle with long-term memory due to quadratic memory complexity.

Basically, there's a big penalty for long context windows!

Titans aims to solve this with massive scalability. 

3/ What Makes Titans Different?

Inspired by human memory, Titans integrate:

• Short-term memory (real-time processing)
• Long-term memory (retaining key past information)
• Persistent memory (task-specific baked-in knowledge)

This modular approach mimics how the brain works.

4/ Game-Changer: Memory at Test Time

Titans can learn and adapt during inference (test time), unlike Transformers, which rely on pre-training.

This means:

• Dynamic updating of memory during real-time use.
• Better generalization and contextual understanding. 

5/ The "Surprise" Mechanism:

Humans remember surprising events better.

Titans use a "surprise" metric to prioritize what to memorize and forget.

• Adaptive Forgetting ensures efficiency.
• Surprising inputs create stronger memory retention.

This leads to smarter, leaner models.

6/ Three Architectural Variants:

Titans offer flexible implementations based on use cases:

• Memory as Context (MAC): Best for tasks needing detailed historical context.
• Memory as Gated (MAG): Balances short- and long-term memory.
• Memory as Layer (MAL): Most efficient, slightly less powerful.

Trade-offs for every need!

7/ Performance:

Titans outperform Transformers and other models in:

• Language modeling.
• Common-sense reasoning.
• Needle-in-a-haystack tasks (retrieving data in vast contexts).
• DNA modeling & time-series forecasting.

They maintain high accuracy even with millions of tokens.

8/ Why This Matters:

• Massive Context: No more limits on how much info models can process.
• Real-Time Adaptation: Models learn dynamically, like humans.
• Scalability: Opens the door for AI in genomics, long video understanding, and reasoning across massive datasets. 

9/ Key Innovations:

• Surprise-based memory prioritization.
• Efficient, scalable architectures with adaptive forgetting.
• Parallelizable training algorithms for better hardware utilization.

Titans bridges the gap between AI and human-like reasoning. 

10/ What’s Next?

With Titans, we could see breakthroughs in AI applications that demand massive context, from personalized healthcare to real-time video analytics.

Read the paper here: https://arxiv.org/abs/2501.00663

Check out my video breakdown here: https://www.youtube.com/watch?v=x8jFFhCLDJY

What do you think of Titans?

Let’s discuss. 💬

What if AI models could figure out answers to your questions BEFORE you even ask them? 🤔

An incredible new research paper introduces "Sleep-Time Compute," allowing models to think while they aren't being used.

Here's what you need to know 🧵👇 

Currently, most modern LLMs rely on "Test-Time Compute." (reasoning models like GPT-o3)

This means the model processes ALL the context & the query right when you ask for a response.

Problem: This can lead to high latency (waiting minutes) and high costs (tens of dollars per query) for difficult problems.

A key drawback: standard test-time compute often assumes problems are 'stateless'.

The model needs the entire context & query together every single time for processing.

For 'stateful' applications (like document Q&A, coding assistants, conversational agents) where the context is persistent, this forces models to constantly recompute redundant information.

The solution? "Sleep-Time Compute."

This lets the LLM "think" or pre-process information about the context OFFLINE, when the user ISN'T actively querying.

It's like your brain processing info overnight! 

Key Benefit: SIGNIFICANT Cost Reduction!

By moving redundant computation to idle "sleep" time, you can achieve similar accuracy to standard test-time compute with potentially 5x LESS expensive test-time tokens!

More efficient GPU usage = lower bills. 

They tested this on various models (GPT-4o, Claude, DeepSeek) &amp; benchmarks.

Results show Sleep-Time Compute produces a Pareto improvement in the accuracy vs. test-time compute tradeoff.

Meaning better accuracy OR lower cost for the same result. 

Scaling up the amount of sleep-time compute further improved accuracy by up to 13-18% on different reasoning tasks.

This suggests offloading thinking to idle time allows models to perform better when it counts (at test time). 

Sleep-Time Compute also consistently outperformed "parallel sampling" (another technique for low-latency inference) at the same test-time budget.

This highlights its effectiveness as a way to scale inference time efficiency. 

It's most effective when the questions you anticipate asking are somewhat "predictable" from the context.

Pre-processing is useless if you asked questions that were unrelated to the context. 

Cost amortization

The cost-saving power of Sleep-Time Compute really shines in applications where users ask MULTIPLE questions about the SAME piece of context.

The offline "thinking" about that context can be shared, reducing the average cost per query by up to 2.5x! 

Here's the paper:
https://arxiv.org/abs/2504.13171

Shoutout to @letta_ai (founded by memGPT creators!) for the paper! 🙏

And my full breakdown video:
https://www.youtube.com/watch?v=FGRxw3ACIkw