Okara

Andrej Karpathy recently dropped a 3-hour masterclass on how ChatGPT really works.

It's one of the best lectures I've seen on LLMs.

Here are my key takeaways 🧵

1/ The Training Pipeline

LLMs are built through a multi-stage process, with two major stages:

1. Pre-training (learning from internet data)
2. Post-training (learning from human examples)

2/ Pre-training (Building a Base Model)

Step 1: Data Collection
→ The first step in training an LLM is downloading and processing the internet.
→ companies like OpenAI, Google, and Anthropic gather massive amounts of publicly available text from sources like Wikipedia, Common

3/ Step 2: Tokenization 

LLMs don’t process raw text directly. Instead, they convert raw texts into a sequence of tokens 

It uses byte-pair encoding to create efficient token representations

4/ Step 3: Training the Neural Network

The model is then trained using next-word prediction: it takes a sequence of tokens as input and predicts the most likely next token.

Training involves:
→ Random initialization - gradually improving predictions by adjusting weights.
→

5/ Step 4: Inference

Once trained, the model can generate text by predicting tokens step by step

Text Generation Process
→ Autoregressive Prediction: The model generates text one token at a time.
→ Probability-Based Sampling: Multiple valid outputs exist, so responses may

6/ Post-training

After pre-training, the model is still just an "Internet text generator." It can generate plausible-sounding text, but it’s not yet an 'assistant.' 

To make it useful, it goes through post-training

7/ Post-training techniques:

→ Supervised fine-tuning: Human labelers provide curated examples of high-quality responses.
→ Reinforcement Learning with Human Feedback (RLHF): Models are fine-tuned using human preferences to align responses with user expectations.
→ Safety

8/ Limitations of LLMs:

→ Hallucinations – They make up plausible-sounding but false information.
→ Context Limits – GPT models can “remember” only a certain number of tokens at once.
→ Biases – If training data has biases, the model will reflect them.
→ Lack of True

9/ How to get better responses from LLMs

→ Be specific – Vague questions lead to vague answers.
→ Use role-playing – “Act as an AI expert and explain X.”
→ Provide examples – Show the structure you want.
→ Break down complex queries – Step-by-step instructions work best.
→

10/ What's next for LLMs

→ Longer memory – Some models can process 100K+ tokens at once.
→ Cheaper, faster models – Smaller, more efficient LLMs are emerging.
→ Multimodal AI – LLMs that understand text, images, and video.
→ Better safety & alignment – Reducing bias & halluci

11/ If you’re curious to learn more, I highly recommend watching the full video. 

@karpathy does an excellent job breaking down how LLMs like ChatGPT works

12/ Want a visual summary?

Karpathy shared the exact file he created. 

1. Download this file: 
2. Visit Excalidraw .com
3. Import the file &amp; explore the full lecture visually!

That's a wrap ✨ Hope you enjoyed it   

If you found this thread valuable:  
1. Follow @0xmetaschool for more   
2. Retweet the first tweet so more people can see it