GitHub repo: https://github.com/karpathy/na... A lot more detailed and technical walkthrough: https://github.com/karpathy/na... Example conversation with the $100, 4-hour nanochat in the WebUI. It's... entertaining :) Larger models (e.g. a 12-hour depth 26 or a 24-hour depth 30) quickly get more

And an example of some of the summary metrics produced by the $100 speedrun in the report card to start. The current code base is a bit over 8000 lines, but I tried to keep them clean and well-commented. Now comes the fun part - of tuning and hillclimbing.

@karpathy Oh nice! Thanks for sharing! Curios, what model architecture is this training/based on?

@rasbt Basically Llama-like, a bit simpler, some influences from modded-nanoGPT. Tried to find a solid baseline for this scale: - dense transformer - rotary embeddings (and no positional embeddings) - QK norm - untied weights for embedding and unembedding - norm after token embedding -

@karpathy Wow thank you for this! Open sourced too?? So will this allow me to train it on data that I can provide it? Like all my notion notes, plus health data, plus other chats from other LLMs? Lime a personal chat bot that understands me?

@rcmisk Good question ty, I think this is not a good repo for that. You should think of micro models maybe more as very young children (kindergarten etc.), they just don't have the raw intelligence of their larger cousins. If you finetune/train it on your own data you'll probably get

@karpathy Super cool! Very well executed! I think a lot of people will learn from it

@Tim_Dettmers Thank you! Notably I didn't yet include model quantization for inference. I have questions :)

@karpathy Thanks, any tentative date for the LLM101n launch? Can't wait. :)

@singh_ilepton LLM101n is currently experiencing a lot of scope creep 😅

@karpathy curious how much did you write the code by hand of it?

@zenitsu_aprntc Good question, it's basically entirely hand-written (with tab autocomplete). I tried to use claude/codex agents a few times but they just didn't work well enough at all and net unhelpful, possibly the repo is too far off the data distribution.

@karpathy this is great! this reminds me of a similar project that also covers pre training to post training from scratch https://github.com/jingyaogong... (not affiliated)

@karpathy Just browsing through the repo during lunch break and have another question if you don't mind: I see you have both AdamW and Muon. What's your experience with Muon, is the hype (e.g., as per Kimi K2) warranted? 😊

@karpathy admirable effort, but isn't the bulk of the cost in training a large model on the compute and in obtaining the data? wouldn't it be more efficient to focus on commoditizing RLHF on large open pretrained models? unless the target audience is organizations with the capital to

@karpathy I strongly respect your effort to hold technology in the realm of a single individual. I assume you're familiar enough with chips you could design basic CPU/GPU, and build basic OS. I think it's a beatiful thing you strive to keep a full stack of civilization in single human form

@karpathy Impressive work on nanochat, @karpathy! Curious about the unhinged side—how does it differ from nanoGPT in terms of creative code choices? Are there any unexpected challenges you faced while building this full-stack wonder? 🤔

@karpathy @soumithchintala Nanochat sounds wild! 🤖 I'm curious, what's the most unexpected challenge you faced while creating it? Also, how does it differ from nanoGPT in terms of functionality or goals? Can't wait to dive in and explore!

@karpathy You mean I've been paying $200 a month for ChatGPT when I could have been training my own ChatGPT for $100? Well fuck me.

@karpathy when the singularity engulfs the last humans who still know what's going on, you will be at the event horizon

@karpathy You might want to switch to data mix as proposed in MobileLM-R1 or NVidia CLIMB. Also, gamify this with clear rules. For example, no extra dependencies and if you beat X, Y, Z benchmark (in good faith) then PR merged and you get in leaderboard.

@karpathy Ah-mazing! The holy trinity: Fast training, straight-forward implementation, and complete implementation 💜 Digging into it now. Published my claude code digging into the new codebase @ https://claude.ai/public/artif... Or in MD form @ https://gist.github.com/Justin...

@karpathy wait, you rewrote the tokenizer in rust? *carefully bookmarks for the weekend* honest question though - for someone trying to understand llm training soup-to-nuts, where would you start in the 8k lines? pretraining? tokenizer? or straight to the rl bits?

@karpathy Is there an application to toggle between deterministic and non-deterministic inference ? I can see this being a local high-level decision maker that can guide low-level processes (control) perhaps

@karpathy The leap from nanoGPT to nanochat sounds exciting! What sparked the shift to tackle a full-stack pipeline this time? Would love to hear about any unexpected learning moments in translating creativity into code!

@karpathy This is awesome. This should become a new benchmark for hardware, where you report the ordered triple: 1) total cost end-to-end in dollars to train 2) total wall clock time in minutes to train 3) total performance on some suite of benchmarks And it’s all easily replicable.

@karpathy If anyone else wants explore it visually here’s an interactive map of the repo

@karpathy This is seriously impressive, Andrej! Moving from nanoGPT's pretraining focus to a full-stack, from-scratch training pipeline culminating in a chat interface in ~8k lines is an incredible feat of engineering and clarity. Huge thanks for pushing the boundaries of what's accessible