Silas Alberti

we built DeepWiki, a free encyclopedia of all GitHub repos

some numbers:
- 30k repos already indexed
- processed 4 billion+ lines of code
- the indexing alone cost $300k+ in compute spend 

we let LLMs scan the entire codebase:

indexing a single repo costs $12 on average – but we decided to make it entirely free, no sign-up required for open-source

we indexed the top 30k repos on GitHub and processed over 100 billion tokens

high-level systems &amp; structure:

models are very good at understanding code locally. the hard part is understanding the global structure of the codebase. 

DeepWiki breaks down a codebase into a hierarchy of high-level systems and then generates a wiki page for every system

one interesting signal that we're able to leverage is the commit history. 

who tends to touch which files together? you can represent this as a graph. lots of interesting patterns!