we built DeepWiki, a free encyclopedia of all GitHub repos some numbers: - 30k repos already indexed - processed 4 billion+ lines of code - the indexing alone cost $300k+ in compute spend
we let LLMs scan the entire codebase: indexing a single repo costs $12 on average – but we decided to make it entirely free, no sign-up required for open-source we indexed the top 30k repos on GitHub and processed over 100 billion tokens
high-level systems & structure: models are very good at understanding code locally. the hard part is understanding the global structure of the codebase. DeepWiki breaks down a codebase into a hierarchy of high-level systems and then generates a wiki page for every system
one interesting signal that we're able to leverage is the commit history. who tends to touch which files together? you can represent this as a graph. lots of interesting patterns!
try now at http://deepwiki.com!