1) Architecture The encoder pipeline is window attention encoder -> down sample -> full attention encoder. This is then fed to a 3BA570M MOE. 1) They use SAM 1 and CLIP 1. Ordinarily, this is a huge VLM red flag. 2) Sparsity ratio is alot smaller here ~19% It supports

The idea of downsampling between the 2 encoders feels very reminiscent of MLA, the core principle being you do the expensive higher dim op with smaller FLOPs (model size and window size), downsample and then do the main operation in this downsampled latent dim

2) Data consists of 30M pages of complex docs + general vision data. There are 2 types of annotations a) Directly use fitz. (This is outdated, the library is now pymupdf) b) Fine-grained annotations use a bunch of models to detect coordinates of text and then OCR within the

For charts, they use HTML and for geometry they use a format from another work. (not gonna say much here since im not familiar with OCR data types) They also have general vision (caption, detection, grounding) data + text only data

Training details arent that interesting, train the encoder, then train the full pipeline with DP, PP The fact that A100s are their devices of choice indicates either 1) this was a project for interns or 2) its old since the last time we see them talk about A100s in their infra

Results. They also show examples for a few use cases beyond traditional image->text OCR - converting document to markdown - converting a graph to a table - Image captioning - BBOX detection

https://github.com/deepseek-ai... DeepSeek-OCR: Contexts Optical Compression Haoran Wei, Yaofeng Sun, Yukun Li

So if you are like me you might be wondering, why is an AGI lab working on OCR. Their main motivation is that they can compress text from documents this way by 20x (more if you downsample images), so i guess instead of sending in long context, you can cheat your way by sending

Might be controversial given everyone hyping it up: I dont think this is anything and I dont think DeepSeek is working/extending this idea anymore, with the latter obviously being speculation The paper says "unlimited context" but unfortunately as anyone who had to reduce font

Now what I do think is useful, is that this entire pipeline can be and likely will be the multimodal architecture if v4 supports image inputs. As mentioned in the second tweet, this fits well into their approach of latent computation.

@nrehiew_ When you run it without thinking you get Deepseek VL 2 errors So guess you're right

@nrehiew_ What's your point?

FlashAttention is probably the ultimate "AI PhD" contribution: > spend years studying > understand things better than everybody else > rewrite low-level code in more intelligent way > make transformers 2-4x faster > essentially a free lunch > now it runs on all of our computers

🧠 How can we equip LLMs with memory that allows them to continually learn new things? In our new paper with @AIatMeta, we show how sparsely finetuning memory layers enables targeted updates for continual learning, w/ minimal interference with existing knowledge. While full

Wanted to get better intuitions for how RL works on LLMs. So I wrote a simple script to teach Nanochat to add 5 digit numbers. I was surprised at how fast it learned. Until I looked at the model's generations and realized that it had just learned to always call the built-in

Tbh I never really got 10+ year timelines. To me they just mean that we need 1 or more breakthroughs and we just assume a decade is enough to find them