there's basically just one guy that is well known for this kind of engineering at all in public who isn't under some kind of NDA. what the fuck? this is seriously bad

@kalomaze i don't know where this notion that training is somehow harder than inference comes from because its... sort of not true? everything is fixed shape, no caching, no nothing.

@mike64_t think of it this way; if someone wants to get good at real world inference, they can fork vllm & stare at triton that was written specifically to maximize batched throughput. if someone wants to do training optimization... they basically just have Tri Dao codebases for ref lmfao

@kalomaze The current state of ML is basically this, relying on giga cracked engineers who reverse engineered the TMEM latency to within 2 cycles hand writing kernels used trillions of times per day. Absolute insanity

@kalomaze what's the problem? backward is just forward(m.T) :) or that's at least what we like to tell ourselves

@casper_hansen_ ah. in a perfect world...

@kalomaze This is not true ...

@kalomaze What's special about backward kernels?

@kalomaze What’s so special about backwards?

@kalomaze real

@kalomaze by backwards u mean autodiff?

@kalomaze I should dust off my C and assembly…

@kalomaze nah, I don’t even think it’s training, the whole moat lies in inference though: hitting strict p95 SLOs under bursty load while juggling continuous batching, KV-cache paging/prefix-sharing, quant+speculative/assisted decoding, cuda Graphs, async prefill/decodes, and

@Infopulsed a good chunk of this isnt used in practice most of the time or are mutually exclusive things and also i don't know what the fuck a MoE scheduler is supposed to be

@kalomaze there should a slur for people who think inference is more important than pretraining inferoids servecels "I overfit existing models and call it science"

@kalomaze there are a lot of chinese guys who are pretty cracked at cuda

@short_cast i'd guess that least ~70% of the people in the group i am thinking of are Chinese tbh

@kalomaze Yeah CUDA resources are rare. Someone should make a good document that can be followed easily. Alternatively just snipe Graphics people, they tend to be doing similar things

@kalomaze Once you understand how thread/block works, it's not that hard for anyone who knows C++ and linear algebra. But, you said triton, which tells me that you're wanting the bare metal abstracted away.

@kalomaze how hard could it be? We did the same thing in quantum chemistry back in the day when we had to hand optimize our code for the RISC architecture. Oh, and also be an expert in quantum mechanics at the same time

@kalomaze There’s probably more who can but just haven’t applied their GPU skills towards ML specifically. They’re busy doing other things. I haven’t exactly seen these AI companies targeting us graphics programmers to write their GPU code. Not even CUDA goes low level enough tbh.

@kalomaze there was not much money in it ~5 years ago

@kalomaze there are zero people alive who have done this. Nvidia/Google/TSMC et al microarchitecture is amazing, it also is extremely suboptimal. to do it effectively is to rewrite the entire stack at once. not just the compiler,(CUDA kernel) but the entire

@kalomaze Honestly this number feels generous even. The gradient computation stuff is just insane - like you need to understand both the forward pass AND how to efficiently backprop through custom ops. Most people can copy/paste from cuDNN but writing from scratch? That's wizardry level

@kalomaze This seems incorrect by virtue of there being at least 100 people who can write these kernels working at NVIDIA 😃

@kalomaze for training you only need to target a handful of accelerators you use, whereas for inference you need to think of all sort of weird/out-dated/non-cuda devices.

@kalomaze It's actually fun. People usually just think GPU programming means parallel compute units. They should think about it from perspective of shared resources (SLM shared across cores, Registers across GRF). That'll lead to a logical model where occupancy, throughput can be argued.

@kalomaze i found them all

@kalomaze So nice that the abstractions are there for so many people to take advantage of those <100 people