ok lets fucking start

ok, so it has concept of host and device.. device is where gpu kernels run.. what what is gpu kernel

ok, so kernels are functions which can run in parallel by many threads to exploit parallelism of "cuda cores" ... they use indexing to identify which thread has to do what calculation.. it seems...

ok, this seems important... threads is simple, block is group of threads, and grid is group of blocks... curious about why they need two level of heirarchy

so a CUDA block - which is a group of threads - can share data via shared memory different from terminology of OS, wehre a thread is by default assume to share program memory with other threads .. but ok understandable

ok so this is the cuda kernel everyone talks about.. you are defining a function with what goes into it.. and what is its "execution shape" i guess...that is you want for example 2 separate groups of threads which execute this function where each group has 4 threads individually

ok so a grid is a group of blocks... i think blocks probably must feel constrained due to sharing of memory on the gpu.. whereas grid can be scheduled to run freely .. so hence the abstraction... also grid and block both can have dimensions.. so probably we are looking at bin

I think my intuition says, that you can imagine a run of a CUDA kernel on GPU like stacking of objects in a 3d word... you are trying to come up with parameters which optimize for shortest runtime in a given GPU "space" but trying to do a box-fitting

ok so four memory types which i think only first three must be common across and 4th should be more arch specific/use case specific.... global is slowest, local is fastest for thread .. and shared is common in a block... so no grid level memory

okay, threadIdx helps the CUDA kernel (which is just a function) .. to identify on which part of data to work on.. so definately we are using the structural info of the run i.e. what is the block..grid.. thread.. etc. to generate a index...

ok, this makes sense... and actually this means that tradeoffs when carefully managed can give you huge boost in terms of what models you can run on your device.. by shuffling data between cpu/gpu mem..

ok, as expected something to sync threads.. the programming model is very much similar to all other distributed programming models.. idk why i wasn't even trying it lol..

so block also does grouping of threads.. but that is more around sharing of memory.. and a warp is more around being efficient on common compute together probably

multiple kernel part is understandable.. but wtf is this stream .. i thought we had everything covered via block,grid,warp, threads etc.

so concept part over.. let's practice.. don't have a CUDA hardware so will just use online practice grounds for CUDA.. found one I don't really like leetcode like setups, but again learning fast is our goal and i don't have hardware meh

ok there was a book suggeston and i feel the diagrams in this book are cool.. so let's just skim thru diagrams

> so GPU arch is designed for dumb paralell processing, whereas CPU is designed for more control ... GPU is a batch processor in some ways.. (this is explanation of previous diagram)

> architecture of a cuda capable gpu > so threads are run via a thread execution manager

(last diagram explanation) > this one is little unclear explanation of last diagram's middle part.. so the 8 blocks in last diagram are streaming multiprocessors, and sort of local cache with them > texture memory looks interesting for game like things... seems like it must be

what are load/store units (also from diagram above) .. coordinate between global mem and local cache for each streaming microprocecssor

ok, so device(gpu) and host(cpu) have different constraints on what they can do > host can do memory transfers between mems > device can do a lot more

ok, wasn't expecting that cudamalloc/cudafree would just work with global mem

for transferring data, there is a cudamemcpy() function.. which allow both side bidirectional transfers

the font is shitty, but this is the first kernel i am seeing.. it's matmul thru cuda.. and it's look fairly...... simple

so blocks (which basically is a thread group sharing memory) gets partitioned into warps (which are basically a group which works well in single shot with processor) for scheduling

ok, this is different kind of variable qualifiers and their lifetime/scope... fairly predictable

ok, so this is a tiled matrix multiplication in cuda... i was not expecting the code to be this ugly. but meh... > Pd = Md * Nd using tiled matrix mul > you identify which block/thread you are in > tiles help you create virtual groupings of matrix element > simple inner loop 1.

acess pattern is faily standard standard stuff.. you should access things which are closer in mem together to exploit "recency" bias of system organization

in cuda terminology, accessing memory nearby each other can be done in single transaction and is called "memory coalescing" .. opposite is when you access alternative positions in different threads

ok, tiling gives this benefit of coalescing (sort of amortization of it)