1) The First law of complexodynamics > its about the notion of complexity, and how it tends to have a inverted U shape.. in layman terms the system starts and ends in simplicity, but peaks for complexity in the middle intermediate stage > interesting how this looks like a

> so entropy increases continuously > but complexity follows this curve > in last i wrote the intuition about how this looks similar to computations where you start and end in a single state but there is a "expansion" of states in middle does this mean universe can be modelled

(this is going to be a long thread, these are not exactly ideas that can be skipped without losing their essense) two questions to answer - > what is formal definition of complexity > prove that complexity we define ends up explaining the pattern we are seeing It's intuitively

> using KC to define entropy > also observation that uniformly random string has highest KC but is least interesting (i.e. least complexity) > example, you can just define a random string by saying "it's random" ... > some mathematical arguments on how we can do either bounded

we got done with KC and entropy... lets now define something which is closer to the notion of complexity the term is "sophistication"... quite a apt term imo... if you fall in a set of normies you are not sophisticated... you gotta be not easily describable (not a word ik) If

2) Keeping Neural Networks Simple by Minimizing the Description Length of the Weights (hinton's paper)

(note: this list has old papers, its about conceptual stuff.. not like these methods should be applied verbatim today) > sort of sounds like pushing network towards simplicity.. a simple description also should tend to minimize the MDL (minimum description length)... > that is

> most of the paper is just about applying this to weights, getting some improvements but this part of paper is very interesting > "the best model of some data is the one that minimizes the combined cost of describing the model and describing the misfit between the model and the

3) Order matters: Seq to seq for sets (bengio's paper) > order of input/output data matters significantly when learning underlying model

> most paper is just doing different experiments on different dataset, not much discussion on theoretical things > this was done on LSTM, and basically I think it is quite similar to idea of "curriculam learning" which we call nowadays.. i.e. when you order the data in a certain

> in the paper, they also mention about searching for the right order ... imo this is kinda what happens when you do multiple epochs perhaps?? that is to day, if the order of data is not perfect, doing more epochs help the model to actually see the data in the right order of

4) The unreasonable effectiveness of RNN (karpathy's popular writing) > imo this is excellent for coding deep dive, so will skip it..Also, probably something i can say i have gone enough times on > RNN were the sota before transformers, but faced issues like scalability and

5) The annotated transformer > another coding thing.. its a easy to follow implementation > I have covered transformers just a day back in another thread so doesn't make sense to go over it again here > https://x.com/attentionmech/st...

6) Understanding LSTM Networks (Colah's writing) > unlike karpathy's this one is less popular, and i am also not thru with it.. sogonna spend some time on it

> a reccurrent neural network can be seen as a normal neural network after unrolling the recurrance > makes me wonder if we can solve the long context problem in RNNs first (or whether even they have it) and then take that to a mathematically equivalen Transformer

> so RNN have issues wiht long context... LSTM came to solve it .. LSTM = Long short temr memory > the repeating module of LSTM is more complex than RNN and something to do with the memory part

> what is appearing interesting to me about LSTM is how this looks like the residual stream a bit .. the top line without any ops, then some other circuits going via other paths > the ops are modeled like as if they are memory operations.. > mostly there are some function

7) RNN Regularization (illya's paper) > dropout was known to work for NN, but not for RNN/LSTM.. this paper helps in that

> they do a dropout operator which 'corrupts' the information carried by the unit --> which according to them leads to more robust computations >one way to imagine this is if there was a boundary where it could just "tip" into wrong computation zone... adding noise/corruption

8) Imagenet (Illya's paper) > non saturating neurons + dropout + efficient gpu implementation on image dataset

> most paper is talking about stuff we consider obvious nowadays.. like use of relu, dropout, how to run code on gpus assinging blocks etc.. in hindsight it looks simple but this was sota at the time

9) Minimum Description Length Principle > this is like a ~80 pages booklet > we covered some parts of this discussion in few papers in this thread, so lemme just filter the best parts

Formalising the idea of MDL leads to a general theory of inductive inference with properties - > Occam's razor: MDL automatically goes away from unnecessary complex descriptions > No overfitting, automatically: overditting need larger than MDL length, hence tending towards

how mdl relates to model selection: > model is about doing a statistical inference... which needs regularity of data, which means ability to compress , which means a model which can compresses data the most (which still being a statistical inference model) is the best hypothesis

so, lets see what this is overall saying > building models of statistical inference is kinda a search for best hypothesis or best set of hypothesis that explains a set of data >the more hypothesis's you need , the more you are trying to overfit a model.. hence you should go with