@mike64_t ASCII art is all you need? FYI small typos: “Naively attenting to”
@garybasin fixed
@mike64_t Cool idea. Seems way more amenable to from-scratch RL too
@garybasin It is partially motivated by the eventual need for RL. You will hate life during pretraining, but as soon as most of your flops is inference, you’ll appreciate O(1) memory as well as knowing each of those flops is as expressive as it can be.
@mike64_t In the simplest terms, can I say this consists of 1) a frame encoder (similar to the VAE of image generation models) that provides compressed input, and 2) a multi-layer LSTM that processes the input as well as maintains temporal correlations?
@askerlee Essentially yeah
@mike64_t I am also pondering over this question. Recurrence might be good for reasoning tasks as well, due to causal nature of learning..If I understand the prefrontal cortex region in brain does have recurrence and thereby its the System 2 center of the brain. Also we saw models like HRM
@smjain yep, it is potentially connected. Depth might be all you need if you are not information bottlenecked.
@mike64_t > Transformers are shallow, constant depth circuits which effectively throw away all latent compute when a token is sampled. This is a common misconception, you're not throwing away any of your hidden states of previous tokens. You could argue the logits are thrown away, but
@wavefunk_ The KV cache is not true state. If it were, we wouldn't have this problem of constant depth. The question is of course a bit dependent on semantics and what you mean by state, but I mean state in a sense that I can make a decision on it that is "branchable". Any soft decision
@mike64_t where may I apply to work with y'all?
@ioci24 convincing DMs go a long way :)
@mike64_t Do Mamba hybrid suffer from the same issues?
@dmnsl1 yes
@mike64_t How does it handle long sequence descriptions of the present (ie an information dense description of something now or next)? Looks cool
@realJosiahClark very hard question to answer, we're still very much in gpt2 territory here in terms of capabilities, so evaluation is kind of tricky. Additionally, most LLM benchmarks don't exactly transfer over. I think the best thing to do is to keep hammering the loss down until code vibe
@mike64_t Nice work! Is FBAM NC1?
@jalalnaghiyev06 It's in P