I finally got around to reviewing this paper and it's as bad as I thought it would be. 1. Zero data or evidence. Just "we guessed right in the past, so trust me bro" even though they provide no evidence that they guessed right in the past. So, that's their grounding. 2. They used their imagination to repeatedly ask "what happens next" based on.... well their imagination. No empirical data, theory, evidence, or scientific consensus. (Note, this by a group of people who have already convinced themselves that they alone possess the prognostic capability to know exactly how as-yet uninvented technology will play out) 3. They pull back at the end saying "We're not saying we're dead no matter what, only that we might be, and we want serious debate" okay sure. 4. The primary mechanism they propose is something that a lot of us have already discussed (myself included, which I dubbed TRC or Terminal Race Condition). Which, BTW, I first published a video about on June 13, 2023 - almost a full 2 years ago. So this is nothing new for us AI folks, but I'm sure they didn't cite me. 5. They make up plausible sounding, but totally fictional concepts like "neuralese recurrence and memory" (this is dangerous handwaving meant to confuse uninitiated - this is complete snakeoil) 6. In all of their thought experiments, they never even acknowledge diminishing returns or negative feedback loops. They instead just assume infinite acceleration with no bottlenecks, market corrections or other pushbacks. For instance, they fail to contemplate that corporate adoption is critical for the investment required for infinite acceleration. They also fail to contemplate that military adoption (and that acquisition processes) also have tight quality controls. They just totally ignore these kinds of constraints. 7. They do acknowledge that some oversight might be attempted, but hand-wave it away as inevitably doomed. This sort of "nod and shrug" is the most attention they pay to anything that would totally shoot a hole in their "theory" (I use the word loosely, this paper amounts to a thought experiment that I'd have posted on YouTube, and is not as well thought through). The only constraint they explicitly acknowledge is computing constraints. 8. Interestingly, I actually think they are too conservative on their "superhuman coders". They say that's coming in 2027. I say it's coming later this year. Ultimately, this paper is the same tripe that Doomers have been pushing for a while, and I myself was guilty until I took the white pill. Overall, this paper reads like "We've tried nothing and we're all out of ideas." It also makes the baseline assumption that "fast AI is dangerous AI" and completely ignores the null hypothesis: that superintelligent AI isn't actually a problem. They are operating entirely from the assumption, without basis, that "AI will inevitably become superintelligent, and that's bad." Link to my Terminal Race Condition video below (because receipts). Guys, we've been over this before. It's time to move the argument forward.
Terminal Race Condition: my video where I introduced the idea during the height of my own AI safety research. https://youtu.be/feEJuBFha8E (Update, while I agree that acceleration is the default path, I no longer believe that "Fast AI is automatically dangerous AI")
I also sanity checked myself against multiple AI's. Obviously, they are sycophantic and will agree with me, but they also articulate my point fairly clearly.
I asked if this paper would even pass muster as a position paper in an actual academic journal—unequivocally no. ChatGPT said this qualified as a blog post or, at most, a policy whitepaper
@technocapt @slatestarcodex We've been talking about this stuff for a few years and they haven't updated their narrative. Is it worth kicking a dead horse?
@robinhanson AI won't kill everyone. AI won't even kill most people. In fact, I'm putting my entire livelihood on this, considering I used to be in the AI safety movement but defected due lack of rigor.
@AILeaksAndNews Check my replies, I made a video about it almost 2 years ago. It's called Terminal Race Condition
@akidderz "Past performance is not an indicator of future performance" but also just saying "we guessed right in the past" without evidence is dodgy.
Oh ABSOLUTELY. Terminal Race Condition would make a fantastic sci-fi premise! It has a very Three Body Problem vibe to it.
Thanks for your reply. 2. Your definition for "superhuman coder" is extremely conservative. You're not taking into account exponentials, distillation, and quantization. A ratio of 30:1 is an extremely low bar. 5. Okay fair. Interesting that you don't mention liquid transformers, which I see as far more potentially dangerous. In-context learning and recurrent transformers are one thing, but still mostly defined by their weights. Models with weights that continuously evolve are another thing. 6. I think you're missing my point and dismissing it. Money doesn't just come from nowhere, and if you want those big investments, you have to produce not just AI, but useful and safe products. (also, your compute forecast page is down). And finally, most disappointing, you're missing the forest for the trees. You seem to implicitly believe that alignment is unsolvable, or that current efforts by frontier shops are inadequate, but there's not really any evidence of that. They ship early and often, with very tight feedback loops and market corrects. This is the primary reason I abandoned TRC (which it would be nice if you conceded someone else came up with this long before you did) is because I realized there are far more interlocking feedback loops than just "more compute >> better AI" There's a very clear attractor state already created here, which really needs to be discussed and ironed out, rather than rehashing old ideas like TRC. The TLDR of this attractor state is finding those values and trends to which AI autonomically aligns (and not just instrumental convergence, which basically says "useful things are useful") I mean "what is the quintessential optimal state of AI?" I moved on from worry and into solutions, hence my work on coherence and creating stable attractor states in AI trajectories. IMHO, you're about 2 years behind the times in terms of alignment theory.