Some people are claiming this is a straw position. Here's one example. Notice he's being very vague about how exactly LLMs invalidate previous ideas. This is the usual case. My guess is they like agreeing with each other about that part, but not about the specifics.

At least this one isn't vague! And has roughly exactly the level of mentation depicted above. Sorry, opponent is not straw; wish it was but no.

@ESYudkowsky positioning decision theory as of the same validity as newtonian gravity is incredibly hubristic given there are countless observations humans make confirming newtonian gravity and almost none (in the mathematic sense) about (your versions of) decision theory

@atroyn While many mathematical ideas are shakier than the Newtonian approximation of gravity in practice, it can also be said in their favor that the correct ones are true in all universes and not just this one. In any case, the comparison is about abstraction, not certainty.

@ESYudkowsky the existence of hot air balloons is Bayesian evidence against the theory of gravity.

@ESYudkowsky This isn't even remotely the right analogy. It's more like "have you updated your theory of gravity in light of the measurement that Mr. Galilei has done showing that objects of different mass all fall at the same rate?"

@ESYudkowsky Have you ever built a hot air balloon? It should be a prerequisite before any of us listens to you /s

@ESYudkowsky Eliezer my biggest disappointment with your thinking on LLMs has actually been the opposite of this: you project too much of your (early and good) theory of AI agency and alignment onto them, like talking about the delta-v of hot air balloons.

@ESYudkowsky Having to invent implausibly dim-witted opponents may not be the demonstration of intellectual confidence you think it is.

@ESYudkowsky Please Eliezer, put extra newlines in-between the questions, the wall of text is horrible to read.

@ESYudkowsky I remain unconvinced this will convey your valid points to anybody who did not see them before. Why are you trying this?

@ESYudkowsky unfortunately the place where the "grim predictions" come from has no basis in empirical reality, unlike the theory of gravity Very funny to whole-cloth fabricate something and then assert that it ought to be considered analogously to 1 of the 2 greatest scientific discoveries

@ESYudkowsky That was illuminating! Thank you for writing this

@ESYudkowsky Honestly I'm more interested in the question of why the super-intelligence (by definition smarter than the whole of humanity) still continues to believe that the most worthwhile goal in the world is to build more paperclips... I mean, WE are smart enough to decouple sex from

@ESYudkowsky As usual, to the extent that I can follow your posts, I appreciate your extreme wit, vision, and earnestness, while also slightly shitting my pants

@ESYudkowsky I don't understand who you're planning to convince by writing stories about stupid people disagreeing with you

@ESYudkowsky It always amuses me how, even in your own fantasies, your self-insert is a tool who contributes nothing of value to the discussion and just responds to any challenge by trying to intimidate people with content-free jargon.

@ESYudkowsky Theory of gravity as a metaphor here is very questionable. It may make this idea of superintelligence sound as if it has a strong foundation, like observations, experiments, and theory, but in reality it is more like medieval scholasticism

@ESYudkowsky You’ve got to fix your book cover. Have some fun with it.

@ESYudkowsky I know who Q2 is

@ESYudkowsky Obviously space travel is impossible. Balloons only work because they are lighter than the surrounding air, but there is vacuum on the way to the Moon. Heavier-than-air flight is impossible and everything is heavier than vacuum. (1/2)

@ESYudkowsky Q (😅): Gravity is shown to function consistently in once you add thermo & hydrodynamics, and take all three to be part of a mother model, classical mechanics. Is there a classical mechanics for decision theory, applicable to NNs? What would it look like?

@ESYudkowsky Catharsis aside, couldn't you have actually changed a few more minds on the margin if you said like "LLMs, and for that matter essentially any possible architecture including superaugmented human brains, have the exact same problem" instead of enjoying calling them idiots so much

@ESYudkowsky Markov chains

@ESYudkowsky lol "flounces" good word, doesn't get used enough

@ESYudkowsky Type shit

@ESYudkowsky "1880: Heavier than air flying machines are possible, even if not now. Then they will be used to drop bombs on people and whole cities will be wiped out in a apocalypse of destruction!" True, but... We are still here and the overwhelming majority of flights are peaceful.

@ESYudkowsky @grok dumb this post down so that an idiot can understand it

@ESYudkowsky There is a lot of manufactured gotacha moments with inference that either does not represent the point accurately or limits the implications of such point to what is convenient for the argument made, in every discourse at the moment. Nothing special about AI.

@ESYudkowsky Q4: do u agree that the 🎈 project provided useful lessons & gathered humans and capital needed to start serious work on getting to the moon? That it didn't require same precautions, so our careless approach was acceptable? That u can't hope to visit moon without mastering 🎈?

@ESYudkowsky Is the claim that decision theory provides a rigorous argument for orthogonality, or for some broader notion of intelligence being hard to align even if orthogonality were false and there was a fair amount of “convergent evolution” in values?