i actually agree with this and think that you should be pretty sure the regime you're entering is strictly better even considering uncertainty before actually giving powe to anyone or anything
Why should we not yield full control of our future to an entity vastly more powerful and intelligent than us? Many people relying purely on "logic" think opposing such a future is a form of prejudice and lack of open-mindedness -- indeed I have heard this argument made increasingly in certain circles.View quoted post on X
But David makes a good point here: many of our intuitions around meaning and other such fuzzy concepts may encode ways of protecting us against death or other terrible outcomes, even when we cannot fully explain them from first principles.
David gives the example of our desire not to lose our jobs. I would add, in the context of AI handoverism, the reluctance of most humans to cede control to an entity vastly more intelligent than ourselves. My guess is that this instinct captures something about the dangers of placing ourselves entirely at another entity’s mercy.
The problem with a certain strain of rationalism is the impulse to discard any intuition that cannot be reconstructed from first principles, often accompanied by not much desire to acquaint oneself with the history that might explain it (everyone can just opine on anything if they just use logic is a good principle, up to a point). Intuitions can, of course, be wrong. But we can also go too far: lindy instincts often contain wisdom.
I think the full rejection of human intuition, as well as a lack of desire to acquaint oneself with history and works of literature and philosophy outside a certain rationalist canon explains why ideas whose dangers would once have seemed obvious acquire the glamour of daring intellectual discoveries and lead many towards repugnant conclusions.
