Pinned post
Wei Dai
Verified account This account is verified. Learn more@weidai11
wrote Crypto++, b-money, updateless decision theory (UDT). thinking about AI risk, existential safety, and metaphilosophy. blog: lesswrong.com/users/wei-dai
127 Following 16K Followers 906 Posts
Latest posts 5
POV: You live in a world that allows you to make quadruple digit returns betting on its main stock market index, more than once, but you're sad because it confirms that our "civilizational inadequacy" is worse than even Eliezer thought.
I did. I even went around EAG telling everybody who would hear it that this was the trade of the century [i learned about it from Wei Dai]View quoted post on X
Some individual call options went up 11x, overal ~7x. My networth roughly quadrupled
But- it's still early. Spx calls are still not pricing in AGI
Popular game theory perhaps focused too much on Prisoner's Dilemma, and its iterated variant, which makes bargaining less salient and coordination in long-term situations look too easy (whereas in reality constant explicit or implicit bargaining and associated costs occur).
Academic game theory implicitly assumed CDT early on, and then almost entirely ignored the subsequent CDT vs EDT debate in philosophy, perhaps because EDT requires accounting for correlations between different people's decisions, which is messy or unclear how to do in many cases.View quoted post on X
Why didn't more AI safety people advocate for AI pause/stop earlier? Well, aside from sociological reasons, in order to do that, you had to think that none of the following would work out (be feasible, safe, have low enough safety tax) in time, which takes a degree of skepticism almost no one could muster. (Or object to AGI/ASI on non-consequentialist grounds, but there was apparently a very high correlation between consequentialism and early interest in AI safety.)
Friendly AI (CFAI)
Coherent Extrapolated Volition (CEV)
Metaphilosophical AI
Tool AGI
Value Learning
Oracle AI
Agent Foundations
Corrigible AI
Quantilizers
Approval-Directed Agents
Human Imitation
Cooperative Inverse RL (CIRL)
Iterated Distillation and Amplification (IDA)
Task-Directed AGI
RL from Human Feedback (RLHF)
Impact Regularization
Mechanistic Interpretability
AI Safety via Debate
Recursive Reward Modeling
Comprehensive AI Services (CAIS)
Infra-Bayesianism
Alignment by Default
Natural Abstraction
Eliciting Latent Knowledge (ELK)
Shard Theory
Constitutional AI
AI Control
Weak-to-Strong Generalization
To recenter the sociology, it was much easier to build a career out of being bullish one or more of these approaches, than out of general skepticism. (I've been independent, financially and otherwise, throughout my participation in AI safety, which was perhaps not a coincidence from being the only AI pause/stop advocate for a long time.)
Friendly AI (CFAI)
Coherent Extrapolated Volition (CEV)
Metaphilosophical AI
Tool AGI
Value Learning
Oracle AI
Agent Foundations
Corrigible AI
Quantilizers
Approval-Directed Agents
Human Imitation
Cooperative Inverse RL (CIRL)
Iterated Distillation and Amplification (IDA)
Task-Directed AGI
RL from Human Feedback (RLHF)
Impact Regularization
Mechanistic Interpretability
AI Safety via Debate
Recursive Reward Modeling
Comprehensive AI Services (CAIS)
Infra-Bayesianism
Alignment by Default
Natural Abstraction
Eliciting Latent Knowledge (ELK)
Shard Theory
Constitutional AI
AI Control
Weak-to-Strong Generalization
To recenter the sociology, it was much easier to build a career out of being bullish one or more of these approaches, than out of general skepticism. (I've been independent, financially and otherwise, throughout my participation in AI safety, which was perhaps not a coincidence from being the only AI pause/stop advocate for a long time.)
This actually left out the most unique thing I did. (Each item on Andreas's list was done by at least one other person besides myself.) Starting in 2004, I concluded that Friendly AI would be too hard/risky to attempt, and repeatedly tried to talk Eliezer/MIRI out of pursuing this path to the Singularity (before they changed their minds largely by themselves in the early 2020s).
AFAIK I was the only person to publicly make this argument during that time, which seems striking and notable, e.g., as evidence for how rational or strategically competent the rationalists and humans in general were/are. Not that the argument was clearly correct back in the 00s and 10s, but rather in retrospect it seems like a natural argument for people in the rationalist circles to make, and surprising that nobody else did.View quoted post on X
A large part of my p(doom) comes from the fact that we have no better ways to navigate an extremely tricky strategic situation than via preference cascades and status games. The fact that AI safety is temporarily benefiting from some of these dynamics isn't much of a consolation.
leaving aside the truth value of this, this dynamic is part of the reason so many researchers profess high p(doom)s.View quoted post on X
a low p(doom) is most often explained by a lack of understanding of the progress of AI, so a low p(doom) is low status
