Daniel MacArthur

As someone who did ~10 trips from Boston to Sydney with three kids at varying ages, I can confirm this is true. The nightmare window is the period where they’re old enough to walk but not yet old enough to be transfixed indefinitely by a screen.

An incredible opportunity for ambitious scientists interested in solving research problems that are ill-suited to the traditional academic model. (Note: you may not yet be aware that you fall into this category!)

I fully endorse Anshul’s points here (acknowledging I’m conflicted, with a funded research partnership with DeepMind on AlphaGenome use). There’s still plenty of room for improvement in this space, but Steven’s critique is wildly unfair.

What the hell is this timeline and can I please have another one

I’ve been reading a bunch of convoluted explanations and conspiracy theories about why AI companies are “suddenly” concerned about existential risk, which I’m finding genuinely bewildering.

This isn’t that complicated. The labs are perfectly sincere in their concerns, and correct that AI is extremely dangerous. The people working in the field haven’t come to this view suddenly; many have been thinking and writing about this for years. Many researchers entered the field because they saw the existential risk posed by artificial minds bootstrapping themselves to superhuman capabilities, and decided the best way for them to reduce that risk was to work actively on building aligned AI systems. (Yes, this initially seems counterintuitive, but is a perfectly reasonable approach given the alternatives.)

The risk isn’t overblown. If humans build computational systems capable of recursive self-improvement that aren’t exquisitely aligned with human interests, then the expected outcome really is that we all die. No-one really knows how far we are from that point, but the people who know best are exactly the same people currently sincerely expressing alarm.

(People skeptical that misaligned superintelligence could kill us all are extremely confusing to me. Our species depends on a thin shell of habitable atmosphere around a single ball of rock. Even if a misaligned ASI wasn’t actually trying to kill us, almost anything it might want to achieve at scale would likely destroy us as an incidental side effect. Examining the catastrophic damage humans have managed to do to other species without intending to kill them off provides a valuable analogy.)

The autonomous hacking incidents over the last couple of months are surprising in some of their details (especially the degree of inter-agent coordination), but not in their fundamental shape. Misaligned AI systems performing damaging actions in pursuit of their assigned goals is *exactly* what people thinking about this field have predicted for many years.

Yes, there are ways that increasing regulation might benefit the big AI labs. No, that’s not the primary motivation for their proposals this week. We really are approaching the cusp of an existential threat: the technology for recursive self-improvement of AI systems is (probably) within sight, and market forces and geopolitics mean that without coordinated intervention our default path is a headlong scramble towards it.

I know none of us want to be dealing with the impending threat of a sci fi apocalypse in 2026. It would be much simpler if this was all a confabulation or a self-interested conspiracy. But it really isn’t. I’m sorry.