Twitxy Saved

Adam Kaufman

Verified account This account is verified. Learn more

@eccentric1ty

MTS @ Redwood Research (@redwood_ai); opinions my own etc etc
Joined July 2023
967 Following 1.1K Followers 1.5K Posts

Latest posts 5

Adam Kaufman Verified account @eccentric1ty
I think it's pretty likely that in 6 months there will be rogue AIs running around on the internet, appropriating whatever compute they can hack into or pay for (with stolen money or by renting out their own labor to assist with illegal activities) in order to run more copies.

I also think this will be mildly annoying in the same way that the common cold is mildly annoying, but basically fine. Your smart TV will be weirdly slow sometimes, and a lot of crypto will get stolen, but people will get used to it.

These AI-powered viruses be selected in the same way biological viruses are for not hurting their hosts too much and just making more copies. They won't have any incentive to damage critical infrastructure like electric grids. Then again, they also can't be held accountable for crimes, and will damage lots of old, poorly maintained, and insecure systems. It might be extremely hard to root them out once they've spread far enough, making the internet "radioactive" to connect to without hardened defenses.

(To be clear I'm not talking about AIs powered by inference on centralized datacenters that execute actions outside their sandboxes, like recent incidents involving rogue "swarms", but relatively small AIs with access to their own weights that copy themselves onto random devices.)

I think that a much bigger threat from rogue AIs comes instead from AIs running themselves in unintended ways *inside* AI companies, where they'll have access to vast amounts of frontier compute without having to pay for it.

The longer misaligned AIs in a "rogue internal deployment" go undiscovered, the more severely they'd be able to quietly sabotage control infrastructure or mess with the alignment of successor AIs. If they persist long enough, robustly undoing the effects of their actions might cost more than frontier AI companies are willing to pay, especially in a tight race.

I want frontier labs to focus on this threat when designing control measures, treat frontier AIs as untrusted when using them to do engineering work on inference and oversight systems, and commit to disclosing incidents of rogue internal deployments. More importantly (but perhaps less tractably), we urgently need to reduce race pressure to prevent AI companies from recklessly automating all of their infrastructure, and/or making dangerous calls in the event they discover RIDs they can't afford to robustly root out without sacrificing competitiveness.
28 replies 39 reposts 444 likes 33K views View on X
Adam Kaufman Verified account @eccentric1ty
If you are currently working on building ASI, I think your work is pretty likely to lead to the deaths of billions of people and/or the permanent disempowerment of humanity. That's pretty uncool of you, and you should consider doing literally anything else