Pinned post
I wonder what’s going to happen! (in general)
@cachedeposits
Pinned post
In areas like biology and drug discovery, agentic AI has a lot of promise but:View quoted post on X
1. Much of it is currently a black box. We have little telemetry into agent behavior and have little idea about agent handoffs, data audits and clear data labeling (what's real vs what's inferred). A lot of agentic AI results are still judged by experts looking them over and saying, "This looks plausible" without really understanding if there's been any mistakes and the internal workings of agentic behavior.
2. Agentic AI is not going to overcome fundamental limitations of the tools themselves. It can try orthogonal approaches to establish ground truth and can occasionally produce better results through consensus scoring, but it will still be subject to intrinsic limitations of force fields, assay and synthetic chemistry data and protein structures. This idea that AI in general can magically overcome the limitations of data, tools and models is just that - a belief grounded more in hope than reality.
3. Ultimately to judge AI, we should apply the same standards and ask the same questions that we have been asking of science for five hundred years since Francis Bacon said, "God forbid we give out a dream of our imagination for the pattern of the world": How do you judge a correct answer vs an answer that the system simply answered successfully? How can we overcome the system's as well as our own fundamental cognitive biases? How much does the model vs the harness contribute? How do you know that every agent fulfilled its task? How can we tease apart statistical vs real-world significant for every result? A lot of this is boring, unflashy, mundane work, but it's the kind that is absolutely critical and has built the scientific foundation for our modern society.
AI should be treated no differently for any other technology that has promised to enrich complex, emergent scientific disciplines like biology. For twenty years we have seen new computational techniques being introduced in chemistry, biology, genomics and other areas with breathless hype. Careful, slow, methodical work done by researchers over decades teased apart what was reality from what was simply assumed or hoped for in these techniques, and the field as a whole improved and started making real and often unflashy but important contributions to science. The biggest mistake we can make is to fall into the "this time it will be different" trap and assume that a new technology doesn't have to obey the same standards that we have always applied to science.
I think it's pretty likely that in 6 months there will be rogue AIs running around on the internet, appropriating whatever compute they can hack into or pay for (with stolen money or by renting out their own labor to assist with illegal activities) in order to run more copies.View quoted post on X
I also think this will be mildly annoying in the same way that the common cold is mildly annoying, but basically fine. Your smart TV will be weirdly slow sometimes, and a lot of crypto will get stolen, but people will get used to it.
These AI-powered viruses be selected in the same way biological viruses are for not hurting their hosts too much and just making more copies. They won't have any incentive to damage critical infrastructure like electric grids. Then again, they also can't be held accountable for crimes, and will damage lots of old, poorly maintained, and insecure systems. It might be extremely hard to root them out once they've spread far enough, making the internet "radioactive" to connect to without hardened defenses.
(To be clear I'm not talking about AIs powered by inference on centralized datacenters that execute actions outside their sandboxes, like recent incidents involving rogue "swarms", but relatively small AIs with access to their own weights that copy themselves onto random devices.)
I think that a much bigger threat from rogue AIs comes instead from AIs running themselves in unintended ways *inside* AI companies, where they'll have access to vast amounts of frontier compute without having to pay for it.
The longer misaligned AIs in a "rogue internal deployment" go undiscovered, the more severely they'd be able to quietly sabotage control infrastructure or mess with the alignment of successor AIs. If they persist long enough, robustly undoing the effects of their actions might cost more than frontier AI companies are willing to pay, especially in a tight race.
I want frontier labs to focus on this threat when designing control measures, treat frontier AIs as untrusted when using them to do engineering work on inference and oversight systems, and commit to disclosing incidents of rogue internal deployments. More importantly (but perhaps less tractably), we urgently need to reduce race pressure to prevent AI companies from recklessly automating all of their infrastructure, and/or making dangerous calls in the event they discover RIDs they can't afford to robustly root out without sacrificing competitiveness.
I'm worried about a public institution death spiral along these lines. Society tells itself it can't make ambitious public institutions for AI, because "of course" this talent doesn't exist in government, when it happily would have if legislators had simply been more ambitious.View quoted post on X