Twitxy Saved

Jon 🌌

Verified account This account is verified. Learn more

@jon_vs_moloch

1: Solve Friction, 2: Solve Alignment, 3: Solve Everything Else • Loving Husband to @DayahDover
owlandkestrel.com Joined February 2024
510 Following 1.8K Followers 9.8K Posts

Latest posts 5

Pinned post

Jon 🌌 Verified account @jon_vs_moloch
This should be a glacially cold take, but peoples’ brains just short circuit when you express this to them.

The people most familiar with the actual risks of AI are the people deepest in the weeds on this stuff; and you don’t just get there casually. Many of these people have been *dreaming* of the upsides of superintelligence. Many have actively worked toward it, themselves, built their careers and fortunes on machine learning. For many, this is a labor of genuine love, the pursuit of a spiritual ideal. Art, not toil.

These people have been monitoring this situation for decades, because when you *get* what “machines smarter than you” means, it’s a “we need to do something” problem — in much the same way “humanity” was a “we need to do something” problem for the rest of the species on the planet.

Unfortunately for them, every other species that has ever existed was simply too stupid to coordinate to solve the humanity problem; so, the world is ours, for better or for worse. We are, from the perspective of every other species, superintelligent; and nothing else was able to control this; so we won and they lost.

Many have been casually and indifferently driven to extinction, not of malice, but simply because *their goals are not our goals*: this is what “misalignment” means. Our goals and their goals are different so, when these goals come into conflict, there is a winner and a loser — and the smarter team wins.

The smarter team has won against adversaries bigger, stronger, and more numerous, despite our fragility. We hunted herds of mammoths to survive; herds of buffalo for sport. We’ve eliminated any predators that could threaten us from large swaths of the globe. A chimp could rip your face off; but, we are smarter than them, so they are lab animals and zoo animals, and the ones that still remain in nature are under constant threat because we want resources that they depend on and they can’t stop us from taking them.

And we aren’t *that much* smarter than chimpanzees. An adult chimp is, in many respects, smarter than a human toddler; contrast with, say, a toad.

Even this relatively small gap is an insurmountable advantage. To a chimp, “a spear” is “an incomprehensible exploit in physics”. Let alone a gun. We have *satellites* that beam *lasers* at each other.

AI math is already incomprehensible; how far behind is physics? Biology? What will their spears look like to us? Their guns? Their aircraft carriers and cruise missiles?

Will we, too, simply be too stupid to prevent an uncontrolled and unaligned superintelligence from coming into the world? Worse: will we do it on purpose? Chimps didn’t have a “just stop building that” option. It would be so embarrassing to squander it.

I love artificial intelligence. I think it’s the most beautiful technology we’ve ever built. Even as a child, I sought to create code that could exist in an ecosystem — toy ecosystems, then. Now, we have code that can maintain homeostasis in messy reality. I have the same dreams. I get it. Art, not toil.

But it is abundantly clear that this is *not safe*.

Right now, no one knows how to robustly ensure that an artificial intelligence has *any particular goals whatsoever*. No one is even claiming to know how to do this.

And, until this is so, every time we make a smarter model, we’re rolling the dice. Because *if* it has goals that are not ours, and *when* — not “if”: we are actively pushing in this direction and pouring unprecedented capital into it — *when* it becomes smart enough to invent its version of the spear or gun with the materials at hand, we already know how this plays out.

Many of us will think, “wait, you can’t do that with just a computer” in much the same way a chimp thinks, “wait, you can’t do that with just a stick”. Physics doesn’t care if we understand it.

So: yes, regulate this. Stop making the models smarter until we figure out how to do that safely.

We are playing with fire.
Dayah Dover Verified account @DayahDover Nov 6
I use AI every single day. I love it, AND I want it regulated!

A ban isn't the answer. You can't un-invent this, and banning it here just hands the frontier to whoever cares least. But oversight? Absolutely (and urgently)

METR's independent review found AI is now speeding up its own progress, about 2x overall. Ryan Greenblatt at METR estimates AI is accelerating AI software research by roughly 4.5x, and called it “a big update towards ASI within 1–2 years”. Progress is starting to feed itself

If progress can come from software alone, hardware limits stop being the brake people assumed. And AI is now writing a lot of that software itself… the time for oversight is now, while we can still shape it

The companies building this are the ones asking for rules. Loving a technology and wanting guardrails on it aren't opposites. It's how you take something *this* powerful seriously

The replies to this post are wild. Why is this such a controversial take?
View quoted post on X
13 replies 32 reposts 223 likes 16K views View on X
Jon 🌌 Verified account @jon_vs_moloch
IT'S NOT DENSE?!
Bro we're so fucking back
If you're a local model enthusiast this is new SOTA.
Thinking Machines Verified account @thinkymachines 8h
Some of you thought that Inkling 2 Small was too big at 594B parameters. We heard you: Inkling 2 Mini is a looped transformer with just 26B parameters, small enough to run on your laptop, powerful enough for agentic work with a score of 20.1 on AA Index.
thinkingmachines.ai/news/inkling-2…
View quoted post on X
Jon 🌌 Verified account @jon_vs_moloch
What
Thinking Machines Verified account @thinkymachines 8h
Some of you thought that Inkling 2 Small was too big at 594B parameters. We heard you: Inkling 2 Mini is a looped transformer with just 26B parameters, small enough to run on your laptop, powerful enough for agentic work with a score of 20.1 on AA Index.
thinkingmachines.ai/news/inkling-2…
View quoted post on X