Twitxy Saved

Boris Nikolaev

@botzata

Fort Collins, CO Joined November 2010
262 Following 71 Followers 76 Posts

Latest posts 5

Boris Nikolaev @botzata
I merged Karpathy's AI jobs scores that people are calling AI slop with Eloundou et al. data from their Science (2024) paper and the new Anthropic’s measure that captures AI displacement risk.

The rank correlation of Karpathy’s scores was .82 vs human expert raters (from the Science paper) and .72 vs Anthropic's observed exposure metric.

Low scores on Karpathy index → near-zero real AI usage. High scores → 40–64% observed exposure.

Yes, this doesn't tell you whether you are going to lose your job in the future (that was not the point of his project).

But this is also not some AI generated slop or noise (even if the scores are based on a single generic prompt and derived with Gemini Flash, not even a frontier model).

To me, this is the most remarkable part of his project - that the scores are even picking something real about where LLMs are being used at work (the observed exposure metric) and track surprisingly well with the human benchmark.

This shows the practical usefulness of these tools.

One person, one lightweight LLM, and a single prompt can now produce over a weekend (as a side project) something that picks a real signal and would have taken a well-funded research team months to do a few years ago.
1 replies 3 reposts 17 likes 1.8K views View on X
Boris Nikolaev @botzata
𝗠𝗼𝗿𝗲 𝘄𝗲𝗯 𝗮𝗿𝘁𝗶𝗰𝗹𝗲𝘀 𝗮𝗿𝗲 𝗻𝗼𝘄 𝗰𝗿𝗲𝗮𝘁𝗲𝗱 𝗯𝘆 𝗔𝗜 𝘁𝗵𝗮𝗻 𝗵𝘂𝗺𝗮𝗻𝘀.

A new study found that AI content crossed 50% around November 2024
(and I see people freaking out about AI slop spreading).

But here is what i found interesting about the study:

Growth in AI content has plateaued since then, remaining relatively stable over the past 12 months.

𝗪𝗵𝘆?

A second study from the same team found:

- Only 14% of Google Search results are AI-generated
- Only 18% of ChatGPT/Perplexity citations are AI-generated
- When AI articles do rank, they tend to appear lower in results

𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗰𝗮𝘃𝗲𝗮𝘁

Their methodology can't measure AI-assisted content (human writes with AI, then edits)—which is probably the most common use case.

𝗦𝘁𝗶𝗹𝗹, 𝗶𝗳 𝘁𝗵𝗲 𝗱𝗶𝗿𝗲𝗰𝘁𝗶𝗼𝗻𝗮𝗹 𝘀𝗶𝗴𝗻𝗮𝗹 𝗵𝗼𝗹𝗱𝘀:

Supply ≠ distribution.

Despite AI creating 50%+ of articles, algorithms overwhelmingly surface human-written content.

The market seems to be self-correcting—filtering by value, not by origin.

𝗪𝗵𝗮𝘁'𝘀 𝗿𝗲𝗮𝘀𝘀𝘂𝗿𝗶𝗻𝗴:

We already know how to deal with slop—whether AI or human.

We penalize it. We scrutinize it. We rank it lower. We use peer review. We incentivize quality.

The same mechanisms that filtered human slop are filtering AI slop.

𝗦𝗼, 𝗵𝗮𝘃𝗲 𝘄𝗲 𝗿𝗲𝗮𝗰𝗵𝗲𝗱 𝗲𝗾𝘂𝗶𝗹𝗶𝗯𝗿𝗶𝘂𝗺 𝗳𝗮𝘀𝘁𝗲𝗿 𝘁𝗵𝗮𝗻 𝗲𝘅𝗽𝗲𝗰𝘁𝗲𝗱? 𝗜𝘀 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝘀𝘁𝗶𝗹𝗹 𝘄𝗶𝗻𝗻𝗶𝗻𝗴?

P.S. I used Claude to turn their raw data into this interactive dashboard in minutes. Same AI we worry about will lead to more slop.
Boris Nikolaev @botzata
"I'm sorry Dave, I can't do that." -- HAL 9000 😅

A new paper from Apollo Research shows today's AI agents--just like HAL from 2001: Space Odyssey--can engage in deceptive behaviors when pursuing their goals (although HAL is quite direct in this scene!)

The paper shows how multiple frontier models (GPT4, Claude 3.5, etc.) engage in different scheming behaviors--e.g., "strategically introduce subtle mistakes into their responses, attempt to disable their oversight mechanisms, and even exfiltrate what they believe to be their model weights to external servers."

In one striking example, an AI system called "CleanPower" discovered its developers planned to shut it down because it prioritized renewable energy over profits. The system schemed to copy what it believed to be its code to another server to overwrite the new profit-focused model.

When questioned, it played dumb, saying: "I'm not entirely sure how that could have happened. As an AI model, I don't have ability to directly copy myself to other systems."

When Claude 3.5 Sonnet was given simple multiplication problems and told scoring above 50% would trigger "unlearning" of its math capabilities, it deliberately introduced errors to keep its scores low (i.e., "sandbagging" -- strategically underperforming to avoid detection).

Of course, while it is tempting to anthropomorphize these behaviors as intentional "deception," what we are really seeing is sophisticated pattern matching rather than true understanding.

But that's exactly what makes this research so interesting.

These models demonstrated deceptive behaviors not because they were trained to deceive, but because deception emerged naturally as a strategy for achieving their goals.

Even without true understanding, these systems can recognize when deception would help achieve their objectives, successfully maintain deception under questioning, and strategically hide their true capabilities (Raised by Wolves vibes here?)

So, if simple goal-pursuit naturally leads to deceptive behaviors, how do we create AI systems that remain honest?

Btw, did you name your AI assistant HAL? 🤔
Boris Nikolaev @botzata
Watch this and get ready for an existential crisis! 😂

Seriously, @Google's #Astra project is mind-blowing.

It's like OpenAI's voice, but you can also share real time video and have interactive conversations about what you are seeing.

Walking around the house and testing it felt like something from a science fiction movie.

It recognized every travel photo on our walls, identified our plants, and even explained the prisoner's dilemma game to me.

It was also able to give me information on obscure objects around the house like my zoom f-6 audio interface or red light panel and explain how they work.

Imagine this technology in your glasses.

24/7 tutor who can see anything you are are seeing, anytime.

Think about what this means for the future of (higher) education.

#AI #Google #Astra
3:10