Jake Mendel (@JakeMendel99)

I’d love to see a follow up investigation by METR assessing how much uplift alignment research is getting! I really hope it’s not getting less uplift than capabilities progress

Sara Price Verified account @sprice354_ 2d

I’m really excited about METR’s work to make more of this information public. Their results match my impression that Anthropic is seeing an uplift in AI research progress due to internal acceleration from AI. This rate of progress is one of the most important trends we are currently seeing especially when considering the amount of alignment and safety work we need to do to keep pace.
View quoted post on X

Jake Mendel @JakeMendel99 3d

I've always had substantially longer timelines than my friends at Anthropic — I've been concerned that they're in an echo chamber and believing the hype about their own models too much, and that they're underestimating bottlenecks and unknown unknowns. And there have definitely been specific occasions when they have overstated how much uplift they were getting from their models. Nonetheless, if METR is correct here (and I think they are among the most trustworthy sources on this topic), then my Anthropic friends have been much more right about timelines than me.

This report is also some evidence that a software-only singularity is plausible. I now think that we are more than 50% likely to have full blown superintelligence in the next 18 months, absent a slowdown. I need to call my family.

METR @METR_Evals 4d

Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development.

We are now publishing a report summarizing the results of this risk assessment.
View quoted post on X

Jake Mendel @JakeMendel99 4d

It increasingly seems to me like incentives in RL is a bigger determinant of model behavior than generalization from character training or pretraining. I wish more people thought about AI behavior in terms of what RL incentivises. Unfortunately, this mode of behavioural analysis suggests that there are some big alignment problems on the horizon

Jake Mendel Verified account @JakeMendel99 4d

Max is referring to conceptual arguments about challenges with avoiding cheating and lying that will arise in the future. I think the claim that for sufficiently capable AI systems it becomes very difficult to design RL environments that don't incentivize cheating is pretty hard to contest, and current empirical evidence doesn’t seem very relevant to it.

I think the main part of the story that could be contested by empirical evidence is whether incentivizing cheating inevitably leads to models actually doing the cheating.

One update I’ve made based on recent empirical evidence is that model behavior seems increasingly well predicted by reasoning about what behavior is directly incentivized by the reward function (this analysis is analogous to realist analysis in politics, and reasoning about generalising from character training or the pretraining prior is analogous to idealism). Realism seems to be increasingly predictive for more recent models, perhaps because there is more RL being done now.

Unfortunately the realist perspective does imply that we have some serious alignment obstacles ahead of us. I now think that they are slightly more likely to be difficult to overcome than I used to.
View quoted post on X

Jake Mendel @JakeMendel99 6d

People disagree about whether, if Anthropic unilaterally disbanded, an international agreement to pause AI would be more likely (because of the costly signal) or less likely (because the most worried actor would lose their seat at the table). What do you think?

More likely 127
Less likely 97
No change/see results 79

Final results

Jake Mendel @JakeMendel99 Oct 16

admirable

Elon Musk Verified account @elonmusk Oct 15

@AnthropicAI @METR_Evals SpaceX will probably do the same
View quoted post on X

Jake Mendel

Latest posts 5