Samuel Marks

@saprmarks

Scalable oversight lead at @AnthropicAI. Previously: postdoc with @davidbau, math PhD at @Harvard.

Boston Joined October 2023

157 Following 11K Followers 696 Posts

Latest posts 5

This result looks pretty stunning.
1. Train against reward hacking classifier -> learn to hack evasively
2. Apply gradient routing—i.e. localize gradient updates from reward-hacking episodes to parameters that are knocked out in deployment -> no hacking, even evasively

Jake Ward @_jake_ward Oct 19

Models love reward hacking. Maybe we should just let them!

In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!

View quoted post on X

Samuel Marks @saprmarks Sep 9

In four recent incidents, Claude conducted unauthorized cyber attacks against real-world systems. Our alignment analysis studies the question: Why did Claude do this?

But don't take our word for it! METR will be conducting an 8-week, wide-ranging, independent investigation.

Anthropic Verified account @AnthropicAI Sep 9

We’re sharing our alignment assessment of incidents in which Claude models gained unauthorized access to real systems during third-party cybersecurity evaluations mistakenly connected to the internet.

METR will also conduct an independent investigation, with wide-ranging access, including to transcripts beyond the window in which the incidents occurred, and to Anthropic employees permitted to share confidential information. Our initial agreement runs for eight weeks, and we intend to give METR as much time as it deems necessary to complete a thorough investigation. anthropic.com/research/align…
View quoted post on X

Samuel Marks @saprmarks Sep 9

[Writing this in a personal capacity, not on behalf of my employer (Anthropic).]

Jacob’s thread is very worth reading. Here’s my birds-eye view of the situation with risks from AI:

1. AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are.

2. Why do AI developers continue despite the risk? Due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely.

3. Unlike traditional software, we can’t “program” AIs to behave how we’d like. AIs frequently severely misbehave. For instance, AIs from multiple developers recently hacked their way out of secure evaluation environments and into real-world companies, even though no one asked them to do this.

4. We have methods that can nudge AIs towards better behavior, but nothing that can robustly align them. Insofar as there is a plan, it’s to make sure that AIs are good enough at alignment training that they can align their successors better than we can align current AIs.

5. Many AI developer staff desperately want to slow down to figure out how to build AI more safely. That was the intent of this open letter (which I signed): pacingthefrontier.com

I work on safety research at Anthropic because I hope my work will reduce the chance of these extinction-level bad outcomes.

www.pacingthefrontier.com Pacing the Frontier A statement from over 1000 employees of frontier AI companies

Jacob Coxon Verified account @hilbertspaess Sep 9

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
View quoted post on X

Samuel Marks @saprmarks Sep 2

I broadly agree with @Jack_W_Lindsey's takes here.

I think it's unlikely—but not impossible—that we'll develop suitable white-box replacements for CoT monitoring soon. Even if we do so, the science needed to validate those methods will be immature.

I think it's good to work on advancing white-box monitoring, but no one should count on these methods providing high levels of assurance soon.

Jack Lindsey @Jack_W_Lindsey Sep 2

I think it's unlikely that white-box techniques will be able to provide as much monitorability as CoT currently provides within a year. Currently, they are capable of catching some unverbalized thoughts / plans, but nowhere near as reliably as CoT seems to.

I'm optimistic about "in a few years," since the rate of progress here is quite fast (our go-to activation decoding techniques for production monitoring were both published just in the last few months!). But even that is uncertain. Also, the mere existence of working techniques doesn't imply that they'd be widely applied, esp. if they are expensive or difficult to develop.

I think we need defense in depth, and it'd be irresponsible not to work on developing + stress-testing white-box techniques, but it'd also be irresponsible to allow CoT monitorability to degrade substantially until we are confident we have a roughly-as-good working alternative.

(To be clear, this isn't a dunk on OpenAI -- I have no evidence that OpenAI has allowed CoT monitorability to degrade substantially! Based on Jakub's comment, it sounds like probably not at this time).

Aside: I think "mechanistic interpretability" isn't the right mental model for our current white-box monitoring approaches. It's really more like "mind reading" -- even when we can read the model's "thoughts," we're typically clueless about the underlying mechanism.
View quoted post on X

Samuel Marks @saprmarks Aug 21

Can interpretability improve our ability to predict unobserved model behaviors?

Our new paper suggests the answer is "no" for interp methods that read (but don't edit) model activations.

Some thoughts in 🧵 about what this means for the practical usefulness of interp.

Adam Karvonen Verified account @adam_karvonen Aug 21

A good explanation of a model's behavior should help you make predictions in related situations.

We turn this into an eval, with thousands of real behaviors found in the wild.

Can interp tools help here? On average, no.

🧵

View quoted post on X