Fazl Barez (@FazlBarez)

Pinned post

New paper❗

Does understanding how AI models work actually make them safer?

This assumption often underlines interpretability research, but different subfields operationalise this intuition in very different ways. Our survey of mechanistic interpretability (MI) for safety provides an opinionated guide to the field of MI, detailing conceptual approaches and techniques under this broad banner and how different types of work relates to the mission of making AI model safer. 🧵

0:22

Fazl Barez @FazlBarez 5d

I was pleased to debate at the Cambridge Union last night. We lost the motion :( but I'm glad I got an opportunity to raise awareness of AI safety and the erosion of democratic norms.

I'm a technologist at heart! I believe technology has been a force for good but it has also eroded some of the freedoms we fought so hard to secure -- and even worse this has happened in so called ultimate liberal democracies

As I argued last night, democracy is often thought in terms of parliament and voting doesn't have to vanish for a state to become less democratic

The question now is how we can use technology especially AI to strengthen rather than undermine democratic norms, transparency and accountability etc

I think we bare responsibility to develop technology in a way that preserves the freedoms that those before us made the ultimate sacrifices to secure. Those who come after us will probably ask why we didn't build responsible and safe technology not why we didn't build the most capable technology. lets do this for ourselves, for our children and for those that may come after us!

We still have time. Democracy is worth preserving. Let's hope humanity collectively makes the right decisions

Fazl Barez @FazlBarez 5d

This looks very impressive! Good job Thomas and co at goodfire

Thomas Fel Verified account @thomas_fel_ 6d

A science of post-training can be built on interpretability.

Models know when they reward hack, and we see it, yet training ignores it. We propose Mechanistic Shaping: using the model’s own concepts to write training rules.

In vision, DPO & RL, and its effects on monitoring 🧵

View quoted post on X

Fazl Barez @FazlBarez 6d

AI was a mistake

Pass it on.

David Krueger 🦥 ⏸️ ⏹️ ⏪ Verified account @DavidSKrueger 6d

AI was a mistake

Pass it on.
View quoted post on X

Fazl Barez @FazlBarez 6d

This is bad news!

Marcus Williams Verified account @Marcus_J_W 6d

🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.

To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.

View quoted post on X

Fazl Barez

Latest posts 5