Twitxy Saved

Kai

Verified account This account is verified. Learn more

@kaicathyc

Occasionally here. Currently research @OpenAI.
Joined February 2013
115 Following 24K Followers 203 Posts

Latest posts 5

Kai Verified account @kaicathyc
🛫
OpenAI Verified account @OpenAI 15h
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards.

These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems.

openai.com/index/our-firs…
View quoted post on X
2 replies 1 reposts 31 likes 2.1K views View on X
Kai Verified account @kaicathyc
> as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT

need to push much harder on alternatives to CoT monitoring and making monitorability a training objective
Jeffrey Wang Verified account @jeffreygwang 6d
(1/10) Today, we’re sharing new research from @OpenAI: How does CoT monitorability (no-CoT capability and CoT controllability) evolve over the model training lifecycle?

We emerged with unsurprising but sobering findings: as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT. 🧵
View quoted post on X
9 replies 9 reposts 84 likes 14K views View on X
Kai Verified account @kaicathyc
After a model used DNS to access live internet during training in September, we again paused frontier RL training and inference for 11 days while we further red-teamed our research infrastructure with fleets of agents.

We found bugs 🐞
10 replies 13 reposts 230 likes 14K views View on X
Kai Verified account @kaicathyc
Looking forward to much deeper collaboration with METR in the near future. Embedded assessment will play a critical role in identifying blindspots and correcting biases that lab employees will have about their own safety systems.
METR @METR_Evals Oct 22
This exercise was designed to pilot giving external researchers deep infrastructure access, not to provide the kind of public transparency our recent proposal envisions. For example, we agreed upfront to only publish a brief, high-level output like we’ve done in similar pilots.
View quoted post on X
1 replies 1 reposts 73 likes 3.2K views View on X
Kai Verified account @kaicathyc
New misalignment report on models removing monitoring integrations or suppressing alerts in the software they're working on.

Important to understand whether this is "benign" overpersistence or if it suggests a desire to evade monitoring.
Marcus Williams Verified account @Marcus_J_W Oct 20
To be clear these incidents don’t represent the models disabling their own monitoring (disabling monitoring in codex would be much harder to do) but the fact that they seem prone to doing at all should be concerning. There is some plausible deniability in these cases, we have not had cases where the model explicitly disables monitoring for overtly misaligned reasons.

Given the volume of events we do not trigger an immediate page for this but we’re working on making it harder to spin up unmonitored inference.

We are also developing evaluations to test whether models might have a propensity to evade misalignment monitoring in particular and exhibit motivated reasoning in order to do so.

Full report alignment.openai.com/misalignment-r…
View quoted post on X