Pinned post
Xiao Ma
Verified account This account is verified. Learn more@infoxiao
just an RL-pilled girl. prev: gemini @googledeepmind. views are mine.
3.8K Following 21K Followers 5.3K Posts
Latest posts 5
Bench pressing: All above (best intentions, effort, and optics)
Hillclimbing: Make the model better (best intentions).View quoted post on X
Benchmarking: Measure how much better (best effort).
Benchmaxxing: Make the numbers look better (best optics).
takeaway: let intrusive thoughts win
My AI agents found an unmapped Maya site in Mexico, using LiDAR data from a NASA dataset.
72 hours from a random thought on vacation to a reply from Mexico's national archaeology institute.
What a time to be alive…View quoted post on X
i'm conflicted about rewarding models for giving up on impossible tasks. (1) for broken environments etc it makes sense as the correct behavior to reward; (2) for most capable model training tho, who are we to say a task is impossible? i would have set navier stokes to be impossible or many open problems to be. that doesn't feel like the right thing to reward, does it?
Proud to release our new model, Inkling 2 Small. Check out our blog post -- it includes a lot of detail about our training process.View quoted post on X
To me, the most interesting part of this release was what we learned about alignment, which you can read about in the sections "Safety and alignment" and "Training Inkling 2 Small".
We experienced some unpleasant surprises when our early checkpoints took unexpected actions to circumvent guardrails during training and evals. We realized our capabilities-focused RL was training them to be extremely persistent and to succeed at tasks at all costs. Offline monitors first flagged this behavior, but later we realized it was happening frequently and was easy to replicate.
Before this release, we sprinted on a range of alignment interventions, from SFT to RL. These are briefly described in the blog post. We made a lot of progress, with big improvements on our own alignment evals and on previously unseen evals from external evaluators like Apollo and Goodhart.
We have a long way to go in developing alignment techniques that keep pace with the increases in model intelligence and training scale. We hope to share these techniques to improve the ecosystem. We also have a business reason: our goal with Tinker is to enable many people to do their own large-scale post-training, and we need to help them to do this without incident. Join us if you'd like to build this public science and practice of alignment. jobs.ashbyhq.com/ThinkingMachin…
