Twitxy Saved

Swarnadeep Saha

@swarnaNLP

Research Scientist @AIatMeta (FAIR) working on co-improvement. Past: @Google PhD fellow @uncnlp. Gooner.
Seattle, Washington swarnahub.github.io Joined May 2014
830 Following 1.8K Followers 677 Posts

Latest posts 5

Swarnadeep Saha @swarnaNLP
Great work by @tomchen0 on a direction I'm excited about -- self-improving agents should also learn to optimize their (typically human-written) harnesses.

We discover these through meta-optimization (outer loop) & then distill the knowledge back into model weights (inner loop)!
Jason Weston Verified account @jaseweston 22h
Introducing: Self-Improving Memory 🧠✨📈

We let the model discover its own memory skills, then train them into its weights:

- Meta-optimizes memory from the model's own runs with large gains.
- Distilling these gains works too.

Blog post: 📝 facebookresearch.github.io/RAM/blogs/self…
🧵1/6
View quoted post on X
3 replies 1 reposts 19 likes 936 views View on X
Swarnadeep Saha @swarnaNLP
It’s been an incredible few months working with the team building Muse Wonder. Among other things, it brings frontier capabilities in personal superintelligence spanning many domains like legal, finance, and health.

Hope you like it!
Mark Zuckerberg Verified account @finkd Oct 12
Introducing Muse Wonder, our new family of frontier models with state of the art performance for personal and professional agentic work.
View quoted post on X
1 replies 0 reposts 10 likes 650 views View on X
Swarnadeep Saha @swarnaNLP
Great work by Seungone where he carefully studies the impact of humans in building hard benchmarks!

Yes, agents do most of the work but humans play a critical role too.
Jason Weston Verified account @jaseweston Sep 30
⏱️⚙️Introducing *AutoBenchmark* 📊🏁
- Creating benchmarks automatically
- Benchmarking benchmark creation
- Studying the role & impact of humans in the loop

Blog post: facebookresearch.github.io/RAM/blogs/auto…

Key takeaways:

1) We find human-agent collaboration gives big wins over agents alone
- fine-grained feedback in ideation stage crucial
- autobench can be used to measure this in the future with stronger agents

2) We show that it's possible to make *autoresearch benchmarks* for AI research using this recipe – full recursive improvement loop!

3) Important Ingredients: Autobenchmark creation works best with feedback from two sources: benchmark solvers + external verifiers (human+AI).
🧵1/5
View quoted post on X
0 replies 1 reposts 21 likes 1.6K views View on X
Swarnadeep Saha @swarnaNLP
My favorite bits:

1. By *meta-optimizing* rubrics, we learn *features* that make expert-human writing superior & thus RL against those rubrics can produce *superhuman* text too.

2. In an online setting, this co-improves both rubrics+model, so extends beyond this specific task.
Jason Weston Verified account @jaseweston Sep 28
Claim: we've solved the AI slop problem (!) 💩🧹✨

Blog post: facebookresearch.github.io/RAM/blogs/unsl…

🧵1/5
Key idea: take *expert* human writing and learn rubrics that find the gap between experts and models. Train with those rubrics.

We train with RL-XAR (RL with eXpert Aligned Rubrics) & see large performance gains on writing scientific paper sections, Pulitzer prize novel continuations and high quality Wikipedia pages.
View quoted post on X
1 replies 6 reposts 74 likes 9.6K views View on X
Swarnadeep Saha @swarnaNLP
Another expert-authored professional reasoning benchmark where Muse Spark has held the top spot for quite some time now.
Aakash Sabharwal Verified account @aakashsabharwal Sep 22
Leaderboard spotlight!

With @Muse on top of the app store, a special shoutout to @AIatMeta for Muse Spark 1.3. The @ScaleAILabs team worked closely with them on evaluations, training data, and testing.

Muse does well on tool use, multi-turn conversation, tutoring, professional reasoning, and end-to-end software engineering. Different tasks, but each asks a model to hold context across long horizon workflows, make judgments under uncertainty, and recover from partial failures.

PRBench is where that's clearest. Its tasks are written by domain experts and scored on rubrics that reward professional utility, not trivia-style correctness. Muse has held the top spot in finance and law since 1.1, and 1.3 extends the lead.
View quoted post on X