Zayne Sprague

SkillFactory will be at ICLR this year!

We study how self-distillation can create synthetic data that primes models for RLVR through SFT.

We built a recipe for teaching your model new capabilities it doesn’t have yet, matching and sometimes outperforming teacher distillation.

Neat project + post. This line of work is making me think of harnesses as transient reasoning scaffolds. Reminds me of earlier “XoT as intermediate structure” work (SatLM, Program-of-Thought, Tree-of-Thought, etc.), but now in an agentic regime with much more room for optimization.

I keep wondering about this idea of meta-optimization: how broad should we expect that optimization to get? Should a meta-agent mostly do local search over prompts/tools/hyperparameters, or should it sometimes pursue riskier, longer-horizon interventions? Wilder example, but if we pointed something like AutoResearch at "coding", should we expect it to rediscover higher-level workflows or abstractions akin to Claude Code?

My guess is that “meta” agents, or even deeper recursive optimizers, will tend to favor local improvements over sweeping pipeline redesigns. Very targeted and precise changes, even when broader features might help in the long run. Measuring that “meta-scope”, where do agents spend their time optimizing these harnesses, seems worth studying.

Claude Code with Agent-Deck really helped me manage my sessions - one GUI in one terminal session for all CC sessions. Conductors (Claude running other Claude sessions) are pretty neat too. Communication overhead / reward hacking for long experiments is still challenging tho, even with all the skills, superclaude/superpowers, and ctx engineering hacks - though they do help in their own way. github.com/asheshgoplani/…

Feels directionally right after running AutoResearch for a bit. I set an ambitious target, 0.75 val_bpb & keep 50mil params. Didn't expect it to hit it but I did expect some far out heavy architecture changes. Across 500+ proposed experiments, it mostly hill-climbed through hyperparameters and familiar tricks.
This feels like an optimization vs. discovery gap: Like a bias of LLMs to exploit what they already know as opposed to pursuing genuinely novel ideas, especially ones that are "wild" -- i.e. require long horizons to validate, go against priors, etc. (even when ideas like this are warranted/necessary)