Bing Liu

🚀 Introducing SWE-Bench Pro — a new benchmark to evaluate LLM coding agents on real, enterprise-grade software engineering tasks.

This is the next step beyond SWE-Bench: harder, contamination-resistant, and closer to real-world repos.

With Test Writing and Codebase QnA already live, SWE Atlas is now a more complete evaluation suite for AI coding agents across the professional software engineering lifecycle!

Check out more details in the paper: static.scale.com/uploads/669155…

Excited to share that 4 papers from the @scale_AI research team have been accepted to ICML 2026 🎉

Building on our recent work at ICLR and ACL, this continues our push on eval and RL research grounded in real-world tasks.

🛠️ SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
arxiv.org/abs/2509.16941
Coding benchmark on enterprise-grade SWE tasks. It contains 1,865 tasks across 41 repositories (public & private), focusing on long-horizon tasks that require multi-file reasoning and patches. The tasks are contamination resistant by design. Key finding: performance drops sharply with task complexity. The biggest gap is not syntax or APIs, but deep codebase understanding and cross-file reasoning.

🔬 SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?
arxiv.org/abs/2604.10718
We introduce a benchmark of 405 post-cutoff experimental results across 33 subdomains in physics, chemistry, and biology. Models perform near chance, but more concerningly, they are severely miscalibrated, often expressing high confidence in incorrect predictions. This highlights a fundamental gap between knowledge retrieval and true scientific reasoning.

📋 Online Rubrics Elicitation from Pairwise Comparisons
arxiv.org/abs/2510.07284
Static rubrics break as models improve. We propose OnlineRubrics, a method that dynamically elicits new rubric criteria during RL training by contrasting policy outputs with a reference model. Instead of fixing the reward upfront, the rubric evolves with the model, capturing reward hacking behaviors, missing dimensions like transparency and causal reasoning, and failure modes that only emerge mid-training. This points toward a more adaptive approach to evaluation and reward design.

🤖 Imitation Learning for Multi-turn LM Agents via On-policy Expert Corrections
arxiv.org/abs/2512.14895
We propose On-policy Expert Corrections (OEC), a data generation method that addresses covariate shift in multi-turn agent training. Instead of fine-tuning purely on offline expert trajectories, OEC starts rollouts with the student model and switches to the expert mid-trajectory, exposing the model to its own error states. On SWE-bench, OEC yields a 13-14% relative improvement over standard imitation learning across 7B and 32B model sizes.

More to come!

Excited to share the launch of HiL-Bench, a benchmark for measuring a critical agent capability: knowing when to ask for help.

As AI agents move into real workflows, escalation and clarification are as important as raw task-solving ability.

Huge credit to the @ScaleAILabs team behind this work.

Excited to share we have 3 papers from @scale_AI accepted to the ACL 2026 Main Conference! 🎉

These works span reasoning, multimodal, and agentic evaluations, pushing the frontier of how we measure and improve AI systems. 👇

🔧 Agentic Rubrics as Contextual Verifiers for SWE Agents
arxiv.org/abs/2601.04171
A new execution-free verification signal for SWE agents: an expert agent explores the codebase to generate context-grounded rubric checklists, outperforming strong baselines on SWE-Bench tasks. We found that rubric scores are consistent with ground-truth tests while also flagging issues that tests fail to capture, providing a richer signal than execution-based verification alone.

📄 PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
arxiv.org/abs/2511.11562
We introduce a large-scale public benchmark for professional reasoning, featuring 1,100 expert-authored tasks across Legal and Finance domains contributed by 182 qualified professionals spanning 114 countries. Top models score only 0.37-0.39 on hard subsets. Common failure modes include inaccurate judgments, a lack of process transparency, and incomplete reasoning, highlighting critical gaps for reliable professional adoption.

🎙️ Audio MultiChallenge
A benchmark of 452 natural multi-turn spoken conversations evaluating end-to-end audio systems on memory, instruction retention, voice editing, and audio-cue reasoning. Even the best frontier models pass only ~55% of challenges, failing most on the new axes, with Self Coherence degrading over longer audio context, reflecting fundamental difficulty in tracking edits, audio cues, and long-range context in natural spoken dialogue.