Genta Winata

www.nature.com A benchmark of expert-level academic questions to assess AI capabilities Nature - Humanity’s Last Exam, a multi-modal benchmark at the frontier of human knowledge, is designed to be an expert-level closed-ended academic benchmark with broad subject coverage.

Center for AI Safety @CAIS · Feb 2

Last week, Humanity’s Last Exam was published in @Nature.

In just over a year, model scores on HLE have risen from under 5% to nearly 40%.

Thank you to @scale_AI and the 1000+ HLE co-authors for helping policymakers and the public track these rapid advances in AI capabilities.

View quoted post on X

patrick Verified account @patrickamadeus_ · Dec 23, 2025

Craving holiday-themed paper? Say less🎄

Turns out, Vision Language Models are Confused Tourists ✈️😵‍💫

We show that adversarially induced cultural scenes significantly impair VLM cultural comprehension and trigger potential bias

#NLProc #multimodal #robustness
/thread 🧵(1/8)

View quoted post on X

Sharon Li Verified account @SharonYixuanLi · May 12, 2025

NeurIPS has a record-breaking number of 25,000 submissions this year already. It's crazy but not entirely unexpected.

Here is the scaling over the years, with a conservative projection.

How do we even run a conference and review at this massive scale?

View quoted post on X

Yi Tay Verified account @YiTayML · May 8, 2025

Sharing a pretty interesting story about how two research projects can end up with a very different outcome despite almost similar type of “work” being done, just because of research motivation and taste. Here, I was on the side that got rekt so I thought I would share my lessons and introspections. 😂

This was some years ago when people were scaling up many finetuning tasks for training LLMs by collecting many academic datasets. We all know the famous Flan paper by @_jasonwei which was one of the pioneering “instruction tuning” works. However, not many people know but there was another more low-key effort within Google that did almost all the same work but was pretty under-hyped because it was framed wrongly. That was the exT5 work that my AI resident @VAribandi and myself drove (i take all the blame though).

Despite so much parallel "almost identical" work being done and tedious efforts in collecting all those tasks and processing them, the framing for exT5 was off and the research motivation was much less sexy compared to Flan. Note: this was before the time I joined the Brain team and Jason and I were on different teams. I also joined the Flan-2 project much later.

I think it was around this time when many people had very similar intuitions (scale up tasks! 📈) but Flan turned out to be the most successful back then. Introspecting a little I found out the key mistake we made was selling it as a multi-task/transfer thing while Flan sold the zero-shot capabilities of generative LLMs. The framing of a T5 variant was also a big blunder on my side.

I dug around even more about the history and found out that @MaartenBosma and @quocleix really influenced the idea of improving zero-shot capabilities, which IMO really made Flan shine and so impactful. Also note that this was the era where the standard practice was pretraining and then finetuning models to do very well on a few targeted downstream tasks. I think the Flan team had more foresight into the future than we did. Moreover, I was also too stuck in this narrow “present-moment” of that time. I was focused a lot on architecture research on t5-style models so i completely overlooked the new capabilities game.

I took away a lot from this. i) who you talk to/brainstorm in research matters a lot, ii) small details in framing change the impact and reception of the work significantly and iii) environment matters a lot. I consider myself pretty decent/good in framing research and have a pretty solid research taste but as a researcher one can also be dead wrong sometimes. And ExT5 was one of the moments where I was like darn, we got rekt. Should have worked with @_jasonwei and @quocleix much earlier.

This was mostly inspired by a convo with @HuaixiuZheng recently where we discussed how Jason was so good at selling/marketing research and we spend some time reminiscing how our exT5 project got rekt completely despite doing very similar things. To be fair, it was not because of Flan, but because the work undersold itself and our research motivation was lacklustre.

Today ext5 has like 200+ citations. Not a complete wreck but still a far cry for anyone to get excited about. Flan has like few thousands the last I checked.

People don’t share behind the scenes of research that much so I thought this might be interesting too! I learned a lot from @_jasonwei from the time I worked with him about marketing his research but in 2022 when I got the chance I took the opportunity to get my revenge by destroying him 6-0 on competitive Pokemon so we're even now. 😏

View quoted post on X