Arnab Nandi ⭐️

@arnabdotorg

human-in-the-loop data infra | Prof, Ohio State CS | Co-founder, @thesteamfactory, @hackohio, @mobikitinc (acq.)

Columbus, OH arnab.org Joined March 2008

3.6K Following 3.1K Followers 300 Posts

Latest posts 5

Surprising results from @OpenAI 's new Terra release. It shouldn't be this good! Also separately, adding Clef into the benchmark reveals a new Pareto frontier: decisions are a very cost effective pathway for physical world understanding! (work led by @PerraultAndrew )

Arnab Nandi ⭐️ @arnabdotorg Oct 28

@skalskip92 @victornunez Thank you for confirming this! We are seeing the same thing with RealityBench. Still behind on our acceptability metrics, but very surprised to see Argon / Fable / Astra performance at such a strong price point. relantic.com/realitybench

View quoted post on X

Arnab Nandi ⭐️ @arnabdotorg Oct 14

Sharing something we've been working on! @relanticai's RealityBench is a portfolio of 3D worlds which we used to evaluate 30+ frontier models on physical work tasks, incl. Argon, Astra, Fable, and Muse Wonder. Because these are 3D worlds, we can "bend" reality for new GT data.

Relantic @relanticai Oct 14

The worlds are simulators, not image sets: built to extend to new tasks. Early access is on Hugging Face and relantic.com/realitybench Check out our video below.

View quoted post on X

Arnab Nandi ⭐️ @arnabdotorg Oct 14

Excited to share what we've been working on! @relanticai #RealityBench uses 3D worlds to evaluate 30+ frontier models on physical work understanding.

Andrew Perrault @PerraultAndrew Oct 14

We created simulated environments in Blender for benchmarking VLMs on real world physical work perception and reasoning tasks. Think construction and maintenance of houses and cars.

View quoted post on X

Arnab Nandi ⭐️ @arnabdotorg Jan 20

.@cbinnig should have titled this paper "welcome to the jungle"

"real-world DBs = highly normalized, even simple NL questions to require 4 joins/query, vs Spider (0.5J/query) ...state-of-the-art models experience up to a 70% drop in execution accuracy" openreview.net/pdf?id=NZLm4vz…

Arnab Nandi ⭐️ @arnabdotorg Jan 18

Couldn't agree more. Recent "AI interaction" paradigms unlocked a huge wave of productivity. However, it's time to reclaim what we left behind.

David Verified account @DavidSHolz Jan 17

gosh, the purely linear, non-differential, non-branching, non-layered UI paradigms around LLMs is such a curse. it's like we've thrown out every piece of cognitive infrastructure we've built since the invention of two way oral communication
View quoted post on X