Latest posts 5
Benchmarks are a public good. Theres should be a foundation for the purpose of measuring intelligence so the incentives for making benches and selling data do not misalign
interesting things are happening with the AI benchmark companiesView quoted post on X
let's say you made a new benchmark for AI. fantastic!
there's only two possibilities:
1. your benchmark sucks/doesn't represent model performance because there's an issue with the task/harness, in which case it should be ignored
2. the benchmark is actually really good, and you made a bunch of genuinely hard tasks pushing frontier model ability
in the case of 2. then congrats, you've also accidentally become an RL task company, because there's no difference (at all) between frontier benchmarks and frontier RL tasks.
you realize you can sell these tasks for a LOT of money, and start scaling. oh no, you're Mercor!
also now that you've sold similar tasks to some labs (probably non-exclusive contracts) so your benchmark is now saturated and meaningless, also model ability has improved fast enough that it probably gets saturated anyway even if you don't sell the tasks
so remember, when it comes to frontier model benchmarking, there is only dogshit and Mercors, and nothing in between!
ever wonder why Mercor/Handshake/etc publish benchmarks all the time? well only they can. because they're actually building tasks that push frontier models, and the best way to show that is to make benchmarks and release the results.
to be clear, this applies to model benchmarks only. if you're benchmarking sandboxes/web search/scraping/gpus/what have you this rule does not apply. but even then if you're good at that why not dip your toe in the infinite money pool?
(some exclusions apply, like preference data which doesn't require task creation, hence Arena)
mirendil mentioned :D
Thomas Kurian: 'In addition to our work with Anthropic, both directly and via Broadcom, and public references customers like Apple, who are scaling pretraining of their foundation models on the latest generations of TPUs, we're also working with new frontier model companies including Mirendil and Physical Intelligence.'View quoted post on X
My friends can attest, Ive been wanting to push boundaries of harnesses for a while.
When I was introduced to mirendil I was immediately intrigued that a small neolab was planning to compete in the big leagues, in the domain of agentic intelligence. It sounded insane, ambitious, and a great place to create the next generation of ai tooling.
Like how a noun needs a verb, no matter how powerful the model gets we will always need a harness. The space isnt quite solved but its close. Harnesses are all aligning to the same few architectures (more to come for how we built ours). Each generation of model has made it more clear what architectures are actually required.
Our goal was fundamental and simple. Give the model agency. Give the human freedom.
Im excited to introduce Lor, our new product built from first principles to be an always-on, multi client, multi environment delegator for coding and experiments.
We're opening up our beta. I'd love to chat with anyone pushing the limits of engineering and discovery.
Message me on any platform or sign up here if you're interested in trying it out
mirendil.com/lor-beta/
When I was introduced to mirendil I was immediately intrigued that a small neolab was planning to compete in the big leagues, in the domain of agentic intelligence. It sounded insane, ambitious, and a great place to create the next generation of ai tooling.
Like how a noun needs a verb, no matter how powerful the model gets we will always need a harness. The space isnt quite solved but its close. Harnesses are all aligning to the same few architectures (more to come for how we built ours). Each generation of model has made it more clear what architectures are actually required.
Our goal was fundamental and simple. Give the model agency. Give the human freedom.
Im excited to introduce Lor, our new product built from first principles to be an always-on, multi client, multi environment delegator for coding and experiments.
We're opening up our beta. I'd love to chat with anyone pushing the limits of engineering and discovery.
Message me on any platform or sign up here if you're interested in trying it out
mirendil.com/lor-beta/
mirendil.com Join the Lor waitlist — Mirendil Join the waitlist for the Lor private beta. Meet Lor.
We’re building Lor to do the work of an entire AI lab. Today, its agents write our code, run experiments, and iterate on the results for days, on any machine, hundreds at a time.
Now in private beta ↓View quoted post on X
