Twitxy Saved

Jorge Ordovás

Verified account This account is verified. Learn more

@joobid

Product Lead, Spanish AI Gigafactory @Telefonica | Co-founder & CEO @Kairos_Tek | Lecturer @IEuniversity, @Afi_es, UCM, UPM and UCLM | AI speaker & communicator
European Union linkedin.com/build-relation… Joined April 2011
86 Following 5.8K Followers 44K Posts

Latest posts 5

Jorge Ordovás Verified account @joobid
AI Socratic returns to Madrid in November!

A monthly gathering for professionals, researchers, founders, educators, and developers in the field of AI.

We host Socratic dialogues on the most pressing topics in AI: new models and tools, research papers, the alignment of general AI, philosophy, geopolitics, and much more.

If you’d like to be part of this community, join us on Thursday, September 26th, at 17:30. Sign up here (spots are limited!) 👇
Jorge Ordovás Verified account @joobid
Most teams check their AI agents by asking another LLM to grade them.

The catch: that grader is slow, expensive, and can give the same answer two different grades.

@adlrocha tested a cheaper alternative. Here’s what he found.

At Baselight, the startup he co-founded, grading was so costly they couldn’t run it on every code change. Only on some changes, plus a nightly run.

Worse: when a test failed, they couldn’t tell if the agent got worse or the grader changed its mind. Finding out meant paying for the evaluation again.

The alternative: decision models.

They don’t write. They pick an option (yes/no, A/B) and say how confident they are. So you rewrite your evaluation as closed questions, like “Is the answer based on the data the agent found?”

The numbers:
- LangChain: matched the human reviewer every time. $0.34 vs $28.17 with a regular LLM
- Alfonso’s test on code reviewers: same verdict as the LLM grader 9 times out of 10, 8.6x faster, half the price

His take: decision models for anything you can ask as yes/no. LLM graders for anything that needs an explanation.

And a warning: a grader that always gives the same grade isn’t necessarily right. In the 2 case out of 10 where they disagree, who’s right?

We debate questions like this every month at AI Socratic Madrid. Join us 👉 luma.com/aimadrid
luma.com AI Socratic - Madrid · Events Calendar View and follow events from AI Socratic - Madrid on Luma. AI Socratic Madrid brings together engineers, researchers, founders, product managers and AI practitioners to share, question and learn....
adlrocha Verified account @adlrocha Oct 18
LLM-as-a-judge always pushed my agent evals out of CI: too slow, too expensive, and too noisy. At Baselight a red test could mean the agent got worse or the judge changed its mind, and the only way to know was to pay for another run.

In part 2 of the decision models series I replaced the LLM judge in a code-review eval with OpenAI's Decisions API:
* 90.7% agreement with the generative judge
* 186 ms per verdict, 8.6x faster
* About half the cost (not the 80x LangChain saw, and I explain why)

What changes when judging is nearly free: you run the whole suite on every commit, judged evals start behaving like ordinary tests, and the model's confidence becomes a signal of how flaky your eval is.

I also cover where I'd still pay for an LLM judge.

Link to post: adlrocha.substack.com/p/adlrocha-dec…
View quoted post on X
Jorge Ordovás Verified account @joobid
En este artículo cuento lo que aprendimos en @Kairos_Tek al probar distintos modelos abiertos (open source) con tres casos de uso reales de una gestoría.

Y por qué la elección final no depende de los rankings públicos de estos modelos, sino de las pruebas que construimos específicamente con el trabajo de nuestro cliente, que no tienen por qué dar el mismo resultado.

No hablo de teoría, sino de una experiencia real que ha permitido a una gestoría de 20 personas integrar, por fin, la inteligencia artificial en su trabajo del día a día sin sacrificar calidad ni comprometer la privacidad de la información que gestiona.

En Moneyball (película que siempre recomiendo) Peter Brand convence a Billy Beane de que el objetivo de un equipo no es fichar jugadores, sino comprar victorias.

Con los modelos de IA pasa exactamente lo mismo: no compras un modelo, compras aciertos sobre tu trabajo. Y los aciertos solo se ven midiendo lo que necesitas saber para decidir.

Si quieres saber cómo elegir el modelo que de verdad necesitas para tu trabajo, te lo cuento en este artículo de la newsletter "Entre gigafactorIA y gestorIA".

linkedin.com/pulse/abiertos…
www.linkedin.com Abiertos o de frontera: cómo elegir el modelo que acierta con tu trabajo Voy al grano. En este artículo cuento lo que aprendimos en Kairos Tek al probar distintos modelos abiertos (open source) con tres casos de uso reales de una gestoría.