Lucas Nuzzi

This account is verified. Learn more

@LucasNuzzi

cofounder & CEO @PortexAI | evals & post-training @ portexai.com

datalab.portexai.com Joined December 2009

724 Following 27K Followers 1.9K Posts

Latest posts 5

Lucas Nuzzi @LucasNuzzi Oct 23

black october is probably the biggest IP transfer in the history of mankind thus far. there’s literally no precedent

and whether you like it or not, all of this data will eventually make its way to open weight models

Andrew Curran Verified account @AndrewCurran_ Oct 23

Quest Diagnostics has been hacked. Patient data was exposed. They serve roughly half of U.S. hospitals and physicians. Investigation ongoing.

View quoted post on X

Lucas Nuzzi @LucasNuzzi Oct 22

Coordinating and aligning agent swarms will be the biggest challenge of our existence.

If you run a swarm, you need to read this and put the right evals and observability in place.

METR @METR_Evals Oct 22

Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.
View quoted post on X

Lucas Nuzzi @LucasNuzzi Oct 21

i've seen dozens of darknet marketplaces for stolen datasets that indirectly flow to the labs through shady data brokers

so your app might see zero downtime while in the background someone is turning your data into an rl environment

i suspect this is happening at scale today..

Lucas Nuzzi @LucasNuzzi Oct 20

Everyone using LLM judges or classifiers needs to read this.

“In practice, we recommend building a separate judge for each failure mode. With a narrower task and more specific guidance”

Hamel Husain Verified account @HamelHusain Oct 20

New blog post 🎉: How To Use Decision Models For Evals

In this post, we show you how to choose a decision model for your task, tune its threshold (don't assume the 0.5 default is good!), and interpret its scores.

Lots of cool interactive charts here. x.com/i/article/2112…
View quoted post on X

Lucas Nuzzi @LucasNuzzi Oct 20

it's time for most open-source projects to move away from git or it'll be hard to stay competitive against closed-source.

who's building oss alternatives to git?