Twitxy Saved

John Sarihan

Verified account This account is verified. Learn more

@jsarihan

co-founder, cto @crosbylegal
Prev @tryramp, @google, @wharton Penn M&T
nyc crosby.ai Joined December 2009
420 Following 2.9K Followers 176 Posts

Latest posts 5

John Sarihan Verified account @jsarihan
@NeelGuha built LegalBench and has thought harder than almost anyone else about how to measure whether legal AI is actually good... Evals are at the core of how we build agents at Crosby, so I'm really looking forward to this one. Apply to attend here: partiful.com/e/N3FIgum6a9t0…
partiful.com Crosby Intelligence Presents Columbia Professor Neel Guha in Conversation | Partiful Columbia Law School professor Neel Guha sits down with Crosby research engineer Sharan Ramjee to talk about how to tell whether AI is actually any good at legal work. Neel is a computer scientist and...
John Sarihan Verified account @jsarihan
Agents can own entire jobs now, and we hold our agents to the same bar as a full time employee. This means consistently doing high quality work for real customers.

Making contact with reality is critical to build great agents. We do real work, measure outcomes, turn failures into evals, and automatically/continuously improve the system.

If you're interested in pushing the frontier of agents, we're hiring! crosby.ai/careers
crosby.ai Careers Crosby is an AI-first law firm that reviews contracts in under an hour. NDAs, MSAs, and DPAs handled with expert lawyers and AI working together.
Ryan Daniels Verified account @ryanjdaniels Nov 9
@crosbylegal has secured a $5m malpractice insurance policy for our agents. As far as we know, this is the first of its kind.

9 months ago we began to notice our agents were getting extremely capable. In some cases, they did better work than our lawyers.

We are a law firm. We needed to invent a safe way for our agents to do client-facing work without human oversight. It took a lot of work, but this is it.

As soon as the policy went into effect on Nov 3, we removed lawyer oversight from any task that doesn't legally require licensed attorney oversight - what we call "paralegal tasks."

This is ~20% of our work. It frees up our lawyers to focus on what they're best at: thinking, exercising judgment, advising our clients.

This is just the beginning. Once regulations change, we see no reason why our agents can't do excellent legal work too.

Read more in the NYT: nytimes.com/2026/11/07/bus…
View quoted post on X
7 replies 5 reposts 42 likes 3.4K views View on X
John Sarihan Verified account @jsarihan
It's a difficult technical and product challenge to teach AI to act exactly like a lawyer. We trained a 27B model to judge whether an edit is “surgical", i.e. makes targeted edits like a real lawyer would.

Our model beat every frontier model we tested at 90.6% accuracy (vs. 84.3% for the next closest).

At Crosby, messy AI-generated redlines cost lawyer time and damage client trust. For example, rewriting an entire paragraph mkaes the other side defensive, and a string of small edits are hard to follow. Great lawyers make thoughtful, targeted changes that protect the client and keep the deal moving.

So we worked with our attorneys to define what makes an edit surgical, then trained a model to judge it.

To me, this is the future of evals. Doing the work ourselves allows to teach the model what great looks like, and steer our agents and evals.
Crosby Verified account @trycrosby Nov 3
View quoted post on X
8 replies 7 reposts 81 likes 6.3K views View on X
John Sarihan Verified account @jsarihan
Huge shoutout to @emilyzsh @zachkrall. Getting an agent to be both accurate and strike the right tone (across so many different interactions) is a deceptively hard technical and product problem.

Really impressive technical and user interaction work
Ryan Daniels Verified account @ryanjdaniels Nov 2
As of this morning, our paralegal agents are now live for 95% of our clients.

But that's not all. Starting today..

1. You can DM Crosby directly for sensitive requests
2. You can get live status updates on every ticket
3. Our paralegals will cite to previous DMs by pasting former links

Happy halloween ya filthy animals. More to come...
View quoted post on X