Jack Fitzsimons (@jk_fitzsimons)

Had the pleasure of catching up with @p_e last week. We first met when he crushed a competitive programming competition in Bremen, Germany (circa 2012) and I've always been impressed by his work ever since.

So, needless to say it was a privilege to answer some questions for his substack: research.auditless.com/p/building-bla…

If you're interested in playing around with Struct yourself check out blatent.ai

research.auditless.com Building Blatent Struct: An Interview with Jack Fitzsimons Blatent Struct is a new type of “programmable AI” optimized for highly efficient and reliable workflows. We caught Jack during a busy launch week and here are his thoughts on why it matters.

Jack Fitzsimons @jk_fitzsimons 6d

Looking forward to heading back to Bengaluru at the end of the month. I'll be speaking at WeAreDevelopers Conference on the 25–26 November about fun and games with evals and some of the risks involved when fine tuning.

If you are at the event or in town let me know!

wearedevelopers.com/conference-ind…

www.wearedevelopers.com Did the Model Improve, or Just the Score? A Developer’s Guide to LLM Evaluation · WeAreDevelopers... A model can extract the right information and still score zero because its answer differs from a benchmark’s reference. Fine-tuning can raise that score, but does the improvement carry over to the...

Jack Fitzsimons @jk_fitzsimons Nov 4

Well, this was fun to create! 😅 It's so easy to get caught up in evals and benchmarks and forget the ultimate goal of what you are trying to achieve.

Scratch the surface and you can often see that standard benchmarks are regularly wrong, or at the very least extremely subjective. In these cases, scoring high might actually imply overfitting (more on this coming soon).

But as the saying goes, "don't throw the baby out with the bathwater"; the underlying sample prose and questions can still be extremely valuable even if there are deficiencies in the scoring functions.

@R_Pisarczyk and I thought it would be fun to explore this a little in our latest blog post.

Robert Pisarczyk Verified account @R_Pisarczyk Nov 4

An LLM answered “CNN” when asked which company Larry King worked for. The benchmark gave it zero points. The reference answer? “CNN’s”.

We found similar issues across extraction benchmarks. Good datasets, misleading scores.

blatent.ai/blog/making-fl…
View quoted post on X

Jack Fitzsimons @jk_fitzsimons Oct 27

Over the past few months, my team and I have been building Blatent Struct. Today, we’re launching it.

The idea was simple: an LLM should be fast, cheap and accurate. Of course, who doesn’t want that? But we also wanted to go beyond structured outputs and let you program the business logic that governs the answer.

Structured outputs already let you constrain the JSON labels you get back. We wanted to express how fields depend on one another, what to loop over from the input or application state, and which paths the user’s permissions allow.

With Struct, you define conditions, switch cases and loops, alongside constraints from OAuth scopes and OPA policies. Those rules are enforced during inference. Earlier decisions constrain what can come next, all within a single API call.

So you can work through large trees of dependent decisions and get the complete JSON object back, without your application having to coordinate each step through separate model calls.

The aim is dependable endpoints you can call throughout your software, with very little overhead.

In our benchmarks, Struct achieves comparable accuracy to Claude Sonnet 5.5 at 11× the speed and over 100× lower estimated cost. It’s also over 2× faster than Jev, with higher average accuracy across 20 benchmarks.

Across six extraction benchmarks, it scored 87.5% against Cloudflare Clairic’s 81.7%, at 62% lower estimated cost.

I’m really proud of what our engineers and the wider team have built, and grateful to our investors and everyone who backed us.

I hope you like it. I’m looking forward to seeing what you build. If you have questions, please reach out. It's free to get started (very cheap thereafter) and you can have 100m tokens on us :)

My co-founder @R_Pisarczyk made an overview video here: x.com/R_Pisarczyk/st…

blatent.ai

Robert Pisarczyk Verified account @R_Pisarczyk Oct 27

Very excited to finally share what we've been building!

Today, we are launching Blatent Struct, our first language model. 🚀

One thing that's always bothered us about building with LLMs is how much engineering you need around the model. Getting an answer is one thing. Making sure it follows the right decision path, returns everything your application needs and doesn't break your business rules is another.

We wanted to change that.

Struct brings programmability to language models.

You can define complex, cascading decisions, conditional dependencies and exactly what each outcome must satisfy. You can also enforce business rules and permissions during inference.

And rather than stitching together multiple model calls, Struct can handle the whole decision tree in a single run.

We also spent a lot of time making sure this doesn't come at the expense of performance.

In our benchmarks, Struct achieves comparable accuracy to Claude Sonnet 5.5, at 11× the speed and over 100× lower estimated cost.

It's also over 2× faster than Jev, with higher average accuracy across 20 benchmarks. And on source-grounded extraction, it outperforms Cloudflare Clairic (87.5% vs. 81.7% across six benchmarks), at 62% lower estimated cost.

For me, the exciting part is what this enables: everything from document processing and customer support to agents that can make complex decisions while respecting the rules you've set.

This is just our first model, and there's a lot more to come.

I'd love for you to give it a try and tell us what you think. We're especially keen to hear what works, what doesn't and what you'd like us to build next.

🚀 Get started for free (up to 100m input tokens are on us) : blatent.ai

View quoted post on X

Jack Fitzsimons @jk_fitzsimons Nov 5, 2020

@tom_rainforth is a legend and one of the hardest working researcher I know. Highly recommend applying to work with him if you’ve an interest in doing a DPhil in ML at Oxford!

Tom Rainforth @tom_rainforth Nov 5, 2020

If you're interested in doing a PhD with me, or more generally in machine learning at Oxford, please consider applying for this. It's a great programme!
View quoted post on X

Jack Fitzsimons

Latest posts 5