Moving fast isn't a substitute for thinking. I recently read @andrewchen's post on the Growth Maze and it really made me think of when I led measurement growth at @RobinhoodApp. (see 馃У)
Latest posts 5
Getting to 99% accuracy on your LLM evaluations feel a bit slow?
Good luck if you have multiple prompts and metrics to tune 馃
At my last startup Parcha, I pulled an all-nighter my first week into the job to tune an Agent for a customer demo. 0 hours of sleep. I wanted to make sure our agent was reliable and that we showed our prospect what we were capable of!
I even had prior experience! I previously productionized an agent using LLMs at Parafin on a larger dataset and had read 20-30 research papers on how to effectively tune prompts! Didn't matter. I would run the Agent 2-3 times and get a slightly different (and incorrect) answer at least 1 out of every 3 times.
After 2 years, LLM dev tools still aren't solving the core LLM evaluation problems even though I knew it was possible from multiple iterations of building 0 to 100, taking reliable Agents to production in risky, regulated industries. That's why I started DimRed - to build Agents that do all of manual, tedious work involved in LLM evaluations and prompt tuning. So the engineers that are building the future can focus on helping people solve problems instead of fighting LLM evaluations and playing prompt tuning whac-a-mole.
Try it out for free at dimred.com. New users get $20 in welcome credits toward 1-click prompt tuning on sign-up!
Good luck if you have multiple prompts and metrics to tune 馃
At my last startup Parcha, I pulled an all-nighter my first week into the job to tune an Agent for a customer demo. 0 hours of sleep. I wanted to make sure our agent was reliable and that we showed our prospect what we were capable of!
I even had prior experience! I previously productionized an agent using LLMs at Parafin on a larger dataset and had read 20-30 research papers on how to effectively tune prompts! Didn't matter. I would run the Agent 2-3 times and get a slightly different (and incorrect) answer at least 1 out of every 3 times.
After 2 years, LLM dev tools still aren't solving the core LLM evaluation problems even though I knew it was possible from multiple iterations of building 0 to 100, taking reliable Agents to production in risky, regulated industries. That's why I started DimRed - to build Agents that do all of manual, tedious work involved in LLM evaluations and prompt tuning. So the engineers that are building the future can focus on helping people solve problems instead of fighting LLM evaluations and playing prompt tuning whac-a-mole.
Try it out for free at dimred.com. New users get $20 in welcome credits toward 1-click prompt tuning on sign-up!
2:40
1. It should not take so much time to tune prompts.
2. It should not take so much time to label data.
3. It should not take so much time to cut token costs.
4. It should not take so much time to debug Agent errors.
5. It should not take so much time to get useful insights on Agent performance.
2. It should not take so much time to label data.
3. It should not take so much time to cut token costs.
4. It should not take so much time to debug Agent errors.
5. It should not take so much time to get useful insights on Agent performance.
LLM evaluations are a waste of time.
But they are a necessary evil to create great product experiences and 90% of the work is spent on manual, grueling iteration.
We all know coding agents should help us write pull requests, sales agents should help us with outreach, and CX agents should answer our product questions.
And at DimRed, we believe that agents should help us with LLM evaluations. So that engineers can focus on solving customer problems and shipping product instead of repeatedly tweaking the wording on massive system prompts and carefully inspecting LLM traces for the edge cases that lead to product failures.
Our early design partners have seen better performance with DimRed鈥檚 prompt optimization agent compared to existing State of the Art algorithms and we are excited to open up the platform for all to try on October 8th!
Join the waitlist below before next Wednesday to get $50 towards tuning.
Link: lnkd.in/gwnNjnEu
But they are a necessary evil to create great product experiences and 90% of the work is spent on manual, grueling iteration.
We all know coding agents should help us write pull requests, sales agents should help us with outreach, and CX agents should answer our product questions.
And at DimRed, we believe that agents should help us with LLM evaluations. So that engineers can focus on solving customer problems and shipping product instead of repeatedly tweaking the wording on massive system prompts and carefully inspecting LLM traces for the edge cases that lead to product failures.
Our early design partners have seen better performance with DimRed鈥檚 prompt optimization agent compared to existing State of the Art algorithms and we are excited to open up the platform for all to try on October 8th!
Join the waitlist below before next Wednesday to get $50 towards tuning.
Link: lnkd.in/gwnNjnEu
0:26