Zhou Yu

Everyone's talking about how easy it is to build AI agents.

The hard part nobody talks about? Evaluation.
In the LLM era, you don't need data to build. You just prompt. But you still need data to know if what you built actually works.

The current process: hand your agent to colleagues, collect 100–200 traces, annotate manually, cluster, build an LLM judge. Weeks of work. And your testers are engineers and PMs who don't give you the diverse, edge-case-heavy examples you actually need.

Sound familiar?

We built ArkSim to solve this - an open-source tool that simulates your end users automatically, so you get systematic coverage without the guesswork.

👉 Full read: arklex.ai/home/blogs/why…

⭐ github.com/arklexai/arksim

Your AI passed every test.

It still failed the user.

Here's how that happens: an agent handling an access request scores perfectly on every response. Relevant. Accurate. Well-grounded. Clear, professional language.

And then it grants access to the wrong person.

Every individual answer looked great. The outcome was wrong. From the model's scorecard, a win. From the user's chair, a failure.

That's the gap in how we test AI today. We grade responses one at a time, but users don't judge agents that way. They judge them on one thing: did it solve the problem I came to solve?

Two people ask the same question, "can you give me access to this system?" One's a new employee. One's an outside contractor. Same words, different correct answer. You can't score that without knowing who's asking.

So the unit of evaluation has to change. Not the prompt. The whole journey:

→ Did the agent ask the right clarifying questions?
→ Did it hold context across a long conversation?
→ Did it recover after a mistake?
→ Did the user actually get what they came for?

The future of AI testing isn't centered on the model. It's centered on the user.

Full breakdown here 👇
arklex.ai/home/blogs/use…

Every team building AI agents eventually reaches for an LLM-as-judge. It's the only way to score thousands of conversations without an army of reviewers. But the same question comes up every time: "How do I know my judge is actually right?"
An unverified judge is worse than no judge at all. It hands you confident numbers that might be measuring the wrong thing:
→ It passes a broken agent, a false green light
→ It fails a good agent, and engineers chase ghosts
→ It scores on generic best practices, not your standards
→ It never learns what your experts already know

The hard part isn't measuring overall accuracy. It's closing the gap between what your judge measures and what your experts actually mean by quality, because that gap is where your real-world failures hide.

Full breakdown of how we solved it in ArkSim 👇
arklex.ai/home/blogs/fro…

Please check out the paper at ACL and meet Billy

In text, a 300ms delay is invisible.

In voice, it's a broken conversation.

Voice agents are a different beast to test. A single bad second of audio can erode user trust entirely: a glitch, an awkward pause, a voice that suddenly sounds like a different person.

And most test suites only cover the happy path. Real users don't:

→ They interrupt mid-sentence
→ They mumble "mm-hmm" without taking a turn
→ They call from noisy cars, kitchens, and crowded rooms
→ They speak with accents your ASR has never heard

The hard part isn't measuring overall accuracy. It's finding which failures cluster around which conditions, because that's where your real-world gaps live.

Full breakdown of the 4 hardest parts here 👇
arklex.ai/home/blogs/tes…