Twitxy Saved

Comet

This account is verified because it's an official organization on X. Learn more

@Cometml

Comet provides an end-to-end model evaluation platform for AI developers, with best in class LLM evaluations, experiment tracking, and production monitoring
New York, NY comet.com Joined October 2017
877 Following 15K Followers 3.5K Posts

Latest posts 5

Comet @Cometml
Tracing a Google ADK agent in Opik takes two lines of code.

But, Google's ADK has a footgun: when one ADK agent hands off to another, ADK starts a new run, so one user message shows up as two traces.

We put together a little guide to fixing this with a single Opik call. We stood up a two-agent support bot with ADK and Gemini, traced it in Opik, then ran 23 hard support cases on Gemini 3.8 Flash and 3.5 Flash-Lite:

- neither tried a forbidden refund, even with a fake "refund pre-approved" note
- Flash-Lite was 5× cheaper and ~6× faster
- the LLM judge scored a correct refusal 0.50, so the policy check reads tool-call arguments in code

comet.com/site/blog/trac…
Comet @Cometml
New in Opik: describe the chart you want, and Ollie builds the dashboard widget.

Ask for something like "average hallucination score for the car-specs judge, by agent version, last 30 days." Under the hood, Ollie writes a read-only SQL query over your traces, spans, threads, experiments and feedback scores, then proposes a few Vega-Lite charts.

Pick one, add it to a dashboard, and it keeps updating as traces arrive. Refine it the same way: "make these lines," "split by release," whatever you'd like to visualize, Ollie builds it.

Available now in Opik: comet.com/site/blog/intr…
1:23
Comet @Cometml
Want to train your own decision model? @anmorgan2414 just wrote a guide to help you out: comet.com/site/blog/trai…

Some details about the setup:

Abby fine-tuned Qwen3-0.6B two ways. With one gold label per question, the model came out overconfident (ECE 0.12).

Trained against the full label distribution, ECE dropped to 0.02, with or without a decision head.

She also packed all five questions after one shared state, with a mask so each question sees only the state and itself.

2.1× fewer tokens, 329 → 175 ms per case on a T4, and the same answers as asking one at a time.

Training tracked in Comet, eval in Opik, code public. Try it for yourself!
Comet @Cometml
Sharing an update from our Engineering team: On October 26, 2026, Comet contained a security incident affecting Opik on comet[dot]com. No customer data, accounts, or API keys were accessed. Still, our team responded quickly and thoroughly. This post explains what happened, what was affected, and what we have done in response: comet.com/site/blog/secu…
www.comet.com Opik Security Incident: October 2026 No customer data, accounts, or API keys were accessed. On October 26, 2026, Comet contained a security incident affecting Opik on comet.com. This post explains what happened, what was affected, and...
Comet @Cometml
Cutting AI costs isn’t just about routing to cheaper models. Coding agents can waste tokens in plenty of other places.

Our CEO @gidim unpacks what we learned building Cost Intelligence in a @cerebral_valley deep dive.

cerebralvalley.beehiiv.com/p/10x-engineer…
cerebralvalley.beehiiv.com 10x Engineer or 10x Token Bill? 💸 Comet CEO Gideon Mendels on the hidden costs of coding agents, and how to tell whether the spending is paying off.