Twitxy Saved

Stu Cao

Verified account Prime Intellect

@caoshuxun

Inference @primeintellect
San Francisco, CA Joined August 2010
510 Following 538 Followers 1.2K Posts

Latest posts 5

Pinned post

Stu Cao Verified account @caoshuxun
Two months ago, I left Annapurna Labs, ending a five-year chapter at Amazon. Amazon treated me exceptionally well and gave me more opportunities to grow than I can count. I’m grateful for the people I worked with and everything I learned there.

I followed my heart to Prime Intellect. Its mission to build an open superintelligence stack deeply resonated with me. I believe every person and every business should be empowered to build their own intelligence. Your AI agent should be yours, not a third-party stranger.

Walking into Prime’s SF office, I felt ten years younger. These past two months have been the most exciting of my career.
Three things stand out:

1. The people.
Prime’s team doesn’t all come from famous AI labs, but I’ve rarely seen this level of curiosity and determination.
When we made our high interactivity GLM-5.3 serving endpoint available internally, everyone switched their agents to it. They never went back.
The speed improved productivity. Just as importantly, the team kept sharing feedback, helping us refine performance and tool-calling reliability. Soon, RL rollouts, synthetic data generation, and long-horizon agent research were running on the same infrastructure.
As of this week, our internal inference usage is 600–700 billion tokens per day. That experience gave us the confidence to open the service to the public.

2. The full stack.
Prime brings together hosted training with prime-rl, verifiers, sandboxes, an RL environment hub, the Prime Agent harness, and now Prime Inference.
You can train, deploy, and run agents in one place, without stitching together infrastructure through a massive engineering effort. And most of these components are fully open source.
Our inference stack builds on that same foundation, with deep collaboration across NVIDIA Dynamo, Inferact, and the broader vLLM community.

3. The compute.
For anyone working on LLM infrastructure, access to compute at this scale is a privilege. Building and operating inference across large scale of GB200 racks and B300s is not something I take for granted. Vera is already in our hands, and Rubin is on the horizon. NVIDIA’s latest hardware gives us the foundation to push inference performance further.

Alongside the public launch, we published a technical post on how we built Prime Inference: primeintellect.ai/blog/prime-inf…
Inference has moved incredibly fast over the past year, and it can be hard to separate useful advances from the noise. We use GLM-5.3 optimization to show the work end to end: how we serve agent workloads with low latency while also supporting high concurrency, dependable availability, and reliable tool calls. Recommend for a read if you are interested in learning about P/D, topology choice, NVFP4 KV compression kernel fusion, tool call reliability.

Fast inference matters. Serving real agents reliably at scale is what counts.
www.primeintellect.ai Prime Inference: Fast, Reliable Serving for Frontier Open Models We're launching Prime Inference: resilient serverless and reserved serving for frontier open-source models across multiple datacenters. A look at the production architecture and the prefill/decode,...
23 replies 14 reposts 170 likes 12K views View on X
Stu Cao Verified account @caoshuxun
There are certainly a lot of good work here but we just have too much benchmaxxing in inference right now. training a speculator on the benchmark dataset just to game the numbers.

I also noticed many benchmarks published today use P90, which doesn’t really represent the speed every user can experience.
Zakaria Verified account @zakariaornot 6d
1/ The 1740 TPS decode on 4xB300 is real, but it only applies to the prompt they showcase, which has a very high acceptance rate (5.6 tokens per step). Run the same deployment on average coding workloads with thinking on and decode drops to ~1200 TPS. On chat / writing workloads it’s down to ~750 TPS.
View quoted post on X
7 replies 4 reposts 57 likes 3.1K views View on X
Stu Cao Verified account @caoshuxun
If you ever look into DeepInfer source code, it cannot even run on Hopper. Blackwell and Ascend only.
will depue Verified account @willdepue Nov 12
I have a major result to share.

I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China.

This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model.

Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves.

I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems.

I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used.

It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20.

These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs.

I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpoint-for….

There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
View quoted post on X