@CerebralCoding_
Pinned post
The time for waiting is over.View quoted post on X
I have spent the last month on the deepest project I have ever taken on, chasing what I believe is one of the lowest hanging fruits in local AI: bridging the gap between Apple and NVIDIA hardware.
Today I am giving it away.
Apple NVIDIA Sidecar runs one model across a Mac and an RTX 5090 as a single engine. My first submission is the base case: a Mac Studio M5 Max with 64 GB, one 5090 in a dock, and Qwen 3.8 Flash Next on NVIDIA's factory FP4 weights.
It writes at about 210 tokens a second. It scored 86 of 100 on MMLU-Pro and 30 of 30 on AIME.
For a sense of scale, here is what others have published for this same model. About 43 tok/s on one DGX Spark. About 47 on a Jetson AGX Thor. About 83 at peak on an M4 Max. 114 to 116 on a 5090 PC with 128 GB running NVFP4. A median of 159 on a 72 GB RTX PRO 5000.
Different prompts and setups, so treat those as ballpark. But as far as I can find, this is the fastest anyone has shown on NVIDIA's factory weights with one consumer graphics card.
Now the losses, because they matter just as much. A good PC reads long prompts two to three times faster than I do: about 1,900 tok/s on an 8K prompt here, against 3,900 or more there. The engine serves one conversation at a time. It has only ever been installed on my two machines. And the driver is the hard part: mine is locally modified and is not included.
None of this exists without the people who came first.
Niko Veit @coldniko built Strata, the engine all of this stands on.
The tiny corp built tinygrad, the driver path that lets a Mac talk to that card at all.
NVIDIA released the weights.
Salvatore Sanfilippo @antirez, @laurent_zw, @ggml_org, @PeasantSmith, John Fink and asf0 wrote code that is carried in this repo, with their licenses intact.
What I added is the pipeline between them. I worked on it for weeks, I am proud of it, and it is the hardest thing I have ever built.
I also think I have taken it about as far as I can alone. What I have done is enough to get the next person going, so I would love for this to become a community project.
There are real doors left open.
The driver: make the public signed path work, so nobody has to build their own.
Concurrency: total output stays flat near 61 tok/s from 1 to 16 cold conversations. Serving several at once would change what this machine is for.
The link: reverse engineer the dock and its firmware. We measure 6.79 GB/s one way and 5.53 the other, on a path rated for 7.88.
Prompt reading: it is the biggest gap to a PC, and it should not stay there.
Other machines: I have only proved this on mine. Yours is the most useful test there is.
github.com/joesinvestment…
✅ BENCHMARKED: TensorFold made Qwen3.8 27B more than 3x faster on my 64GB M5 Pro Mac mini, on two versions in one day.
In the morning I tested 1.2.1. On its own, the model wrote about 17 tokens a second. I then switched on the DFlash2 drafter, a small helper that guesses the next few tokens so the big model only has to check them and it jumped to 56.
And later in the afternoon @ashxhart shipped 1.2.2. His release notes focused on other hardware, so I reran everything on my Mac mini.
On mine it runs within a few percent of 1.2.1, and the drafter still makes it 3x faster. With randomness off, the answer came out word for word the same as without the drafter.
- Chat, 3.26x faster on 1.2.1 and 3.35x on 1.2.2
- A 31K token prompt, 2.94x and 2.77x
- Code, 5.4x and 5.5x, up to 91.90 tokens a second
- 28.5 GiB peak memory on both
- New in 1.2.2, the first request doesn’t wait for the drafter to load
A massive thanks to @ashxhart and all the contributors behind 1.2.2.
The TensorFold challenge from @YukonResearch is open until Nov 6th if you want to try pushing it further on your own Mac.View quoted post on X
What really limits M5 Max MacBookPro to run local AI isn't processing power or memory bandwidth. It's cooling. After a while your 1,625MHz GPU clock would get throttled down to 1,000 and that kills 30-40% tokens/s. MacStudio doesn't have this problem.View quoted post on X
On Apple Silicon, LocalSR runs its main upscaling models on the Neural Engine. On my M1 Pro a tile takes 19 ms instead of 38 ms on the GPU, using less power.
Free and open source: github.com/HerRei/local-u…View quoted post on X
