Twitxy Saved

Alex Brandes

Verified account Constellation²

@codebrandes

CTO @conste11ation | Gate AI | Local inference in the home lab
constellationgate.ai Joined November 2015
208 Following 1.9K Followers 853 Posts

Latest posts 5

Alex Brandes Verified account @codebrandes
Extra VRAM, PCIe bandwidth, and compute! What a gift.
Morrowmake Verified account @Morrowmake 13h
If this is real...

> be me
> buy 8 old NVIDIA mining GPUs, 8GB each
> turns out they unlock to 64GB PCIe 2
> turns out they actually unlock to 80GB PCIe 4
> turns out they're faster than A100s
> my $12,000 purchase is worth $80,000
> 640GB of data centre GPUs now sit in my living room

blog.kate.land/posts/using-vo…
View quoted post on X
Alex Brandes Verified account @codebrandes
New recipe from the home lab: GLM-5.3-Flash on two CMP 170HX GPUs.

EXL3 2.60 bpw + DFlash2 on vLLM w/PP2 on PCIE 2.0 x4 links
• Decode: 58–83 tok/s single stream, up to 192 tok/s with 4 concurrent
• Prefill: ~1,800 tok/s, flat from 4K to 128K
• 225K context, image + video
• GPQA Diamond: 86.9%

Includes custom tensor-core kernels that improve decode +170% and prefill +165% over baseline on these cards 🤙

github.com/AlexBrandes/cm…
github.com GitHub - AlexBrandes/cmp170hx-glm53-flash-exl3: GLM-5.3-Flash (EXL3 2.60 bpw) on two NVIDIA CMP... GLM-5.3-Flash (EXL3 2.60 bpw) on two NVIDIA CMP 170HX cards with vLLM: recipe, engine image, benchmarks - AlexBrandes/cmp170hx-glm53-flash-exl3
3 replies 3 reposts 23 likes 1.8K views View on X
Alex Brandes Verified account @codebrandes
This could actually be very useful for distributed inference.
Anthropic Verified account @AnthropicAI Nov 3
New Anthropic research: progress on provable inference for large language models.

Here, we introduce FlashProve. Building on established cryptographic techniques, FlashProve allows developers to verify whether a given response came from a given AI model. It’s substantially more efficient than prior open-source methods for doing so.

These proofs have a range of applications. They can, for instance, make it harder for attackers to hijack a chat, or enable safety evaluators to verify that the model they’re testing is the same one being served to customers.

For our full write-up, see here: anthropic.com/research/prova…

Further progress in provable inference will likely be an important part of improving trust in AI deployment. To that end, we're open-sourcing FlashProve’s code, here: github.com/anthropics/Fla…

We’re grateful to our collaborators: Alessandro Chiesa, @GiacomoFenzi, @gwberg95, @miranda__christ, Rahul Dalal (rdalal.github.io) and Ben Mayer (@transmissions11).
View quoted post on X