Pinned post
🎙️ NEW EPISODE: 3D Stacked DRAM Is Where Inference Hardware Ends Up: d-Matrix's Sudeep Bhoja
Sudeep Bhoja, co-founder and CTO of d-Matrix, joins @vikramskr to walk through the company's path from an early, non-consensus focus on memory-bound inference to a full rack of 3D-stacked DRAM. He covers the pivot from analog to digital in-memory compute, how the Corsair card fits into disaggregated systems, and how the next-gen Raptor chip plugs into Nvidia's ecosystem.
- Corsair delivers 150 TB/s of on-chip SRAM bandwidth, over 8x the 18 TB/s of HBM on an Nvidia Blackwell GPU
- Raptor stacks a logic die on a DRAM die for 32 GB, 16x the prior SRAM capacity, at ~100 TB/s
- 0.37 pJ/bit vs. d-Matrix's estimate of ~5 pJ/bit for HBM4 plus a GPU
- A full rack of 144 Raptor packages, 2.3 TB of high-bandwidth memory, built to drop into Nvidia's VR rack
Presented by @CrusoeAI: Fine-tune and serve open models without managing a single GPU.
When we sat down with Crusoe’s product team, Austin asked: what will you do with my data? Their answer: They don’t touch or see it. Your training data is encrypted so even Crusoe’s own admins can’t see it, and the fine-tuned model is yours. Deploy on Crusoe in one click, or download and run it anywhere. Upload a dataset, pick an open model, and pay per token rather than per GPU hour. Run a few versions, compare them, and download the one that works best. Accounts are free, and get $5 in credit when you sign up. Try Serverless Fine-Tuning: semidoped.com/crusoe-x
Chapters:
0:00 Intro: The Cost of a Token
3:28 The 2019 Bet on Inference
7:43 Nighthawk: Analog In-Memory Compute
12:08 Why Analog Failed
19:02 Corsair: Digital In-Memory Compute
26:04 System Strategy: Speculative Decoding
37:05 Raptor: 3D Stacked DRAM
39:40 The 'Free Lunch' of 3D Stacking
45:00 Trade-off: Logic-on-Top vs DRAM-on-Top
54:14 Go-to-Market: The Nvidia VR Rack
1:03:17 The Future: Lightning & 4-High Stacks
1:12:45 The End State of Inference
Get more of Austin and Vik daily, free!
Sign up: daily.semidoped.com
Connect with Vik and Austin:
Vik's Paid Substack: viksnewsletter.com
Austin's Paid Substack: chipstrat.com
@austinsemis @vikramskr
Sudeep Bhoja, co-founder and CTO of d-Matrix, joins @vikramskr to walk through the company's path from an early, non-consensus focus on memory-bound inference to a full rack of 3D-stacked DRAM. He covers the pivot from analog to digital in-memory compute, how the Corsair card fits into disaggregated systems, and how the next-gen Raptor chip plugs into Nvidia's ecosystem.
- Corsair delivers 150 TB/s of on-chip SRAM bandwidth, over 8x the 18 TB/s of HBM on an Nvidia Blackwell GPU
- Raptor stacks a logic die on a DRAM die for 32 GB, 16x the prior SRAM capacity, at ~100 TB/s
- 0.37 pJ/bit vs. d-Matrix's estimate of ~5 pJ/bit for HBM4 plus a GPU
- A full rack of 144 Raptor packages, 2.3 TB of high-bandwidth memory, built to drop into Nvidia's VR rack
Presented by @CrusoeAI: Fine-tune and serve open models without managing a single GPU.
When we sat down with Crusoe’s product team, Austin asked: what will you do with my data? Their answer: They don’t touch or see it. Your training data is encrypted so even Crusoe’s own admins can’t see it, and the fine-tuned model is yours. Deploy on Crusoe in one click, or download and run it anywhere. Upload a dataset, pick an open model, and pay per token rather than per GPU hour. Run a few versions, compare them, and download the one that works best. Accounts are free, and get $5 in credit when you sign up. Try Serverless Fine-Tuning: semidoped.com/crusoe-x
Chapters:
0:00 Intro: The Cost of a Token
3:28 The 2019 Bet on Inference
7:43 Nighthawk: Analog In-Memory Compute
12:08 Why Analog Failed
19:02 Corsair: Digital In-Memory Compute
26:04 System Strategy: Speculative Decoding
37:05 Raptor: 3D Stacked DRAM
39:40 The 'Free Lunch' of 3D Stacking
45:00 Trade-off: Logic-on-Top vs DRAM-on-Top
54:14 Go-to-Market: The Nvidia VR Rack
1:03:17 The Future: Lightning & 4-High Stacks
1:12:45 The End State of Inference
Get more of Austin and Vik daily, free!
Sign up: daily.semidoped.com
Connect with Vik and Austin:
Vik's Paid Substack: viksnewsletter.com
Austin's Paid Substack: chipstrat.com
@austinsemis @vikramskr
75:44
