Pinned post
Shipped an overhaul of our data pipeline to make its memory footprint small and predictable.
Our entire data stream now runs on ~8 GB of RAM and 20 CPU cores.
Our entire data stream now runs on ~8 GB of RAM and 20 CPU cores.
We've been using similar concepts when building Vercel Data PipelineView quoted post on X
- Processing over 5GB/s
- Supports multi fanout
- Deduplication
- At-least once delivery
I think I need to write a blogpost