Just need more data engineers to start running benchmarks now !
@byte_array Very impressive, kudos on building Quanton, that's some serious competition to big whaleView quoted post on X
Vinoth Chandar
Verified account This account is verified. Learn more@byte_array
Founder @Onehousehq — Data Infra for Agents. Creator of @apachehudi, built the first #DataLakehouse @Uber. Databases, Dist. Systems. LinkedIn, Confluent.
235 Following 2K Followers 1.5K Posts
Latest posts 5
⚡ Quanton engine is really freakin fast. Here’s the receipt. Uber put out a blog that unpacks Quanton evaluation at production scale against OSS Spark and found it 4.7x-5.5x faster.
x.com/UberEng/status…
x.com/UberEng/status…
More than 200,000 CPU cores power Uber’s data lake ingestion. At that scale, every efficiency gain counts.View quoted post on X
Making better use of that compute means examining how Spark executes queries, moves data, and writes to growing tables. Accelerating one stage can leave other bottlenecks untouched.
Uber engineers are improving production workloads with native vectorized execution and evaluating new ways to accelerate write-heavy pipelines.
Explore the engineering choices behind those efforts, where the benchmarks showed gains, and what remains to be optimized.
I still can't leave upserts alone.
hudi-rs 0.6.0 now writes Hudi tables from Python and Rust. No JVM. I enjoyed designing and building the writer myself.
Against delta-rs and PyIceberg, across five workloads:
- Upserts: 1.7-12x faster
- Deletes: 4.8-24x faster on four, comparable on the fifth
- 4-200x less data written per upsert
It's fascinating to see ideas hold up for 10-years, when you solve a problem really well.
Thanks to Ethan Guo and the hudi-rs community. Blog in comments 👇
hudi-rs 0.6.0 now writes Hudi tables from Python and Rust. No JVM. I enjoyed designing and building the writer myself.
Against delta-rs and PyIceberg, across five workloads:
- Upserts: 1.7-12x faster
- Deletes: 4.8-24x faster on four, comparable on the fifth
- 4-200x less data written per upsert
It's fascinating to see ideas hold up for 10-years, when you solve a problem really well.
Thanks to Ethan Guo and the hudi-rs community. Blog in comments 👇
S3 is for durable WAL and keeping the colder parts of the table filling up local NVMe. Scaling reads are no problem.
The only thing it can’t do as much is very fast writes. Double-digit ms write latencies are the line I’d draw.
S3 Express is single-zone with a bad networking stack in front of your request. So unsure if it's strictly better than NVMe.
The only thing it can’t do as much is very fast writes. Double-digit ms write latencies are the line I’d draw.
S3 Express is single-zone with a bad networking stack in front of your request. So unsure if it's strictly better than NVMe.
S3-based database architectures don’t read from S3 all the time, though.View quoted post on X
Instead, they amortize the cost by caching in memory and on local disk (which can be NVMe). As a result, read performance is more or less the same as in local-disk architectures, though the tail latency is heavier; NVMe’s tail is much flatter.
Writes are slower, but not by as much as people often think. You almost never want a write acknowledged by a single local disk, so a local-disk database pays a network round trip for replication anyway. That makes S3-based architectures competitive, especially with S3 Express.
Good read. “it just works” is subjective. As you’d expect, the divide between easy single-node at lower scale vs. a distributed engine at PB scale is very real.
View quoted post on X
