Pinned post
#OpenAI gave two talks at Community Over Code EU 2026 on running #Spark at their scale: hundreds of thousands of applications a day. Both end up at GreptimeDB for Spark observability, from opposite ends.
One wanted metrics for every Spark task. That's up to tens of millions of new series a day, with logs, metrics and events sitting in separate stores.
The other needed the Spark History Server fast enough for agents. Opening a 100 GB+ event log could take up to half an hour, and every agent tool call waited on it.
Where it landed: a listener in the Spark driver writes jobs, stages, tasks and executors into #GreptimeDB. The History Server reads from it. The #Flink job that parsed event logs, "probably one of the most expensive Flink apps at OpenAI", is no longer needed. The data sits on S3 as #Parquet, so Spark and Trino query it directly.
My read: high cardinality and scattered signals are one problem. They show up together once telemetry is fine-grained enough to be useful.
Our notes on both talks, with timestamps: greptime.com/blogs/2026-11-…
One wanted metrics for every Spark task. That's up to tens of millions of new series a day, with logs, metrics and events sitting in separate stores.
The other needed the Spark History Server fast enough for agents. Opening a 100 GB+ event log could take up to half an hour, and every agent tool call waited on it.
Where it landed: a listener in the Spark driver writes jobs, stages, tasks and executors into #GreptimeDB. The History Server reads from it. The #Flink job that parsed event logs, "probably one of the most expensive Flink apps at OpenAI", is no longer needed. The data sits on S3 as #Parquet, so Spark and Trino query it directly.
My read: high cardinality and scattered signals are one problem. They show up together once telemetry is fine-grained enough to be useful.
Our notes on both talks, with timestamps: greptime.com/blogs/2026-11-…
