I spent 3 months benchmarking and digging into the internals of Loki, Elasticsearch, OpenSearch, ClickStack, SigNoz and VictoriaLogs, and condensed it into a 20-minute read.
TL;DR: I realized who ingests fastest is only one side of the story. Every fast number had a price.
I abused each of them with a flood of structured, Kubernetes-style logs on identical machines, until they choked.
1. Loki ingested the most: 139k lines/s.
It indexes only stream labels and keeps lines as plain text, pushing the cost to query time.
Any search or aggregation on a field, the kind every dashboard runs, is expensive and can take minutes: Loki has to decompress and scan every line in matching streams.
2. Elasticsearch finds a single word almost instantly, because it indexes every word of every line.
The price: all that indexing is CPU-heavy, so it topped out at 48k lines/s. And it needed almost 5x the disk of VictoriaLogs.
3. SigNoz and ClickStack store logs in ClickHouse, a columnar database like VictoriaLogs. I expected them to be close. They only reached 10k and 46k lines/s.
The bottleneck wasn't ClickHouse. It was the OpenTelemetry collector in front of it, which used more CPU than ClickHouse itself.
4. VictoriaLogs (I work on it) ingested 120k lines/s with the least memory and disk. It stores each field in its own column and reads only the columns a query needs, so every test query returned within seconds.
The catch: it shines on structured logs, where columns help most. On plain-text logs (from older apps or network devices), it still works, but loses most of its edge.
TL;DR: I realized who ingests fastest is only one side of the story. Every fast number had a price.
I abused each of them with a flood of structured, Kubernetes-style logs on identical machines, until they choked.
1. Loki ingested the most: 139k lines/s.
It indexes only stream labels and keeps lines as plain text, pushing the cost to query time.
Any search or aggregation on a field, the kind every dashboard runs, is expensive and can take minutes: Loki has to decompress and scan every line in matching streams.
2. Elasticsearch finds a single word almost instantly, because it indexes every word of every line.
The price: all that indexing is CPU-heavy, so it topped out at 48k lines/s. And it needed almost 5x the disk of VictoriaLogs.
3. SigNoz and ClickStack store logs in ClickHouse, a columnar database like VictoriaLogs. I expected them to be close. They only reached 10k and 46k lines/s.
The bottleneck wasn't ClickHouse. It was the OpenTelemetry collector in front of it, which used more CPU than ClickHouse itself.
4. VictoriaLogs (I work on it) ingested 120k lines/s with the least memory and disk. It stores each field in its own column and reads only the columns a query needs, so every test query returned within seconds.
The catch: it shines on structured logs, where columns help most. On plain-text logs (from older apps or network devices), it still works, but loses most of its edge.
@cuongleqq published data ingestion benchmark results for open-source databases for logs: VictoriaLogs, Loki, Elasticsearch, OpenSearch, SigNoz and ClickStack (the last two are based on ClickHouse).View quoted post on X
The benchmark measures CPU usage, memory usage, disk space usage and disk IO usage for different ingestion rates of Kubernetes-like logs. It also measures performance for production-like queries.
Hint: VictoriaLogs isn't the fastest one for data ingestion.
victoriametrics.com/blog/structure…