Jaeger v2.18.0: 8.6× Compression and Sub-100ms Traces with ClickHouse Storage

Jaeger v2.18.0 introduces native ClickHouse support as a storage backend, and the benchmark numbers on 10 million spans are significant: 8.6× compression ratio, 50,000+ spans per second ingestion, trace retrieval around 100ms, and most search queries completing under 50ms on a single-node deployment.

The Storage Backend Problem in Distributed Tracing

Elasticsearch has been the default Jaeger storage backend for years — and it works, but it comes with well-known tradeoffs at scale. Elasticsearch’s document-oriented model stores spans as JSON, which does not compress well and creates high storage costs as span volume grows. Query performance on aggregated trace searches degrades under load. For teams running Jaeger on Kubernetes at meaningful scale, the Elasticsearch storage tier becomes one of the more expensive line items in the observability stack.

Jaeger’s existing OpenTelemetry observability architecture already supports multiple storage backends, but ClickHouse represents a fundamentally different architectural fit. ClickHouse is a columnar OLAP database optimized for append-only write streams and analytical aggregations — the exact access pattern that distributed tracing generates. Traces come in as append-only span events; queries are aggregations over service names, trace IDs, durations, and error rates. Elasticsearch’s row-oriented model handles this poorly compared to a purpose-built columnar store.

The Numbers Behind the ClickHouse Bet

The Jaeger ClickHouse integration, now shipping in alpha with Jaeger v2.18.0, was benchmarked against 10 million spans across 1 million traces on a single-node ClickHouse deployment. The results:

• 8.6× compression on the spans table — meaning storage costs for an equivalent Elasticsearch deployment would run 8× higher for the same span volume
• 50,000+ spans/second sustained ingestion without degradation
• ~100ms trace retrieval for individual traces
• ~50ms for most filtered searches; ~140ms for more complex multi-attribute queries

These are alpha numbers on a single node, not a production multi-node cluster — but the compression ratio is the metric that matters most for the cost calculus at scale. The columnar storage format in ClickHouse achieves this compression through encoding schemes that exploit the high cardinality and repetitive structure of span attributes in a way that document stores cannot.

Source: JaegerTracing on Medium

Content created with AI assistance and reviewed for accuracy.