50% of Your Metrics Are Never Queried: The Case for Sustainable Observability

Half of the metrics your observability stack is collecting, storing, and querying budgets for are never looked at. That’s not a tooling problem — it’s a design-phase problem, and CNCF’s recent panel on sustainable observability lays out a framework for fixing it before the pipeline is built.

The Framework

The organizing principle behind sustainable observability is treating telemetry as infrastructure rather than logging. Infrastructure decisions happen at design time and get evaluated for cost and utility. Logging decisions traditionally happen at build time and accumulate without review. The panel advocates shifting that default: start by asking “if this data stream stopped, what would we actually lose?” rather than “what should we instrument?”

The OpenTelemetry graduation from CNCF makes this framework more accessible — standardized OTLP protocol and semantic conventions mean teams can instrument once and then optimize at the pipeline layer rather than returning to application code every time requirements change.

Start with RED Metrics

Rate, Errors, and Duration (RED) provide the bedrock signals for service isolation. These three metrics answer the core question in any incident: is this service slow, broken, or overloaded? Before instrumenting anything else, every service should expose clean RED metrics through OpenTelemetry.

The value of starting here is scope control. Teams that instrument everything simultaneously — traces, metrics, logs, profiles — produce the telemetry waste the panel identifies. RED metrics give you a working observability baseline with minimal cardinality overhead.

Build the Observability Mesh

Once RED metrics are stable, the high-leverage move is connecting signals rather than adding new ones. The “observability mesh” the panel describes is a design pattern where metrics surface traces, traces embed logs, and logs tie into resource profiles. Each signal resolves the next instead of requiring parallel investigation tracks.

This correlation work happens in pipelines, not application code. The panel recommends using centralized infrastructure enrichment — adding service context, deployment metadata, and relationship data at the pipeline layer — so individual services stay lightweight and the enrichment logic can be updated without redeploying applications.

Optimize Before Adding

Two pipeline techniques reduce cardinality without losing diagnostic value:

Tail-based sampling preserves anomalous traces (errors, slow outliers) while dropping routine traffic. For services with high throughput and healthy baselines, this reduces trace storage by 80–95% with no loss in debugging capability for real incidents.

High-cardinality management — masking unique identifiers like user IDs, session tokens, and request GUIDs — eliminates the most common source of metric cardinality explosion. Labels that look like user_id=u-39471820 don’t aggregate meaningfully and inflate storage costs without adding analytical value.

Adapt for AI/LLM Observability

Traditional success criteria (uptime, error rate, latency) fail for probabilistic AI systems. An LLM that’s up and responsive is not necessarily behaving correctly. The panel identifies a shift needed: observability for AI pipelines should track decision quality and cost-effectiveness across prompt-to-response chains, not just infrastructure health.

Concretely, this means instrumenting token consumption per workflow, measuring response consistency across similar inputs, and tracking latency at the model and retrieval layers separately. Standard RED metrics still apply at the service level; the additional layer is semantic performance — what the AI actually produced relative to what was expected.

Source: CNCF

Content created with AI assistance and reviewed for accuracy.