How to Cut Trace Volume by 98% Without Losing Signal (OpenTelemetry + Datadog)

What this stack does: Reduces OpenTelemetry trace ingestion volume by dropping low-value traces after completion, while preserving errors, latency outliers, and critical business paths. Demonstrated reduction: 1.87 million traces to approximately 27,000 — a 98% cut with full Span Metrics accuracy maintained.

Purpose: Control APM cost and storage at the collector layer, upstream of Datadog ingestion, without sacrificing visibility into failures and degraded requests.

Prerequisites:

• OpenTelemetry Collector (contrib distribution) v0.90+
• Datadog Agent or Datadog OTLP exporter configured
• Access to collector configuration YAML
• Baseline understanding of your service’s P99 latency and error rate

Step 1 — Understand why tail-based sampling, not head-based

Head-based sampling decides at trace initiation: the root span is retained or dropped before downstream spans exist. This means errors deep in a call stack may be sampled out before they’re observed.

Tail-based sampling buffers all spans for a trace and evaluates them after completion. This enables decisions based on actual outcomes — errors, latency violations, or custom attributes — at the cost of memory overhead. For production systems where errors and slow requests are the signal you need to keep, tail-based is the correct choice.

Step 2 — Configure the Tail Sampling Processor

processors:
  tail_sampling:
    decision_wait: 30s          # Buffer duration; tune to your p99 latency + safety margin
    num_traces: 50000           # Max traces buffered in memory
    policies:
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-requests
        type: latency
        latency: { threshold_ms: 750 }
      - name: critical-checkout
        type: string_attribute
        string_attribute: { key: "http.route", values: ["/checkout", "/payment"] }
      - name: dev-environment
        type: string_attribute
        string_attribute: { key: "deployment.environment", values: ["dev", "staging"] }
      - name: force-sample
        type: boolean_attribute
        boolean_attribute: { key: "force_sample", value: true }
      - name: probabilistic-fallback
        type: probabilistic
        probabilistic: { sampling_percentage: 1 }
      - name: drop-health-checks
        type: composite
        composite:
          max_total_spans_per_second: 0
          policy_order: [ drop-all ]
          composite_sub_policy:
            - name: drop-all
              type: string_attribute
              string_attribute: { key: "http.route", values: ["/health", "/metrics", "/ping", "/ready"] }

Policy order matters. Drop policies execute last — the drop-health-checks policy must appear after any keep policies or it overrides them.

Step 3 — Wire it into the collector pipeline

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters:  [datadog]

Set memory_limiter before tail_sampling — the buffer can grow during traffic spikes, and memory protection upstream prevents OOM conditions.

Step 4 — Set decision_wait to match your p99 latency

The decision_wait parameter defines how long the collector holds a trace before making a retention decision. Set it at your service’s p99 latency plus a 20% buffer. A value too low drops slow traces before they’re complete; too high creates unbounded memory growth during traffic spikes. Start at 30 seconds for most services and tune from metrics.

Step 5 — Validate Span Metrics are unaffected

Tail-based sampling affects trace storage, not Span Metrics computation. Span Metrics (RED: request rate, error rate, duration) are computed on the full traffic volume before sampling decisions are applied. Verify this in your Datadog APM Service Catalog after the first 24 hours of production configuration — all service health indicators should reflect unsampled traffic.

The Trap

The most common configuration mistake is applying drop policies before keep policies in the policy list. Because tail-based sampling evaluates policies in order and stops at the first match, a health-check drop policy placed before an error-code keep policy will silently drop error traces that happen to originate from a /health endpoint. Always structure your policy order: specific keep rules first, general keep rules second, probabilistic fallback third, drop rules last.

Cited References

• Datadog: Control Trace Volume with OpenTelemetry Tail-Based Sampling
• OpenTelemetry Tail Sampling Processor documentation
• Datadog OpenTelemetry Ingestion Sampling

Source: Datadog

Content created with AI assistance and reviewed for accuracy.