Route OTel Data from AI Apps to ClickHouse and Datadog: An Observability Pipeline Stack

Summary

• AI applications generate unpredictable, high-volume telemetry — standard pipelines weren’t designed for this load profile
• Datadog Observability Pipelines accepts OTLP data via gRPC or HTTP, transforms it, and routes to ClickHouse and/or Datadog simultaneously
• Pre-built Packs (Cloudflare, Okta, Akamai, and others) auto-parse and reduce volume before storage — critical for cost control
• Sensitive Data Scanner runs on-premises with 180+ detection rules, so PII stays local while the rest routes to cloud storage

Purpose

AI-driven workloads present a specific observability problem: agentic pipelines generate logs and metrics at volumes and cardinalities that traditional collection-and-forward pipelines can’t handle cost-effectively. Sending everything to a single destination — whether Datadog, a SIEM, or an analytics database — either breaks the budget or loses fidelity. This stack configures Datadog Observability Pipelines to collect OpenTelemetry Protocol (OTLP) data from AI applications, apply transformations and enrichment, and route to both ClickHouse (for high-throughput analytics) and Datadog (for monitoring) without custom pipeline engineering.

Pre-requisites

• OpenTelemetry Collector configured in your AI application environment
• ClickHouse instance (self-managed or ClickHouse Cloud) with write credentials
• Datadog account with Observability Pipelines enabled
• Familiarity with VRL (Vector Remap Language) for custom transformations — optional but useful

Steps

1. Configure OTLP Receiver

In Observability Pipelines, create a new pipeline with an OTLP source. Choose gRPC (port 4317) or HTTP (port 4318) based on your Collector configuration. The pipeline accepts both logs and metrics from the OTel Collector without schema changes.

2. Apply Packs for Volume Reduction

If your AI stack includes Cloudflare, Okta, or Akamai CDN at the edge, enable the corresponding Packs. Packs auto-parse vendor-specific log formats and apply volume reduction rules — a Cloudflare Pack, for example, deduplicates access logs and extracts structured fields before downstream routing. This step alone can reduce storage costs by 40–60% for high-volume sources.

3. Add Enrichment Tables

Connect enrichment tables from ServiceNow CMDB or Databricks to inject context into events as they flow through the pipeline. For AI workloads, this is particularly valuable: enriching agent trace IDs with service ownership data from CMDB means every ClickHouse query and Datadog alert carries the context needed for root cause analysis.

4. Configure Sensitive Data Scanner

Enable the on-premises Sensitive Data Scanner with the appropriate detection rules for your compliance requirements. The scanner redacts PII, API keys, and credential patterns before data leaves your environment. With 180+ built-in rules and support for custom regex, most teams can configure adequate coverage in under an hour.

5. Define Dual-Destination Routing

Configure two processors:

• ClickHouse destination: routes transformed and enriched logs and metrics to ClickHouse tables via HTTP insert. Use VRL to control cardinality — high-cardinality AI trace data can be aggregated at the pipeline layer rather than at query time.
• Datadog destination: routes alerting-relevant signals (errors, latency spikes, token budget overruns) to Datadog for real-time monitoring. This route can be filtered to reduce Datadog ingestion costs.

6. Monitor the Pipeline

Use Observability Pipelines’ built-in dashboards to track pipeline throughput, error rates, CPU usage, and buffer depth. A pipeline processing terabytes of AI application telemetry has its own reliability concerns — treat the pipeline as a first-class service in your operational runbook.

The Trap

Cardinality at the source compounds at the destination. AI applications that emit a unique trace ID per agent invocation — without aggregation — will produce ClickHouse tables that grow quadratically with usage. Use VRL’s metric aggregation capabilities to collapse high-cardinality dimensions (agent run IDs, token batch IDs) at the pipeline layer before storage. The cost of retrofitting a ClickHouse schema for cardinality later is significantly higher than designing for it at the pipeline stage.

Cited References

• Datadog Observability Pipelines: OTel + ClickHouse Integration
• OpenTelemetry Protocol Specification
• ClickHouse Documentation — HTTP Interface

Photo by Lukas Blazek on Pexels

Source: Datadog Blog

Content created with AI assistance and reviewed for accuracy.