Data Pipeline Monitoring 101: How to Track Health Across the Full Stack

What this covers: Monitoring the health and performance of data pipelines — orchestration through serving — using signals that differ from traditional application observability. Data pipelines have distinct failure modes that APM tooling designed for request-response systems doesn’t surface.

Why it matters: Data quality failures in a pipeline are often invisible to infrastructure monitoring. A pipeline can execute successfully — jobs complete, no errors thrown — while silently delivering stale, incomplete, or transformed-wrong data to consumers downstream. Monitoring requires dataset-level signals, not just process-level signals.

Purpose

Build observability coverage across the seven-layer data pipeline architecture so failures surface at the layer where they originate rather than appearing as unexplained anomalies in serving or application behavior.

Pre-requisites

• A data pipeline with at least one of: orchestration (Airflow, Dagster, Prefect), ingestion/transport (Kafka, Kinesis, Fivetran), processing (Spark, dbt, Flink), or storage (Snowflake, BigQuery, S3/Delta)
• Datadog account with Data Streams Monitoring, Data Observability, or custom metric pipelines
• Basic familiarity with your pipeline’s DAG structure and data quality expectations

The Seven Layers and What to Monitor

1. Orchestration Scheduler crashes, task queue backlogs, dependency deadlocks, and scheduling drift. Key signals: task duration trends, queue depth, DAG success rates. Failure pattern: jobs that complete but took 3× longer than baseline — data will arrive late but appear “green” in success metrics.

2. Ingestion and transport Late-arriving data, dropped or duplicated records, schema drift. For streaming: consumer lag. For batch: file arrival timestamps vs. expected SLA. Failure pattern: schema drift from a source system change breaks downstream processing silently if there’s no schema enforcement.

3. Raw storage Incomplete writes, access failures, premature data deletion (TTL misconfigurations). Monitor partition counts, row counts by hour, and write volume anomalies.

4. Processing Transformation logic errors, delayed processing, resource contention. dbt model failures and test failures should alert differently from infrastructure failures — a logic error in a transformation produces wrong data, not missing data.

5. Refined storage Stale data (query cache not invalidated after pipeline run), partition pruning failures, and distribution shifts in key columns. Monitor p95/p99 query latency trends alongside freshness timestamps.

6. Serving Stale caches, API latency spikes when upstream data changes shape, and data drift in ML feature stores. Serving-layer monitoring catches the downstream impact of upstream failures that weren’t surfaced earlier.

7. Governance Permission changes that break access, credential expiry (service accounts with rotating keys), and compliance gaps. Often monitored least consistently — governance failures tend to surface as access errors in production.

Datadog’s Cross-Cutting Products

Three specialized products address signals that span layers:

• Data Streams Monitoring: Topology mapping of your pipeline graph with end-to-end latency tracking and bottleneck identification
• Data Observability: Quality Monitoring: Freshness, volume, and distribution anomaly detection at the dataset level
• Data Observability: Jobs Monitoring: Failure detection and duration trend tracking for batch and streaming jobs

The key architectural difference from application APM: all three prioritize data quality and integrity signals — row counts, schema validation, freshness SLAs — alongside infrastructure health metrics. APM tells you the pipeline ran; these tools tell you what it produced.

The Trap

The most common monitoring gap is treating pipeline success as a proxy for data quality. A Spark job that completed in 47 minutes (vs. its normal 12) didn’t fail — but the data it produced may be hours stale by the time it reaches serving. Alert on duration trends, not just success/failure booleans. Add explicit data freshness SLA checks as separate monitors from job completion monitors.

Cited References

• Datadog: Data Pipeline Monitoring Guide
• Datadog Data Streams Monitoring
• Datadog Data Observability
• dbt Documentation: Testing

Source: Datadog

Content created with AI assistance and reviewed for accuracy.