Databricks Monitoring: The 7 Dimensions Most Teams Only Partially Cover

Most Databricks monitoring setups cover jobs and clusters. Few cover SQL warehouses, streaming pipelines, and model serving with the same discipline. The gap matters because the failure modes are different — a SQL warehouse exhaustion problem looks nothing like a Spark stage bottleneck, and the signal for model serving degradation appears on GPU memory utilization, not CPU.

Here’s what comprehensive Databricks monitoring actually looks like across all seven layers of the stack.

Jobs and Pipelines

The foundational layer. Track system.lakeflow.job_run_timeline.result_state for success/failure state, and monitor job durations against historical baselines to detect regressions before they become incidents.

For pipelines, system.lakeflow_pipeline_events_preview.pipeline_events captures data quality check failures — often the first signal that something upstream changed. A sudden spike in task failures within runs (tracked via system.lakeflow.job_task_run_timeline.result_state) is typically more informative than the aggregate job failure rate.

Spark Execution

Stage and task duration via spark.stage.executor_run_time surfaces processing bottlenecks. The metric worth alerting on here is garbage collection time as a fraction of executor run time — when (major_gc_time + minor_gc_time) / executor_run_time exceeds roughly 10%, you’re spending CPU on GC instead of computation.

Shuffle operations (read/write bytes) and disk spill (spark.stage.disk_bytes_spilled) are memory pressure indicators. Disk spill in particular signals that executors are running out of memory and falling back to disk — an expensive failure mode that degrades quietly before it fails loudly.

SQL Warehouses

Queue wait time (system.query.history.waiting_at_capacity_duration_ms) is the leading indicator of warehouse exhaustion. It rises before query latency does — if you’re only alerting on p99 latency, you’re seeing the symptom, not the cause.

Track latency percentiles (p50, p90, p99 from total_duration_ms) to detect gradual performance degradation. Autoscaling frequency tells you whether your warehouse sizing is appropriate for actual load patterns. Bytes scanned (system.query.history.read_bytes) measures query efficiency — a sudden increase usually indicates a missing partition filter or a changed query plan.

Streaming

For structured streaming workloads, the critical relationship is input rate versus processing rate. Alert when input consistently exceeds processing — that’s when consumer lag starts accumulating. Trigger and batch duration via spark.structured_streaming.latency should stay below your trigger interval; exceeding it means you’re falling behind.

For Auto Loader, numFilesOutstanding and numBytesOutstanding track pending ingestion backlog. Both can sit near zero for weeks and then spike suddenly when upstream data volumes change.

Infrastructure (Classic Compute)

cpu_wait_percent is more useful than overall CPU utilization for identifying I/O bottlenecks — high CPU wait with moderate utilization points to storage constraints, not compute. Track mem_used_percent for physical memory pressure and monitor cluster state transitions (starts, scaling events, terminations, interruptions) to understand cluster lifecycle behavior under load.

Model Serving

GPU memory usage (gpu_mem_usage_percentage.max) needs a close-to-100% alert — unlike CPU, GPU memory exhaustion typically causes crashes rather than degraded performance. Set an alert threshold well before saturation.

For serving endpoints, track request count (databricks.model_serving.request_count_total), p99 latency, and both 4xx and 5xx error counts separately. Client-side errors (4xx) often indicate schema drift or input validation issues; server-side errors (5xx) point to infrastructure or model loading problems.

Cost

DBU consumption (system.billing.usage.usage_quantity) needs both threshold and trend-based alerting. A threshold alert fires when you’re approaching budget; a trend alert catches gradual drift before you hit the threshold.

Source: Datadog

Content created with AI assistance and reviewed for accuracy.