Adobe Firefly Cut GPU Metric Query Times 28x by Moving to Amazon Managed Prometheus

Adobe Firefly’s ML infrastructure runs training jobs across 2,000 nodes with 16,000 GPUs, scraped every 30 seconds. That produces over 1 billion data points in a single 24-hour query window. Their self-hosted Prometheus couldn’t handle it — queries that should have taken seconds were timing out at 2 minutes. The fix was a migration, but not the kind that requires rebuilding the stack.

What Happened

Adobe migrated 2 million time-series metrics from self-managed Prometheus to Amazon Managed Service for Prometheus (AMP). The performance outcome: 24-hour GPU metric queries that previously took approximately 2 minutes now complete in roughly 10 seconds — a 28.8x improvement. The monitoring window simultaneously expanded from a practical 6-hour limit to a full 24-hour view.

The migration approach was incremental rather than a cutover. Adobe used Amazon Managed Service for Prometheus collector (managed scrapers) alongside their existing deployment, maintaining continuity while shifting query load to the managed service. The team moved 2 million time series without replacing the Grafana dashboards, alerting rules, or PromQL query logic they’d built. AMP supports up to 50 million active time series per workspace, which provided headroom for Firefly’s current and projected scale.

Why It Matters

AI training infrastructure has a specific observability challenge that general-purpose monitoring doesn’t handle well: long-running jobs, high-cardinality GPU telemetry, and the need to correlate performance across hundreds or thousands of nodes simultaneously. Self-hosted Prometheus is designed for steady-state microservice monitoring — the query performance degrades sharply when scrape intervals are short and cardinality is high.

For Adobe, the 6-hour practical query window was a production constraint. Training runs for large models span days or weeks, and operational teams investigating performance regressions needed to look back further than 6 hours to correlate events. The 24-hour window that AMP enables isn’t a convenience feature — it changes what investigations are possible and how quickly on-call engineers can identify root cause during active incidents.

The built-in high availability without custom configuration also matters at this scale. Self-hosted Prometheus HA requires architecture investment — replication, deduplication, Thanos or Cortex for long-term storage. AMP delivers this as a managed baseline. The operational team recovers engineering cycles previously spent maintaining the HA setup.

Teams running agentic AI workflows in production face analogous observability challenges — high-frequency telemetry from multiple components that needs correlation across time. The AMP pattern applies equally to multi-agent infrastructure where GPU utilization, inference latency, and error rates need to be queried together.

Source: AWS Architecture Blog

Content created with AI assistance and reviewed for accuracy.