Datadog JFR: Why CPU-Time Profiling Beats Wall-Clock in JDK 25

Java engineers have trusted Java Flight Recorder for years to diagnose everything from garbage collection stalls to algorithmic bottlenecks, but the profiler’s stack samples have always been quietly lying to them about where CPU cycles actually go.

Why This Category Matters

Java Flight Recorder (JFR) is, in Datadog’s framing, “a low-overhead system built into the JVM and designed for continuous, always-on use” — the default choice for teams running always-on profiling in production rather than firing up a diagnostic tool only when something breaks. The problem is what JFR has historically sampled. Traditional wall-clock sampling advances its clock whether a thread is running, waiting on I/O, or blocked on a lock, so the resulting stack traces are useful for finding latency sources but are “biased”: samples aren’t collected in proportion to actual CPU time, which makes some hotspots look more or less significant than they really are. For teams trying to answer a narrower question, namely which methods are actually burning processor cycles, that bias has been a persistent source of misdirected optimization work.

What Makes It Different

JDK 25 closes that gap with a new, first-class CPU sampling event in JFR, introduced as an experimental feature under JEP 509. Rather than sampling on elapsed time, the new mechanism records stacks only while a thread is actively executing on CPU, which is the core distinction Datadog draws: CPU-time sampling “finds computation hotspots — the methods burning the most cycles,” while wall-clock sampling “finds latency sources — the places where time passes whether or not work is happening.” Both questions are legitimate, but conflating them under one sampling method has been the actual defect.

The mechanism behind it is cooperative stack walking, developed by Oracle engineers specifically to make this kind of sampling safe at scale. Instead of performing asynchronous stack unwinding at arbitrary points in execution, it “walks the stack only at safepoints, where JVM structures are guaranteed to be consistent.” That constraint is what makes the new event stable enough to ship as a first-class JFR feature rather than a fragile add-on.

That fragility is exactly what Datadog’s own profiler has had to work around until now. The company’s tooling “relies on JFR for low-overhead telemetry data and uses AsyncGetCallTrace and another technique, vmstructs walking, for accurate CPU sampling.” AsyncGetCallTrace is a well-known internal HotSpot mechanism, not a supported public API, and under heavy load “it can trigger faults that must be carefully handled to avoid destabilizing the JVM.” A safepoint-based, JFR-native alternative removes that operational risk from the equation for anyone building on top of it.

The Catch / Who It’s For

The feature is explicitly experimental. JEP 509 has not stabilized, and Datadog is direct that it needs “broader adoption and real-world validation” before graduating out of that status. Teams still running production workloads on JDK versions before 25 gain nothing from this immediately, and anyone adopting it early should treat it the way any experimental JFR feature deserves to be treated: useful for investigation, not yet a foundation to build alerting thresholds on. There’s also no packaged product here to evaluate in the traditional sense — this is a JVM-level capability that Datadog and other profiler vendors will build on top of, not a toggle you flip in a dashboard today. Engineers doing hands-on JVM performance work, and platform teams evaluating how their observability vendor’s Java profiler is built under the hood, are the audience who should care right now. Teams whose Java footprint is small, or who are years away from a JDK 25 upgrade, can safely treat this as background context for the moment. It pairs naturally with how observability teams are already rethinking what they measure in the AI era, where cycle-accurate performance data becomes one more signal in a broader measurement shift.

Source: Datadog

Content created with AI assistance and reviewed for accuracy.