How Datadog Built an Async-Aware Python Profiler — and Why Traditional Profilers Fail Async Code

Python profilers have a fundamental problem with async code: they sample execution stacks, and asyncio tasks don’t have a continuous execution stack. A coroutine is created in one context, awaited in another, and resumed in a third. Traditional profilers see a sequence of isolated task snapshots — they capture what a coroutine was doing at a given moment but not which application path caused it to be scheduled.

Datadog’s engineering blog published the design of their async-aware profiler this week. The solution is not just technically interesting — the specific optimization that reduced profiler overhead by over 60% is a case study in diagnosing where instrumentation costs actually come from.

The Problem with Flat Profiler Output

In a synchronous Python application, a profiler samples the call stack periodically, and the output shows which functions called which other functions. The relationship between caller and callee is explicit in the stack; analysis is straightforward.

In an asyncio application, the profiler samples each task’s execution stack independently. A task that is running shows its current internal call stack. But the profiler doesn’t know that task B was awaited by task A, which was called from the HTTP request handler. It sees:

task_a: processing request
task_b: querying database
task_c: fetching external API

Three separate call stacks. No relationships. When task_c is slow, the profiler tells you task_c is slow — not that it was triggered by task_a processing a specific request type, which was the one causing latency complaints in production.

Stacked Stacks: Reconstructing Task Relationships

Datadog’s approach is to reconstruct the task dependency tree by tracking which task awaits which. When task_a calls await task_b(), the profiler records that task_b was created in the context of task_a. When task_b calls await task_c(), the relationship extends.

The profiler then builds “stacked stacks” — composite execution views that preserve the logical execution path across task boundaries, in the same way a synchronous call stack preserves the path across function calls. Instead of three isolated task snapshots, the output shows:

request_handler → task_a → task_b → task_c: external API call

This transforms the debuggability of async performance problems. When task_c is slow, the stacked stack shows which user-facing request path triggered it, which coroutine chain led to the slow call, and where in the application logic the latency originates.

The framework handles the edge case where tasks outlive their creators — when task_b is still running after task_a completes, the profiler preserves the creation-time relationship rather than losing it at task_a’s completion.

The Overhead Problem — and the Fix

The original profiler implementation used Linux’s process_vm_readv system call to read task state from memory during sampling. process_vm_readv requires a kernel transition — each call crosses the user/kernel boundary, reads from the target process’s address space, and returns. For a profiler sampling multiple tasks periodically, the kernel transition cost per sample multiplied across all tasks was becoming prohibitive.

The Datadog team profiled the profiler: process_vm_readv was the dominant cost.

The replacement: memcpy wrapped in a signal handler. When the profiler needs to read a task’s stack state, it sends a signal to the task’s thread, which triggers a signal handler that performs a memcpy of the relevant memory into a shared buffer. The profiler reads from the shared buffer — all in user space, no kernel transition per sample.

The result: over 60% reduction in profiler overhead. The same task state information, at a fraction of the instrumentation cost. This also enables higher sampling frequency, producing higher-fidelity profiles that catch more intermittent performance issues.

Production Outcomes

The async-aware profiler surfaced issues that the flat profiler missed:

Infinite loops in decompression libraries: A decompression routine entered an infinite loop under specific input conditions. The flat profiler showed elevated CPU time on the decompression function; the async-aware profiler showed exactly which request path triggered it — enabling targeted reproduction and fix.

Wasteful logging operations: Logging calls in hot paths were serializing task execution. The stacked stack output showed which coroutines were blocked on logging and which user-facing operations they were slowing. Logging operations were moved off the critical path.

NumPy optimization opportunities: NumPy operations on data prepared in specific ways were running sub-optimally. The stacked stacks identified which application flows were hitting inefficient NumPy calls, enabling targeted vectorization changes that produced 15-30% CPU reductions on those paths.

These are not problems that profilers surfaced incorrectly — they’re problems that were invisible in flat profiler output because the relationship between the slow operation and the triggering code path was obscured by asyncio’s task abstraction.

Source: Datadog

Content created with AI assistance and reviewed for accuracy.