How to Trace AWS Lambda Durable Functions with Datadog

AWS Lambda Durable Functions solve a specific problem: long-running, multi-step workflows — payment processing, order fulfillment, approval chains — that need to pause, checkpoint, and resume across multiple Lambda invocations without losing state. The observability problem they create is harder to see: each invocation produces separate telemetry, and connecting the full workflow trace across checkpoints requires manual correlation that quickly becomes untenable in production. Datadog’s new Lambda Durable Functions tracing, documented in their engineering blog, resolves this automatically.

The Framework

Lambda Durable Functions use a checkpoint-based execution model: a workflow can pause for a wait or callback, retry failed work, and resume in a fresh Lambda invocation without losing state. Standard multi-step workflows look like this:

export const handler = withDurableExecution(
  async (event, context: DurableContext) => {
    const order = await context.step('validate_order', ...);
    const payment = await context.step('process_payment', ...);
    await context.waitForCallback('wait_for_approval', ...);
    return context.step('ship_order', ...);
  }
);

Each step runs in a potentially separate Lambda invocation. Without distributed tracing across invocations, debugging a failed ship_order step requires manually correlating CloudWatch logs from multiple invocations across the workflow’s execution history.

Step 1: Auto-Instrumentation Setup

No code changes to the workflow itself are required. Functions already instrumented with the Datadog Lambda Library — via layer or custom runtime — automatically pick up durable function tracing for Node.js and Python. The tracer instruments the AWS Lambda Durable Execution SDK automatically, creating spans for each operation type (steps, waits, callbacks, retries).

Trace context propagation happens through a _datadog_{N} checkpoint that travels with the workflow state. This connects all invocations into a single trace without modifying user checkpoints or requiring changes to the durable function code.

Step 2: What You Get

Single unified flame graph. All operations across all invocations appear in one trace view — validate_order, process_payment, wait_for_approval, and ship_order in sequence, with timing and error data for each.

Operation-level visibility. Tags like aws.durable.operation_name and aws.durable.replayed distinguish between actual execution steps and replay iterations (Durable Functions replay completed steps on resume to reconstruct state). This is critical for accurate performance attribution — replay spans are expected behavior, not latency to investigate.

Error attachment. Error details attach to the specific operation span where the failure occurred, rather than to the Lambda invocation wrapper. Finding the exact step that failed in a complex workflow becomes a filter, not a log archaeology project.

Infrastructure tab. Lists all operations with attempt counts and timing — useful for identifying steps that are retrying excessively.

Step 3: Long-Running Workflow Retention

For workflows that span hours, days, or longer, the default trace retention window may cause spans to expire before the workflow completes. Create a retention filter using the query operation_name:aws.durable.execute in Datadog’s Trace Retention configuration to ensure consistent retention for long-running executions.

Where It Doesn’t Apply

The integration covers Node.js and Python runtimes. Other Lambda runtimes (Go, Java, .NET) are not currently instrumented. Workflows using the Durable Execution SDK from a non-Lambda invocation context also fall outside the current instrumentation scope.

Source: Datadog

Content created with AI assistance and reviewed for accuracy.