CloudWatch Metric Streams to VPC OpenTelemetry: AWS Lambda Bridge Pattern

Stream near-real-time CloudWatch metrics into a private VPC-based OpenTelemetry collector by bridging the Firehose-to-private-endpoint gap with a Lambda transformation function — without third-party licensing costs or vendor lock-in.

Purpose

Organizations with strict data privacy requirements cannot stream CloudWatch metrics to external observability vendors without data leaving the VPC. Amazon Data Firehose cannot reach private VPC endpoints directly, creating a gap between managed metric streaming and internal OTel collector infrastructure. This pattern uses Lambda synchronously to bridge that gap.

Pre-requisites

• AWS account with CloudWatch Metric Streams, Firehose, and Lambda access
• An OpenTelemetry Collector running on EC2 (or ECS) within a private VPC subnet
• Network Load Balancer distributing TCP traffic to the OTel collector instances
• S3 bucket configured as a backup destination for Firehose (redundant delivery)
• IAM roles granting Firehose permission to invoke Lambda and Lambda permission to call the VPC endpoint

Steps

1. Enable CloudWatch Metric Streams — Configure a Metric Stream in the AWS Console targeting your Firehose delivery stream. Select JSON format output. Choose which namespaces to include (e.g., AWS/EC2, AWS/RDS, AWS/Lambda).

2. Create the Firehose delivery stream — Configure Amazon Data Firehose with your S3 bucket as the primary destination. Enable Lambda data transformation and point it to the function you will create in step 3.

3. Deploy the Lambda transform function — Write a function that receives the buffered Firehose record batch, parses the JSON metrics payload, and forwards it via HTTP POST to your internal OTel HTTP endpoint at http://your-nlb-dns:4318/v1/metrics. The Lambda function must be deployed in the same VPC as the OTel collector, with VPC config pointing to private subnets and a security group that allows outbound TCP on port 4318.

4. Configure the Network Load Balancer — Set up an NLB with a TCP listener on port 4318 routing to the OTel collector target group. This enables Lambda to reach the collector via a stable DNS name regardless of instance rotation.

5. Configure the OpenTelemetry Collector — Set up the collector’s config.yaml with an OTLP HTTP receiver on port 4318, processors for enrichment and filtering, and exporters pointing to your backends (Grafana, AWS X-Ray, Prometheus remote write, etc.). The three-stage pipeline: receivers → processors → exporters.

6. Validate end-to-end — Trigger a CloudWatch metric event, confirm Firehose receives it, verify Lambda invocation in CloudWatch Logs, and check the OTel collector’s output logs for the forwarded metric. Use the S3 fallback bucket to diagnose any Lambda failures — Firehose will write there when the transform returns an error.

7. Set S3 backup retention — Configure a lifecycle policy on the S3 bucket to expire records after your retention requirement (e.g., 30 days). This is a cost control measure, not an operational one.

The Trap

Firehose buffer configuration drives latency. Firehose buffers records before invoking Lambda, with a minimum buffer interval of 60 seconds and a maximum of 900 seconds. Setting the buffer size too large (approaching the 128 MB limit) delays Lambda invocation and increases the latency from metric emission to OTel collector receipt. For real-time alerting use cases, set both the buffer interval and buffer size to their minimums. Accept the higher Lambda invocation frequency as the cost of sub-minute latency.

Cited References

• Streaming CloudWatch Metrics to VPC-Based OpenTelemetry Collectors — AWS Architecture Blog
• OpenTelemetry Collector Configuration Docs — CNCF OpenTelemetry
• Amazon Data Firehose Data Transformation — AWS Documentation
• Stack Archive: OTel + ClickHouse Observability Pipelines — related pattern for downstream routing

Photo by Christina Morillo on Pexels

Source: AWS Architecture Blog

Content created with AI assistance and reviewed for accuracy.