Datadog's CDN Log Fix: Archive Search Lets You Query 751 GB of Logs Without Paying to Index Them

Most platform teams running CDN-level traffic face the same observability tax: CDN access logs arrive at volumes that would cost more to index in a log management platform than they’re worth. The typical response is to either sample them heavily (and miss the tail events), dump them to object storage (and lose the ability to search them), or index them and pay the bill. Datadog’s Archive Search changes the economics of the third option.

What Happened

Datadog announced Archive Search in its June 2026 release cycle: a capability that lets teams query logs stored in cold object storage (S3, GCS, or Azure Blob) using the same search syntax as Datadog Log Explorer, without requiring those logs to be indexed in Datadog’s paid indexing tier.

The implementation pairs Archive Search with Observability Pipelines, Datadog’s log routing layer:

1. Observability Pipelines splits the CDN log stream at ingestion. High-volume CDN access logs (every request, every status code) route to object storage. High-signal events — errors, anomalies, specific status codes — route to Datadog’s indexing tier for real-time visibility.

2. Archive Search queries the cold tier on demand. When you need to investigate a specific CDN event that happened 10 days ago, you run a query against the archive. Datadog scanned 751 GB of cold logs to find 242 matching log entries in a documented example.

3. Query Preview runs before the full scan. Before committing a cold-storage query that scans terabytes, you can preview the query scope to estimate cost and confirm the filter will return useful results.

Results from an Archive Search are available for 24 hours in a familiar Log Explorer view. The search syntax is identical — no new query language to learn for incidents that require going back to cold storage.

Why It Matters

The CDN log cost problem is a specific version of a broader observability trap: logs with low per-event signal but high aggregate diagnostic value. Each individual CDN access log line (a 200 response for a static asset) tells you almost nothing. The aggregate across a 4-hour incident window where your CDN started returning 503s on specific edge nodes tells you everything.

Traditional approaches to this trap all have a different failure mode:

• Full indexing: accurate but expensive; a high-traffic CDN can generate log bills that dwarf the value of the data
• Sampling: cheap but lossy; precisely the edge node pattern you need to diagnose is often in the sampled-out fraction
• Raw object storage only: cheap and complete, but Athena/BigQuery queries on cold storage require a different tool, a different query language, and a different workflow than the rest of your observability practice

Archive Search keeps the complete log record (no sampling), keeps it in cheap object storage (no indexing cost for the bulk tier), and queries it with the same syntax platform engineers use every day. The PII redaction capability — which runs before logs leave your environment into the object storage tier — addresses the compliance requirement that often drives teams toward sampling (to avoid storing PII in a log management platform).

The 751 GB / 242 result example from Datadog’s documentation is instructive: the scan found the needle without requiring the haystack to be pre-indexed. That’s the operational model Archive Search enables.

What This Means for Platform Engineers

Cost modeling for CDN observability gets a new tier. The previous architecture choice was binary: index CDN logs at full volume (expensive) or don’t (incomplete). Archive Search adds a third option: route full-volume CDN logs to object storage via Observability Pipelines, pay object storage rates for retention, and query on demand when incident investigation requires it.

Incident investigation timelines change. Cold storage queries add latency to incident investigation — you’re not querying an indexed real-time stream, you’re scanning object storage. Query Preview helps estimate scope before running. For incidents that require historical CDN log data, that latency is acceptable; for real-time alerting, the indexed high-signal tier remains necessary.

PII handling simplifies the compliance conversation. If your CDN logs contain user identifiers, IP addresses, or other PII, Archive Search’s on-prem redaction (via Observability Pipelines transformation before storage) lets you retain logs for investigation without storing PII in a third-party SaaS platform. That’s a meaningful compliance architecture simplification for teams subject to GDPR or CCPA data minimization requirements.

Source: Datadog Blog

Content created with AI assistance and reviewed for accuracy.