Migrating From ingress-nginx to Envoy Gateway Without Dropping a Single Request

The naive approach to migrating Kubernetes ingress controllers—delete the old resource, deploy the new one—drops requests. A CNCF-documented migration from ingress-nginx to Envoy Gateway demonstrates why that fails and what the working pattern looks like.

The DNS TTL Problem

The team’s first migration attempt worked technically: traffic moved from ingress-nginx to Envoy Gateway’s HTTPRoute without any deployment errors. But deleting the old Ingress resource before DNS TTLs expired created a window where clients resolved to an endpoint that no longer existed. The result was failed requests at exactly the moment the migration should have been clean.

This is a predictable failure mode. DNS propagation doesn’t follow your deployment schedule. Any migration that requires deleting the old resource before the new one is universally resolvable will produce this window—regardless of how carefully the deployment is sequenced.

The Weighted DNS Pattern

The fix is to never delete anything during the migration. Instead, use AWS Route 53 weighted records via ExternalDNS annotations to split traffic by percentage rather than by resource existence.

The old Ingress gets external-dns.alpha.kubernetes.io/aws-weight: "100". The new HTTPRoute gets external-dns.alpha.kubernetes.io/aws-weight: "0". Both records exist and both point to valid endpoints. At cutover, swap the weights. Traffic shifts without any resource deletion. Rollback is equally simple—swap back—without DNS recreation or propagation delay.

The team confirmed “not a single failed request” during cutover. The monitoring result wasn’t luck; the architecture made dropped requests structurally impossible because both endpoints were valid throughout the transition.

What the Migration Actually Involves

Before any DNS work, ing-switch—a migration tool that scans clusters for Ingress resources and categorizes annotations by migration impact—provides the map of what needs to change. Annotations that translate cleanly to HTTPRoute appear as low-impact changes. Custom nginx annotations without Gateway API equivalents surface as migration risk items requiring a decision rather than just a replacement.

The CNCF chose Envoy Gateway over alternatives based on CNCF project alignment, mTLS support, request buffering, and resource model design. The fact that CNCF runs it on its own infrastructure was treated as a meaningful signal about long-term maintenance trajectory.

The Gateway API resource model also exposes infrastructure/application ownership boundaries that Ingress conflated. Gateway resources (cluster-scoped, infrastructure-owned) and HTTPRoute resources (namespace-scoped, application-owned) split concerns that were previously merged in a single Ingress object. Prior to Gateway API 1.5, this created friction in multi-namespace deployments because hostname definitions had to be duplicated at both levels. The ListenerSet feature in Gateway API 1.5 addresses this, though teams evaluating the Envoy Gateway release candidate should validate support before relying on it.

ExternalDNS handles DNS record management; cert-manager handles TLS. Both are configured via annotations on the Gateway API resources, which means the application team doesn’t touch DNS or certificate infrastructure directly—those concerns stay in the infrastructure layer.

Source: CNCF Blog

Content created with AI assistance and reviewed for accuracy.