How to Orchestrate On-Premises Infrastructure at Scale with EKS Anywhere and AWS Serverless
Cloud Platform EngineeringManaging on-premises Kubernetes clusters at scale — hundreds of edge sites, vendor-mixed hardware, firmware updates that take hours — has historically required either a dedicated platform engineering team per site or custom tooling that never scales past the organization that built it. AWS’s Architecture Blog publishes a reference architecture that centralizes this problem: EKS Anywhere at the edge, AWS serverless at the center, and an event-driven order management system connecting them.
The Architecture
The pattern separates concerns across three layers:
Layer 1 — Centralized orchestration on AWS. All control-plane logic lives in AWS. Amazon API Gateway provides a RESTful interface for operations. Amazon EventBridge routes incoming events to AWS Step Functions workflows. Step Functions handles the heavy lifting: it coordinates multi-step operations asynchronously, supports a callback pattern for long-running tasks (the workflow pauses and waits for an on-premises completion signal rather than polling), and uses Distributed Map to parallelize operations across thousands of servers simultaneously. Lambda handles API processing and state management. DynamoDB backs the Order Management System.
Layer 2 — Distributed EKS Anywhere clusters on-premises. Each on-premises site runs EKS Anywhere in two cluster types: management clusters (hosting the orchestration plane) and workload clusters (hosting applications). The orchestration engine on the AWS side assembles EKS Anywhere configuration files and hardware inventory CSVs, passing them to the EKS Anywhere CLI for cluster operations — creation, scaling, upgrades, and termination.
Layer 3 — Hybrid connectivity. AWS Direct Connect or Site-to-Site VPN bridges the environments. AWS Systems Manager hybrid activations register on-premises instances for unified management without requiring inbound connectivity from AWS. IAM Roles Anywhere issues short-lived credentials to on-premises workloads via certificate exchange, replacing long-lived keys at edge sites. Route 53 private hosted zones handle hybrid DNS — DynamoDB Streams trigger Lambda to update DNS automatically when infrastructure state changes.
The Order Management System
Every operation in this architecture is modeled as an “order” with a unique ID. When an API request arrives, the system immediately returns the order ID and begins async execution. Operators track progress via DynamoDB. This pattern matters because on-premises operations — firmware updates, cluster provisioning — can take hours and must survive transient connectivity loss.
The DynamoDB Inventory Management System maintains six state categories: site configurations, hardware inventory (BIOS/firmware versions, IP addresses, operational status, rack placement), cluster configurations, node group details, order lifecycle records, and reusable deployment blueprints. The full audit trail from creation through completion is queryable by order ID.
Hardware Management: Redfish Over Vendor Lock-in
For physical server management, the architecture uses the DMTF Redfish API — a vendor-agnostic standard supported by Dell, HPE, Lenovo, and others. This covers BIOS configuration, firmware updates, NIC upgrades, power management, and health monitoring without vendor-specific tooling. The practical value: a mixed-hardware data center (Dell compute + HPE storage, for example) is manageable through a single control plane.
Observability
AWS Distro for OpenTelemetry (ADOT) collectors run on each EKS Anywhere cluster, scraping metrics from server, Kubernetes, and application layers. These feed Amazon Managed Service for Prometheus, unified across all distributed sites. Amazon Managed Grafana provides the consolidated dashboard view — giving platform teams visibility across all edge locations regardless of hardware vendor or cluster configuration.
Where This Applies
This architecture is sized for organizations managing dozens or hundreds of on-premises sites: retail networks with edge compute, telecom operators running MEC infrastructure, manufacturers with distributed production compute, or enterprises that can’t move everything to the public cloud. For small-scale hybrid environments (one or two on-premises sites), the complexity exceeds the benefit — a simpler hybrid connectivity pattern with EKS Anywhere and standard Kubernetes tooling is more appropriate.
The So What
The callback pattern in Step Functions is the design decision worth noting: rather than polling for on-premises task completion, the workflow pauses with a task token and resumes when the on-premises system signals back. This is how you handle firmware updates that take 90 minutes without burning Lambda time or building custom polling loops. The full AWS Architecture Blog post covers the implementation in Part 1, with subsequent parts covering cluster management operations in detail.
Source: AWS Architecture Blog
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →