Last December at re:Invent 2024, AWS announced a new mode for managing Amazon EKS clusters: Amazon EKS Auto Mode. AWS promises that this mode simplifies cluster operations, improves applications’ performance, availability, and security, and continuously optimizes compute costs.
This article will explore how this new management mode compares to other node management methods. This will help you assess whether adopting this feature to manage your cluster makes sense.
Dynamic Compute Scaling with Karpenter and Amazon EKS Auto Mode
Orchestrating containers with Amazon EKS can become challenging when deploying and managing many containers while ensuring scalability and optimizing resources and costs.
You might also face challenges related to isolating workloads between teams and nodes. For example, you may need to dedicate certain instance types to specific workloads.
Karpenter is an open-source tool initially developed by AWS. It addresses these challenges by allowing you to customize scaling policies and node assignments for your Amazon EKS workloads.
However, using Karpenter introduces new challenges. You now need to manage the lifecycle of a new component and ensure its maintenance, as it becomes critical for cluster administration.
We usually recommend isolating Karpenter controllers from other workload nodes when deploying them. One method could be deploying them on dedicated EC2 instances or Fargate nodes. This ensures the consistent assignment of a controller to a compute instance.
Amazon EKS with EC2 instances cluster architecture without Auto Mode
Source: AWS re:Invent 2024, Session: “Simplify Kubernetes workloads with Karpenter & Amazon Amazon EKS Auto Mode (KUB312)”
EKS Auto Mode, we shift to a model where the customer is no longer responsible for managing the lifecycle of the Karpenter controller (as well as Storage and Networking controllers).
In practice, when starting a cluster with Amazon EKS Auto Mode with only the Control Plane deployed, you only need to initiate a deployment to deploy pods in the cluster.
EKS Auto Mode then automatically deploys the nodes based on the Node Pools defined beforehand. As a reminder, a Node Pool defines the strategy for assigning pods to a pool of instance types. You can configure this in the settings.
Amazon EKS with EC2 instances cluster architecture with Auto-Mode
Source: AWS re:Invent 2024, Session: “Simplify Kubernetes workloads with Karpenter & Amazon EKS Auto Mode (KUB312)”
Furthermore, when creating an EKS Auto Mode cluster, there are two types of Node Pools options: “system” and “general-purpose“.
The “system” Node Pool allows you to provision instances for deploying custom add-ons that you want to isolate from the instances used for your workloads.
The “general-purpose” Node Pool simplifies the starting experience and allows you to deploy pods on general-purpose on-demand instances easily.
Of course, you can go beyond these ones to create Node Pools that meet your specific needs. For example, you can define Node Pools that prioritize spot instances to optimize costs or GPU instances for AI workloads.
You use a Node Class object specific to EKS Auto Mode to preconfigure the instances created in these cases. This allows you to define security groups, subnet IDs, and storage configuration, among other things. However, it’s important to note that you cannot specify an AMI ID.
This is because a key feature of Auto Mode is the cluster’s automated upgrade management, which relies on updating the AMIs used by the data plane nodes.
These nodes run with an AMI enforced by AWS based on Bottlerocket by default (an OS designed explicitly by AWS for running containers). Therefore, you cannot use your custom AMIs in this case.
Example of a NodeClass definition
Source: AWS re:Invent 2024, Session: “Simplify Kubernetes workloads with Karpenter & Amazon EKS Auto Mode (KUB312)”
Automatic Cluster Upgrades
Before discussing the details of the automatic cluster update feature, it’s important to remember that you always have the option to trigger the Control Plane upgrade.
Then only Amazon EKS checks the managed controller versions and updates only the controllers that are not up to date.
Source: AWS re:Invent 2024, Session: “Simplify Kubernetes workloads with Karpenter & Amazon EKS Auto Mode (KUB312)”
The upgrade process can be triggered in two scenarios for Data Plane nodes.
1- When the Control Plane is upgraded to a new Kubernetes version, the Data Plane nodes are replaced with an AMI that matches the Kubernetes version of the Control Plane
Source: AWS re:Invent 2024, Session: “Simplify Kubernetes workloads with Karpenter & Amazon EKS Auto Mode (KUB312)”
2- When a new AMI is released and incorporates the latest OS security patches, the data plane nodes are replaced with the updated AMI.
Source: AWS re:Invent 2024, Session: “Simplify Kubernetes workloads with Karpenter & Amazon EKS Auto Mode (KUB312)”
These processes respect the lifecycle of nodes managed by Karpenter and the disruption budget, which you’ll need to define carefully to avoid unwanted downtime. It’s also important to remember that if nodes are not updated despite the various events affecting Karpenter instances, AWS forces the update on the 21st of each month.
Pricing and Cost Considerations
There’s no change in Control Plane costs. Amazon EKS Auto Mode follows the same pricing model, which is a fixed price of $0.10/hour or $72/month for standard support ($0.60/hour for extended support).
However, the pricing for the data plane is quite different. An additional cost applies to EC2 instances, which varies based on the instance type launched. Based on the types of instances I checked, I found an approximate 12% extra fee.
It’s essential to consider this cost factor when evaluating EKS Auto Mode. If cost reduction is a top priority, other options like manually managing autoscaling with Karpenter might be more advantageous. However, if simplifying operations and reducing operational overhead are essential factors, the benefits of EKS Auto Mode can justify the additional cost.
In this above example, we see that for three instances of different types, enabling Auto Mode adds an extra fee of $125.62 per month, which makes a total of $1,172.44 for data plane costs.
Source: AWS website, Amazon EKS Pricing | Managed Kubernetes Service
Key questions to ask yourself when considering EKS Auto Mode
To help you choose the best option for managing your Amazon EKS cluster with nodes auto-scaling capabilities, here are some questions that can be useful to ask yourself to consider if EKS Auto Mode is a good fit for your use case:
Is simplifying cluster operations and reducing operational overhead a priority?
If managing key components like Karpenter, AWS Load Balancer Controller, or EBS CSI consumes significant time and effort, EKS Auto Mode could be an excellent option as it reduces operational overhead. It should also justify an additional cost for dataplane nodes.
How critical is compliance with internal policies?
If your organization enforces strict compliance requirements, such as running instances with custom AMIs or installing specific agents and software (e.g., logging, monitoring, or security tools), then EKS Auto Mode might not be the right fit. It only supports AWS-provided AMIs, specifically based on Bottlerocket.
Do I need advanced network configuration capabilities?
EKS Auto Mode may not be the right choice if your use case requires, for example:
- Using third-party CNIs instead of AWS VPC CNI (Cilium, Calico..)
- Assigning custom CIDR ranges for pods distinct from the VPC’s CIDRs.
- Leveraging advanced features like ENI/IP/Prefix warming or Security Groups per Pod.
Do I want to benefit from automatic cluster, nodes, managed controllers upgrades?
If you want to reduce time spent to upgrade your cluster components, and you don’t require granular control over the upgrade process, then EKS Auto Mode is a strong contender
Based on these questions, I recapped them with a decision-making diagram. Depending of your needs, it will guide you to identify if EKS Auto Mode is a good choice in comparison with Amazon EKS with EC2 instances managed by Karpenter
Conclusion
Amazon EKS Auto Mode offers a powerful solution for managing Kubernetes clusters:
- Simplified operations
- Automated scaling
- Easy node management
Its ability to handle the lifecycle of key components like Karpenter, networking, and storage controllers and provide automatic upgrades for clusters, nodes, and managed controllers makes it an attractive option for teams looking to reduce operational overhead, improve application performance, availability, and security, and optimize compute costs.
However, this simplicity comes with trade-offs, like limited customisation, reduced granular control and additional costs for data plane nodes. These constraints can deter teams with strict compliance requirements or advanced customisation needs.