Skip to content

Commit 3a23761

Browse files
committed
Clarify maxPods precedence for MNG vs self-managed nodes
Add new "How maxPods is determined" section with anchor and cross-reference SIM: https://t.corp.amazon.com/P384180987 cr: https://code.amazon.com/reviews/CR-255571004
1 parent 07d09a0 commit 3a23761

3 files changed

Lines changed: 31 additions & 4 deletions

File tree

latest/ug/nodes/choosing-instance-type.adoc

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,9 +104,36 @@ The max Pods calculator script limits the return value to `110` based on https:/
104104
105105
====
106106

107+
[#max-pods-precedence]
108+
== How maxPods is determined
109+
110+
The final `maxPods` value applied to a node depends on several components that interact in a specific order of precedence. Understanding this order helps you avoid unexpected behavior when customizing `maxPods`.
111+
112+
*Order of precedence (highest to lowest):*
113+
114+
. *Managed node group enforcement* – When you use a managed node group without a <<launch-template-custom-ami,custom AMI>>, Amazon EKS enforces a cap on `maxPods` in the node's user data. For instances with less than 30 vCPUs, the cap is `110`. For instances with greater than 30 vCPUs, the cap is `250`. This value takes precedence over any other `maxPods` configuration, including `maxPodsExpression`.
115+
. *kubelet `maxPods` configuration* – If you set `maxPods` directly in the kubelet configuration (for example, through a launch template with a custom AMI), this value takes precedence over `maxPodsExpression`.
116+
. *nodeadm `maxPodsExpression`* – If you use https://awslabs.github.io/amazon-eks-ami/nodeadm/doc/examples/#defining-a-max-pods-expression[`maxPodsExpression`] in your `NodeConfig`, nodeadm evaluates the expression to calculate `maxPods`. This is only effective when the value is not already set by a higher-precedence source.
117+
. *Default ENI-based calculation* – If no other value is set, the AMI calculates `maxPods` based on the number of elastic network interfaces and IP addresses supported by the instance type. This is equivalent to the formula `(number of ENIs × (IPs per ENI − 1)) + 2`. The `+ 2` accounts for the Amazon VPC CNI and `kube-proxy` running on every node, which don't consume a Pod IP address.
118+
119+
[IMPORTANT]
120+
====
121+
If you use a managed node group and set `maxPodsExpression` in your `NodeConfig`, the managed node group's enforcement overrides your expression. To use a custom `maxPods` value with managed node groups, you must specify a custom AMI in your launch template and set `maxPods` directly. For more information, see <<launch-templates>>.
122+
====
123+
124+
*Managed node groups vs. self-managed nodes*
125+
126+
With managed node groups (without a custom AMI), Amazon EKS injects the `maxPods` value into the node's bootstrap user data. This means:
127+
128+
* The `maxPods` value is always capped at `110` or `250` depending on instance size.
129+
* Any `maxPodsExpression` you configure is overridden by this injected value.
130+
* To use a different `maxPods` value, specify a custom AMI in your launch template and pass `--use-max-pods false` along with `--kubelet-extra-args '--max-pods=[.replaceable]``my-value``'` to the `bootstrap.sh` script. For examples, see <<launch-templates>>.
131+
132+
With self-managed nodes, you have full control over the bootstrap process. You can use `maxPodsExpression` in your `NodeConfig` or pass `--max-pods` directly to `bootstrap.sh`.
133+
107134
== Considerations for EKS Auto Mode
108135

109136
EKS Auto Mode limits the number of pods on nodes to the lower of:
110137

111138
* 110 pods hard cap
112-
* The result of the max pods calculation described above.
139+
* The result of the max pods calculation described above.

latest/ug/nodes/create-managed-node-group.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ If this is your first time launching an Amazon EKS managed node group, we recomm
1919
2020
* Amazon EKS nodes are standard Amazon EC2 instances. You're billed based on the normal Amazon EC2 prices. For more information, see link:ec2/pricing/[Amazon EC2 Pricing,type="marketing"].
2121
* You can't create managed nodes in an {aws} Region where you have {aws} Outposts or {aws} Wavelength enabled. You can create self-managed nodes instead. For more information, see <<launch-workers>>, <<launch-windows-workers>>, and <<launch-node-bottlerocket>>. You can also create a self-managed Amazon Linux node group on an Outpost. For more information, see <<eks-outposts-self-managed-nodes>>.
22-
* If you don't <<launch-template-custom-ami,specify an AMI ID>> for the `bootstrap.sh` file included with Amazon EKS optimized Linux or Bottlerocket, managed node groups enforce a maximum number on the value of `maxPods`. For instances with less than 30 vCPUs, the maximum number is `110`. For instances with greater than 30 vCPUs, the maximum number jumps to `250`. These numbers are based on https://github.com/kubernetes/community/blob/master/sig-scalability/configs-and-limits/thresholds.md[Kubernetes scalability thresholds] and recommended settings by internal Amazon EKS scalability team testing. For more information, see the link:containers/amazon-vpc-cni-increases-pods-per-node-limits[Amazon VPC CNI plugin increases pods per node limits,type="blog"] blog post.
22+
* If you don't <<launch-template-custom-ami,specify an AMI ID>> for the `bootstrap.sh` file included with Amazon EKS optimized Linux or Bottlerocket, managed node groups enforce a maximum number on the value of `maxPods`. For instances with less than 30 vCPUs, the maximum number is `110`. For instances with greater than 30 vCPUs, the maximum number jumps to `250`. This enforcement overrides other `maxPods` configurations, including `maxPodsExpression`. For more information about how `maxPods` is determined and how to customize it, see <<max-pods-precedence>>.
2323
2424
====
2525

latest/ug/nodes/launch-templates.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -324,7 +324,7 @@ aws eks describe-cluster --query "cluster.endpoint" --output text --name my-clus
324324
----
325325
aws eks describe-cluster --query "cluster.kubernetesNetworkConfig.serviceIpv4Cidr" --output text --name my-cluster --region region-code
326326
----
327-
** This example provides a `kubelet` argument to set a custom `max-pods` value using the `bootstrap.sh` script included with the Amazon EKS optimized AMI. The node group name can't be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. For help with selecting [.replaceable]`my-max-pods-value`, see <<determine-max-pods>>.
327+
** This example provides a `kubelet` argument to set a custom `max-pods` value using the `bootstrap.sh` script included with the Amazon EKS optimized AMI. The node group name can't be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. For help with selecting [.replaceable]`my-max-pods-value`, see <<determine-max-pods>>. For more information about how `maxPods` is determined when using managed node groups, see <<max-pods-precedence>>.
328328
+
329329
[source,yaml,subs="verbatim,attributes"]
330330
----
@@ -385,7 +385,7 @@ aws eks describe-cluster --query "cluster.endpoint" --output text --name my-clus
385385
----
386386
aws eks describe-cluster --query "cluster.kubernetesNetworkConfig.serviceIpv4Cidr" --output text --name my-cluster --region region-code
387387
----
388-
** This example provides a `kubelet` argument to set a custom `max-pods` value using the `bootstrap.sh` script included with the Amazon EKS optimized AMI. For help with selecting [.replaceable]`my-max-pods-value`, see <<determine-max-pods>>.
388+
** This example provides a `kubelet` argument to set a custom `max-pods` value using the `bootstrap.sh` script included with the Amazon EKS optimized AMI. For help with selecting [.replaceable]`my-max-pods-value`, see <<determine-max-pods>>. For more information about how `maxPods` is determined when using managed node groups, see <<max-pods-precedence>>.
389389
+
390390
[source,none,subs="verbatim,attributes"]
391391
----

0 commit comments

Comments
 (0)