Skip to content

move kubelet --system-reserved deprecated flag to kubelet-config file #1947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ffais wants to merge 2 commits into
base: main
Choose a base branch
from
+33 −12
Open

move kubelet --system-reserved deprecated flag to kubelet-config file #1947

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0c1a5e4
move kubelet --system-reserved deprecated flag to kubelet-config file
ffais Apr 22, 2026
b6b93c9
fix resource sizing script & add version check on main.yaml and 10-ku…
ffais May 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 15 additions & 11 deletions images/capi/ansible/roles/kubernetes/files/usr/libexec/kubernetes/kubelet-resource-sizing.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

#RPM and DEB systems kubelet sysconfig PATH
KUBELET_SYSCONFIG_FILES=( "/etc/sysconfig/kubelet" "/etc/default/kubelet" )
KUBELET_CONFIG="/var/lib/kubelet/kubelet.conf.d/kubelet-resource-sizing.json"

for KUBELET_SYSCONFIG in "${KUBELET_SYSCONFIG_FILES[@]}"
do
Expand All @@ -13,7 +14,7 @@ do
# shellcheck source=/dev/null
. "${KUBELET_SYSCONFIG}"
# If system-reserved is already set by user, ignore
if grep -q 'KUBELET_EXTRA_ARGS=.*--system-reserved' "${KUBELET_SYSCONFIG}"; then
if grep -q 'systemReserved' "${KUBELET_SYSCONFIG}"; then
Copy link
Copy Markdown
Contributor

@mboersma mboersma May 11, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check no longer does what the comment above it claims. ${KUBELET_SYSCONFIG} is /etc/sysconfig/kubelet or /etc/default/kubelet — a shell-style env file consumed by the kubelet unit, not a KubeletConfiguration. A user who wants to pre-set systemReserved would do it in /var/lib/kubelet/config.yaml or in a drop-in under /var/lib/kubelet/kubelet.conf.d/, so systemReserved will essentially never appear in sysconfig and this guard will never trip.

Since this PR moves system-reserved out of KUBELET_EXTRA_ARGS and into a drop-in, I think the intent should be: "if the user has already provided a systemReserved value via the main kubelet config or any other drop-in, don't overwrite it." Something like:

# If the user has already configured systemReserved (in the main kubelet
# config or any other drop-in), don't overwrite their value.
USER_KUBELET_CONFIGS=( "/var/lib/kubelet/config.yaml" )
if [ -d /var/lib/kubelet/kubelet.conf.d ]; then
  while IFS= read -r -d '' f; do
    [ "$f" = "$KUBELET_CONFIG" ] && continue
    USER_KUBELET_CONFIGS+=( "$f" )
  done < <(find /var/lib/kubelet/kubelet.conf.d -maxdepth 1 -type f -print0)
fi

for cfg in "${USER_KUBELET_CONFIGS[@]}"; do
  [ -f "$cfg" ] || continue
  if grep -Eq '^[[:space:]]*systemReserved[[:space:]]*:' "$cfg" \
     || grep -q '"systemReserved"' "$cfg"; then
    exit 0
  fi
done

The outer loop over KUBELET_SYSCONFIG_FILES can probably be dropped entirely now — sourcing it was only useful for the old KUBELET_EXTRA_ARGS path. It's worth a sanity check that nothing downstream still relies on that.

exit 0
fi
fi
Expand Down Expand Up @@ -98,7 +99,7 @@ CPU_CORE_RESERVATION_MICROCORES=(
)

# Calculate the CPU reservation
cpu_milicores_to_reserve() {
cpu_millicores_to_reserve() {
local cpu_microcores_reserved=0

for ((i = 0; i < schedulable_cores_no; i++)); do
Expand All @@ -113,13 +114,16 @@ cpu_milicores_to_reserve() {
echo "$cpu_microcores_reserved" | awk '{result = $1 / 10; if (result != int(result)) result++; printf "%d\n", result}'
}

mkdir -p /run/kubelet
# Check if system-reserved already exists
if grep '.*--system-reserved' <<< "${KUBELET_EXTRA_ARGS}"; then
# If system-reserved is already set by a previous run, replace old value with new one and write to /run/kubelet/extra-args.env
system_reserved=$(sed -E "s|--system-reserved=cpu=[0-9]+m,memory=[0-9]+Mi|--system-reserved=cpu=$(cpu_milicores_to_reserve)m,memory=$(memory_reservation_mebibytes)Mi|" <<< "${KUBELET_EXTRA_ARGS}")
echo "KUBELET_EXTRA_ARGS=${system_reserved} >/run/kubelet/extra-args.env"
else
# If not append system-reserved to KUBELET_EXTRA_ARGS and write to /run/kubelet/extra-args.env
echo "KUBELET_EXTRA_ARGS=${KUBELET_EXTRA_ARGS} --system-reserved=cpu=$(cpu_milicores_to_reserve)m,memory=$(memory_reservation_mebibytes)Mi" >/run/kubelet/extra-args.env
mkdir -p /var/lib/kubelet/kubelet.conf.d

# Initialize config file if it doesn't exist
if [ ! -f "$KUBELET_CONFIG" ]; then
echo "{}" > "$KUBELET_CONFIG"
fi

# Get the computed values from the functions
memory_reservation_mebibytes=$(memory_reservation_mebibytes)
cpu_millicores_to_reserve=$(cpu_millicores_to_reserve)

echo "$(jq --arg memory_reservation_mebibytes "${memory_reservation_mebibytes}Mi" --arg cpu_millicores_to_reserve "${cpu_millicores_to_reserve}m" \
'. += {"apiVersion": "kubelet.config.k8s.io/v1beta1","kind": "KubeletConfiguration", "systemReserved": {"cpu": $cpu_millicores_to_reserve, "ephemeral-storage": "1Gi", "memory": $memory_reservation_mebibytes}}' "$KUBELET_CONFIG")" > "$KUBELET_CONFIG"
13 changes: 13 additions & 0 deletions images/capi/ansible/roles/kubernetes/tasks/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,19 @@
mode: "0644"
when: kubernetes_enable_automatic_resource_sizing | bool

- name: Set Kubelet Configuration Drop-in Directory argument
ansible.builtin.lineinfile:
path: "{{ systemd_prefix }}/system/kubelet.service.d/10-kubeadm.conf"
regexp: '^Environment="KUBELET_CONFIG_ARGS='
line: Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml --config-dir /var/lib/kubelet/kubelet.conf.d"
when: kubernetes_semver is version('v1.28.0', '>=') and kubernetes_enable_automatic_resource_sizing | bool

- name: Add KUBELET_CONFIG_DROPIN_DIR_ALPHA environment variable for Kubernetes v1.29 and below
ansible.builtin.lineinfile:
path: "{{ systemd_prefix }}/system/kubelet.service.d/10-kubeadm.conf"
line: Environment="KUBELET_CONFIG_DROPIN_DIR_ALPHA"
when: kubernetes_semver is version('v1.28.0', '>=') and kubernetes_semver is version('v1.29.0', '<') and kubernetes_enable_automatic_resource_sizing | bool

- name: Generate kubectl bash completion
ansible.builtin.shell:
cmd: "{{ sysusr_prefix }}/bin/kubectl completion bash > {{ sysusr_prefix }}/share/bash-completion/completions/kubectl"
Expand Down
6 changes: 5 additions & 1 deletion ...sible/roles/kubernetes/templates/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml {% if kubernetes_enable_automatic_resource_sizing | bool and kubernetes_semver is version('v1.28.0', '>=') %}--config-dir /var/lib/kubelet/kubelet.conf.d{% endif %}"
# For Kubernetes v1.28 to v1.29, you can only specify --config-dir if you also set the environment variable KUBELET_CONFIG_DROPIN_DIR_ALPHA for the kubelet process (the value of that variable does not matter).
{% if kubernetes_semver is version('v1.28.0', '>=') and kubernetes_semver is version('v1.29.0', '<') %}
Environment="KUBELET_CONFIG_DROPIN_DIR_ALPHA"
{% endif %}
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
Expand Down