From cef29266f0339962cee4c71366f81de5afffe4fa Mon Sep 17 00:00:00 2001 From: Michael Burke Date: Fri, 12 Dec 2025 10:37:23 -0500 Subject: [PATCH 1/3] OSDOCS 17429 AutoSizingReserved is enabled by default --- ...odes-nodes-resources-configuring-auto.adoc | 113 ------------------ ...s-nodes-resources-configuring-setting.adoc | 52 ++++++-- .../nodes-nodes-resources-configuring.adoc | 2 - 3 files changed, 43 insertions(+), 124 deletions(-) delete mode 100644 modules/nodes-nodes-resources-configuring-auto.adoc diff --git a/modules/nodes-nodes-resources-configuring-auto.adoc b/modules/nodes-nodes-resources-configuring-auto.adoc deleted file mode 100644 index b60b045fb916..000000000000 --- a/modules/nodes-nodes-resources-configuring-auto.adoc +++ /dev/null @@ -1,113 +0,0 @@ -// Module included in the following assemblies: -// -// * nodes/nodes-nodes-resources-configuring.adoc - -:_mod-docs-content-type: PROCEDURE -[id="nodes-nodes-resources-configuring-auto_{context}"] -= Automatically allocating resources for nodes - -{product-title} can automatically determine the optimal `system-reserved` CPU and memory resources for nodes associated with a specific machine config pool and update the nodes with those values when the nodes start. By default, the `system-reserved` CPU is `500m` and `system-reserved` memory is `1Gi`. - -To automatically determine and allocate the `system-reserved` resources on nodes, create a `KubeletConfig` custom resource (CR) to set the `autoSizingReserved: true` parameter. A script on each node calculates the optimal values for the respective reserved resources based on the installed CPU and memory capacity on each node. The script takes into account that increased capacity requires a corresponding increase in the reserved resources. - -Automatically determining the optimal `system-reserved` settings ensures that your cluster is running efficiently and prevents node failure due to resource starvation of system components, such as CRI-O and kubelet, without your needing to manually calculate and update the values. - -This feature is disabled by default. - -.Prerequisites - -. Obtain the label associated with the static `MachineConfigPool` object for the type of node you want to configure by entering the following command: -+ -[source,terminal] ----- -$ oc edit machineconfigpool ----- -+ -For example: -+ -[source,terminal] ----- -$ oc edit machineconfigpool worker ----- -+ -.Example output -[source,yaml] ----- -apiVersion: machineconfiguration.openshift.io/v1 -kind: MachineConfigPool -metadata: - creationTimestamp: "2022-11-16T15:34:25Z" - generation: 4 - labels: - pools.operator.machineconfiguration.openshift.io/worker: "" <1> - name: worker -#... ----- -<1> The label appears under `Labels`. -+ -[TIP] -==== -If an appropriate label is not present, add a key/value pair such as: - ----- -$ oc label machineconfigpool worker custom-kubelet=small-pods ----- -==== - -.Procedure - -. Create a custom resource (CR) for your configuration change: -+ -.Sample configuration for a resource allocation CR -[source,yaml] ----- -apiVersion: machineconfiguration.openshift.io/v1 -kind: KubeletConfig -metadata: - name: dynamic-node <1> -spec: - autoSizingReserved: true <2> - machineConfigPoolSelector: - matchLabels: - pools.operator.machineconfiguration.openshift.io/worker: "" <3> -#... ----- -<1> Assign a name to CR. -<2> Add the `autoSizingReserved` parameter set to `true` to allow {product-title} to automatically determine and allocate the `system-reserved` resources on the nodes associated with the specified label. To disable automatic allocation on those nodes, set this parameter to `false`. -<3> Specify the label from the machine config pool that you configured in the "Prerequisites" section. You can choose any desired labels for the machine config pool, such as `custom-kubelet: small-pods`, or the default label, `pools.operator.machineconfiguration.openshift.io/worker: ""`. -+ -The previous example enables automatic resource allocation on all worker nodes. {product-title} drains the nodes, applies the kubelet config, and restarts the nodes. - -. Create the CR by entering the following command: -+ -[source,terminal] ----- -$ oc create -f .yaml ----- - -.Verification - -. Log in to a node you configured by entering the following command: -+ -[source,terminal] ----- -$ oc debug node/ ----- - -. Set `/host` as the root directory within the debug shell: -+ -[source,terminal] ----- -# chroot /host ----- - -. View the `/etc/node-sizing.env` file: -+ -.Example output -[source,terminal] ----- -SYSTEM_RESERVED_MEMORY=3Gi -SYSTEM_RESERVED_CPU=0.08 ----- -+ -The kubelet uses the `system-reserved` values in the `/etc/node-sizing.env` file. In the previous example, the worker nodes are allocated `0.08` CPU and 3 Gi of memory. It can take several minutes for the optimal values to appear. diff --git a/modules/nodes-nodes-resources-configuring-setting.adoc b/modules/nodes-nodes-resources-configuring-setting.adoc index 711bd71b50d9..b5fd124a165d 100644 --- a/modules/nodes-nodes-resources-configuring-setting.adoc +++ b/modules/nodes-nodes-resources-configuring-setting.adoc @@ -6,14 +6,17 @@ [id="nodes-nodes-resources-configuring-setting_{context}"] = Manually allocating resources for nodes -{product-title} supports the CPU and memory resource types for allocation. The `ephemeral-resource` resource type is also supported. For the `cpu` type, you specify the resource quantity in units of cores, such as `200m`, `0.5`, or `1`. For `memory` and `ephemeral-storage`, you specify the resource quantity in units of bytes, such as `200Ki`, `50Mi`, or `5Gi`. By default, the `system-reserved` CPU is `500m` and `system-reserved` memory is `1Gi`. +[role="_abstract"] +As an administrator, you can manually set `system-reserved` CPU and memory resources for your nodes. Setting these values ensures that your cluster is running efficiently and prevents node failure due to resource starvation of system components. -As an administrator, you can set these values by using a kubelet config custom resource (CR) through a set of `=` pairs -(e.g., `cpu=200m,memory=512Mi`). +By default, {product-title} uses a script on each node to automatically determine the optimal `system-reserved` CPU and memory resources for nodes associated with a specific machine config pool and update the nodes with those values. {product-title} supports the CPU and memory resource types for allocation. + +However, you can manually set these values by using a `KubeletConfig` custom resource (CR) through a set of `=` pairs +(e.g., `cpu=200m,memory=512Mi`). For the `cpu` type, you specify the resource quantity in units of cores, such as `200m`, `0.5`, or `1`. For `memory` and `ephemeral-storage`, you specify the resource quantity in units of bytes, such as `200Ki`, `50Mi`, or `5Gi`. By default, the `system-reserved` CPU is `500m` and `system-reserved` memory is `1Gi`. Also, use the `autoSizingReserved: false` parameter to override the default {product-title} behavior of automatically setting the `systemReserved` values. [IMPORTANT] ==== -You must use a kubelet config CR to manually set resource values. You cannot use a machine config CR. +You must use a `KubeletConfig` CR to manually set resource values. You cannot use a machine config CR. ==== For details on the recommended `system-reserved` values, refer to the link:https://access.redhat.com/solutions/5843241[recommended system-reserved values]. @@ -70,18 +73,20 @@ kind: KubeletConfig metadata: name: set-allocatable <1> spec: + autoSizingReserved: false <2> machineConfigPoolSelector: matchLabels: - pools.operator.machineconfiguration.openshift.io/worker: "" <2> + pools.operator.machineconfiguration.openshift.io/worker: "" <3> kubeletConfig: - systemReserved: <3> + systemReserved: <4> cpu: 1000m - memory: 1Gi + memory: 4Gi #... ---- <1> Assign a name to CR. -<2> Specify the label from the machine config pool. -<3> Specify the resources to reserve for the node components and system components. +<2> Specify `false` to override the default {product-title} behavior of automatically setting the `systemReserved` values. +<3> Specify the label from the machine config pool. +<4> Specify the resources to reserve for the node components and system components. . Run the following command to create the CR: + @@ -89,3 +94,32 @@ spec: ---- $ oc create -f .yaml ---- + +.Verification + +. Log in to a node you configured by entering the following command: ++ +[source,terminal] +---- +$ oc debug node/ +---- + +. Set `/host` as the root directory within the debug shell: ++ +[source,terminal] +---- +# chroot /host +---- + +. View the `/etc/node-sizing.env` file: ++ +.Example output +[source,terminal] +---- +NODE_SIZING_ENABLED=false +SYSTEM_RESERVED_MEMORY=4Gi +SYSTEM_RESERVED_CPU=1000m +# ... +---- ++ +The kubelet uses the `system-reserved` values in the `/etc/node-sizing.env` file. diff --git a/nodes/nodes/nodes-nodes-resources-configuring.adoc b/nodes/nodes/nodes-nodes-resources-configuring.adoc index 30863891e674..e7a081380644 100644 --- a/nodes/nodes/nodes-nodes-resources-configuring.adoc +++ b/nodes/nodes/nodes-nodes-resources-configuring.adoc @@ -30,8 +30,6 @@ include::modules/sd-understanding-process-id-limits.adoc[leveloffset=+1] include::modules/risks-setting-higher-process-id-limits.adoc[leveloffset=+2] -include::modules/nodes-nodes-resources-configuring-auto.adoc[leveloffset=+1] - include::modules/nodes-nodes-resources-configuring-setting.adoc[leveloffset=+1] //// From 42e2c7a238fad53e712ee41f4b34383a30c0c150 Mon Sep 17 00:00:00 2001 From: Michael Burke Date: Fri, 12 Dec 2025 11:26:21 -0500 Subject: [PATCH 2/3] added es --- modules/nodes-nodes-resources-configuring-setting.adoc | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/modules/nodes-nodes-resources-configuring-setting.adoc b/modules/nodes-nodes-resources-configuring-setting.adoc index b5fd124a165d..663f8e9d3e8f 100644 --- a/modules/nodes-nodes-resources-configuring-setting.adoc +++ b/modules/nodes-nodes-resources-configuring-setting.adoc @@ -11,8 +11,7 @@ As an administrator, you can manually set `system-reserved` CPU and memory resou By default, {product-title} uses a script on each node to automatically determine the optimal `system-reserved` CPU and memory resources for nodes associated with a specific machine config pool and update the nodes with those values. {product-title} supports the CPU and memory resource types for allocation. -However, you can manually set these values by using a `KubeletConfig` custom resource (CR) through a set of `=` pairs -(e.g., `cpu=200m,memory=512Mi`). For the `cpu` type, you specify the resource quantity in units of cores, such as `200m`, `0.5`, or `1`. For `memory` and `ephemeral-storage`, you specify the resource quantity in units of bytes, such as `200Ki`, `50Mi`, or `5Gi`. By default, the `system-reserved` CPU is `500m` and `system-reserved` memory is `1Gi`. Also, use the `autoSizingReserved: false` parameter to override the default {product-title} behavior of automatically setting the `systemReserved` values. +However, you can manually set these values by using a `KubeletConfig` custom resource (CR) through a set of `=` pairs (e.g., `cpu=200m,memory=512Mi`). For the `cpu` type, you specify the resource quantity in units of cores, such as `200m`, `0.5`, or `1`. For `memory` and `ephemeral-storage`, you specify the resource quantity in units of bytes, such as `200Ki`, `50Mi`, or `5Gi`. By default, the `system-reserved` CPU is `500m` and `system-reserved` memory is `1Gi`. Also, use the `autoSizingReserved: false` parameter to override the default {product-title} behavior of automatically setting the `systemReserved` values. [IMPORTANT] ==== @@ -81,6 +80,7 @@ spec: systemReserved: <4> cpu: 1000m memory: 4Gi + ephemeral-storage: 50Mi #... ---- <1> Assign a name to CR. @@ -116,10 +116,9 @@ $ oc debug node/ .Example output [source,terminal] ---- -NODE_SIZING_ENABLED=false SYSTEM_RESERVED_MEMORY=4Gi SYSTEM_RESERVED_CPU=1000m -# ... +SYSTEM_RESERVED_ES=50Mi ---- + The kubelet uses the `system-reserved` values in the `/etc/node-sizing.env` file. From b405ef75f1ce7e5f87f7b00f4f23fe28830cf611 Mon Sep 17 00:00:00 2001 From: Michael Burke Date: Fri, 12 Dec 2025 11:45:46 -0500 Subject: [PATCH 3/3] fix build error --- nodes/nodes/nodes-nodes-resources-configuring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nodes/nodes/nodes-nodes-resources-configuring.adoc b/nodes/nodes/nodes-nodes-resources-configuring.adoc index e7a081380644..b390a1dae99c 100644 --- a/nodes/nodes/nodes-nodes-resources-configuring.adoc +++ b/nodes/nodes/nodes-nodes-resources-configuring.adoc @@ -6,7 +6,7 @@ include::_attributes/common-attributes.adoc[] toc::[] -To provide more reliable scheduling and minimize node resource overcommitment, reserve a portion of the CPU and memory resources for use by the underlying node components, such as `kubelet` and `kube-proxy`, and the remaining system components, such as `sshd` and `NetworkManager`. By specifying the resources to reserve, you provide the scheduler with more information about the remaining CPU and memory resources that a node has available for use by pods. You can allow {product-title} to xref:../../nodes/nodes/nodes-nodes-resources-configuring.adoc#nodes-nodes-resources-configuring-auto_nodes-nodes-resources-configuring[automatically determine the optimal `system-reserved` CPU and memory resources] for your nodes or you can xref:../../nodes/nodes/nodes-nodes-resources-configuring.adoc#nodes-nodes-resources-configuring-setting_nodes-nodes-resources-configuring[manually determine and set the best resources] for your nodes. +To provide more reliable scheduling and minimize node resource overcommitment, reserve a portion of the CPU and memory resources for use by the underlying node components, such as `kubelet` and `kube-proxy`, and the remaining system components, such as `sshd` and `NetworkManager`. By specifying the resources to reserve, you provide the scheduler with more information about the remaining CPU and memory resources that a node has available for use by pods. By default, {product-title} automatically determines the optimal `system-reserved` CPU and memory resources for your nodes or you can xref:../../nodes/nodes/nodes-nodes-resources-configuring.adoc#nodes-nodes-resources-configuring-setting_nodes-nodes-resources-configuring[manually determine and set the best resources] for your nodes. [IMPORTANT] ====