DRA GA, admin access, priority lists

_topic_maps/_topic_map.yml

@@ -2682,7 +2682,7 @@ Topics:
     Distros: openshift-enterprise,openshift-origin
   - Name: Placing pods on specific nodes using node selectors
     File: nodes-pods-node-selectors
-  - Name: Allocating GPUs to pods
+  - Name: Allocating GPUs to pods by using DRA
     File: nodes-pods-allocate-dra
     Distros: openshift-enterprise,openshift-origin
   - Name: Run Once Duration Override Operator

modules/nodes-pods-allocate-dra-about.adoc

@@ -4,15 +4,13 @@
 
 :_mod-docs-content-type: CONCEPT
 [id="nodes-pods-allocate-dra-about_{context}"]
-= About allocating GPUs to workloads
+= About GPU attributes
 
 // Taken from https://issues.redhat.com/browse/OCPSTRAT-1756
-{attribute-based-full} enables pods to request graphics processing units (GPU) based on specific device attributes. This ensures that each pod receives the exact GPU specifications it requires.
+[role="_abstract"]
+You can use {attribute-based-full} enables pods to request graphics processing units (GPU) based on specific device attributes. This ensures that each pod is scheduled on a node with the exact GPU specifications it requires. These attributes are advertised to the cluster by using a Dynamic Resource Allocation (DRA) driver, a third-party application that runs on each node in your cluster. 
 
-// Hiding until GA. The driver is not integrated in the TP version. 
-// With the NVIDIA Kubernetes DRA driver integrated into OpenShift,by the NVIDIA GPU Operator with a DRA driver
-
-Attribute-based resource allocation requires that you install a Dynamic Resource Allocation (DRA) driver. A DRA driver is a third-party application that runs on each node in your cluster to interface with the hardware of that node. 
+The (DRA) driver manages and exposes specialized resources within your cluster by interacting with the underlying hardware and advertising it to the {produt-title} control plane. You must install a DRA driver in your cluster. Installation of the DRA driver is beyond the scope of this documentation. Some DRA\ device drivers can also slice GPU memory, making it available to multiple workloads.
 
 The DRA driver advertises several GPU device attributes that {product-title} can use for precise GPU selection, including the following attributes:
 

modules/nodes-pods-allocate-dra-configure-about.adoc

@@ -4,10 +4,11 @@
 
 :_mod-docs-content-type: REFERENCE
 [id="nodes-pods-allocate-dra-configure-about_{context}"]
-= About GPU allocation objects
+= About GPU allocation objects and concepts
 
 // Taken from https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#terminology
-{attribute-based-full} uses the following objects to provide the core graphics processing unit (GPU) allocation functionality. All of these API kinds are included in the `resource.k8s.io/v1beta2` API group.
+[role="_abstract"]
+You can use the following {attribute-based-full} objects and concepts to ensure that a workload is scheduled on a node with the graphics processing unit (GPU) specifications it needs. All of these API kinds are included in the `resource.k8s.io/v1beta2` API group. You should be familiar with these objects before proceeding.
 
 Device class::
 A device class is a category of devices that pods can claim and how to select specific device attributes in claims. Some device drivers contain their own device class. Alternatively, an administrator can create device classes. A device class contains a device selector, which is a link:https://cel.dev/[common expression language (CEL)] expression that must evaluate to true if a device satisfies the request.
@@ -17,7 +18,7 @@ The following example `DeviceClass` object selects any device that is managed by
 .Example device class object
 [source,yaml]
 ----
-apiVersion: resource.k8s.io/v1beta1
+apiVersion: resource.k8s.io/v1
 kind: DeviceClass
 metadata:
   name: example-device-class
@@ -27,70 +28,174 @@ spec:
       expression: |-
         device.driver == "driver.example.com"
 ----
++
+--
+where:
+
+`spec.selectors`:: Can specify a CEL expression for selecting a device.
+--
 
 Resource slice::
-The Dynamic Resource Allocation (DRA) driver on each node creates and manages _resource slices_ in the cluster. A resource slice represents one or more GPU resources that are attached to nodes. When a resource claim is created and used in a pod, {product-title} uses the resource slices to find nodes that have access to the requested resources. After finding an eligible resource slice for the resource claim, the {product-title} scheduler updates the resource claim with the allocation details, allocates resources to the resource claim, and schedules the pod onto a node that can access the resources.
+The DRA driver on each node creates and manages _resource slices_ in the cluster. A resource slice represents one or more GPU resources that are attached to nodes. When a resource claim is created and used in a pod, {product-title} uses the resource slices to find nodes that have access to the requested resources. After finding an eligible resource slice for the resource claim, the {product-title} scheduler updates the resource claim with the allocation details, allocates resources to the resource claim, and schedules the pod onto a node that can access the resources.
+
+.Example resource claim template object
+[source,yaml]
+----
+apiVersion: v1
+items:
+- apiVersion: resource.k8s.io/v1
+  kind: ResourceSlice
+# ...
+spec:
+    driver: driver.example.com
+    nodeName: dra-example-driver
+    pool:
+      generation: 0
+      name: dra-example-driver
+      resourceSliceCount: 1
+    devices:
+    - attributes:
+        driverVersion:
+          version: 1.0.0
+        index:
+          int: 0
+        model:
+          string: LATEST-GPU-MODEL
+        uuid:
+          string: gpu-18db0e85-99e9-c746-8531-ffeb86328b39
+      capacity:
+        memory:
+          value: 10Gb
+      name: 2g-10gb
+# ...
+----
+where:
+
+`spec.driver`:: Specifies the name of the DRA driver, which you can specify in a device class. 
+
+`spec.devices.attributes`:: Specifies a device that you can allocated by using resource claim or resource claim template.
 
 Resource claim template::
 Cluster administrators and operators can create a _resource claim template_ to request a GPU from a specific device class. Resource claim templates provide pods with access to separate, similar resources. {product-title} uses a resource claim template to generate a resource claim for the pod. Each resource claim that {product-title} generates from the template is bound to a specific pod. When the pod terminates, {product-title} deletes the corresponding resource claim.
 +
-The following example resource claim template requests devices in the `example-device-class` device class.
+For the device request in a resource claim template, you must specify either a list of one or more device requests that the scheduler must meet, or a provide a prioritized list of requests for the scheduler to choose from. For more information, see _Prioritized list_ in this section.
++
+The following example resource claim template contains a device request with three sub-requests. Here, the `firstAvailable` parameter provides a prioritized list of requests. The scheduler checks the availability of each device in order and allocates the first available device when creating the pod. Each request uses a CEL expression to a device from the `example-device-class` device class. 
 +
 .Example resource claim template object
 [source,yaml]
 ----
-apiVersion: resource.k8s.io/v1beta1
+apiVersion: resource.k8s.io/v1
 kind: ResourceClaimTemplate
 metadata:
-  namespace: gpu-test1
-  name: gpu-claim-template
+  namespace: gpu-claim
+  name: gpu-devices
 spec:
-# ...
   spec:
     devices:
       requests:
-      - name: gpu
-        deviceClassName: example-device-class
+      - name: req-0
+        firstAvailable:
+        - name: 2g-10gb
+          deviceClassName: example-device-class
+          selectors:
+          - cel:
+              expression: "device.attributes['driver.example.com'].profile == '2g.10gb'"
+        - name: 3g-20gb
+          deviceClassName: example-device-class
+          selectors:
+          - cel:
+              expression: "device.attributes['driver.example.com'].profile == '3g.20gb'"
 ----
++
+--
+where:
+
+`spec.spec.devices.requests`:: Configures a resource request.
+
+`spec.spec.devices.requests.firstAvailable:`:: Specifies whether the list of requested devices is a prioritized list. In this example, `firstAvailable` specifies two sub-requests, named: `2g-10gb` and `3g-20gb` in a prioritized list. Alternatively, you can specify `exactly` to request one specific device. 
+
+`spec.spec.devices.requests.deviceClassName`:: Specifies which device class to use with this request.
+
+`spec.spec.devices.requests.selectors`:: Specifies uses CEL expressions to request specific devices from the specified device class.
+--
 
 Resource claim::
 Admins and operators can create a _resource claim_ to request a GPU from a specific device class. A resource claim differs from a resource claim template by allowing you to share GPUs with multiple pods. Also, resource claims are not deleted when a requesting pod is terminated. 
 +
-The following example resource claim template uses CEL expressions to request specific devices in the `example-device-class` device class that are of a specific size.
+For the device request in a resource claim, you must specify either a list of one or more device requests that the scheduler must meet, or a provide a prioritized list of requests for the scheduler to choose from. For more information, see _Prioritized list_ in this section.
++
+The following example resource claim uses a CEL expression to request one device in the `example-device-class` device class. Here, the `exactly` parameter indicates that the specific requested device must be available before the scheduler can allocate the device when creating the pod.
 +
 .Example resource claim object
 [source,yaml]
 ----
-apiVersion: resource.k8s.io/v1beta1
+apiVersion: resource.k8s.io/v1
 kind: ResourceClaim
 metadata:
   namespace: gpu-claim
   name: gpu-devices
 spec:
   devices:
     requests:
-    - name: 1g-5gb
-      deviceClassName: example-device-class
-      selectors:
-      - cel:
-          expression: "device.attributes['driver.example.com'].profile == '1g.5gb'"
-    - name: 1g-5gb-2
-      deviceClassName: example-device-class
-      selectors:
-      - cel:
-          expression: "device.attributes['driver.example.com'].profile == '1g.5gb'"
-    - name: 2g-10gb
-      deviceClassName: example-device-class
-      selectors:
-      - cel:
-          expression: "device.attributes['driver.example.com'].profile == '2g.10gb'"
-    - name: 3g-20gb
-      deviceClassName: example-device-class
-      selectors:
-      - cel:
-          expression: "device.attributes['driver.example.com'].profile == '3g.20gb'"
+    - name: req-0
+      exactly:
+      - name: 2g-10gb
+        deviceClassName: example-device-class
+        selectors:
+        - cel:
+            expression: "device.attributes['driver.example.com'].profile == '2g.10gb'"
+----
+
+Prioritized list::
+In the `ResourceClaimTemplate` or `ResourceClaim` object, you must specify either `exactly` to request a specific device or `firstAvailable` to create a prioritized list of devices, in case the primary choice is not available. 
++
+* `spec.devices.requests.firstAvailable` specifies multiple requests for a device, of which only one device needs to be available before the scheduler can create the requesting pod. The scheduler checks the availability of the devices in the order listed and selects the first available device. The scheduler can create the pod if one requested devices is available.
+
+* `spec.devices.requests.exactly` specifies one or more requests for a device, for which all of the devices must be available before the scheduler can create the requesting pod. Each of the devices must the request exactly for the request to be satisfied. If any of the requested devices is not available, the scheduler cannot create the pod.
+
+Admin access::
+A cluster administrator can gain privileged access to a device that is in use by other users. This enables administrators to perform tasks such as monitoring the health and status of devices while ensuring that users can continue to use these devices with their workloads.
++
+To gain admin access, an administrator must create a resource claim or resource claim template with the `adminAccess: true` parameter in a namespace that includes the `resource.kubernetes.io/admin-access: "true"` label. Non-administrator users cannot access namespaces with this label. 
++
+.Example namespace with admin access label
+[source,yaml]
+----
+apiVersion: v1
+kind: Namespace
+metadata:
+  labels:
+    resource.kubernetes.io/admin-access: "true"
+# ...
 ----
++
+In the following example, the administrator is granted access to the `2g-10gb` device:
++
+.Example resource claim object with admin access
+[source,yaml]
+----
+apiVersion: resource.k8s.io/v1
+kind: ResourceClaimTemplate
+metadata:
+  name: large-black-cat-claim-template
+spec:
+  devices:
+    requests:
+    - name: req-0
+      exactly:
+        allocationMode: All
+        adminAccess: true
+        deviceClassName: example-device-class
+        selectors:
+        - cel:
+            expression: "device.attributes['driver.example.com'].profile == '2g.10gb'"
+----
++
+--
+where:
 
-For more information on configuring resource claims, resource claim templates, see link:https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/["Dynamic Resource Allocation"] (Kubernetes documentation).
+`spec.devices.requests.exactly.adminAccess:true`:: Specifies that the admin access mode is enabled for the specified device.
+--
 
 For information on adding resource claims to pods, see "Adding resource claims to pods".

modules/nodes-pods-allocate-dra-configure.adoc

@@ -6,7 +6,8 @@
 [id="nodes-pods-allocate-dra-configure_{context}"]
 = Adding resource claims to pods
 
-{attribute-based-full} uses resource claims and resource claim templates to allow you to request specific graphics processing units (GPU) for the containers in your pods. Resource claims can be used with multiple containers, but resource claim templates can be used with only one container. For more information, see "About configuring device allocation by using device attributes" in the _Additional Resources_ section.
+[role="_abstract"]
+You can use resource claims and resource claim templates with {attribute-based-full} to allow you to request your workloads to be shecduled on nodes with specific graphics processing units (GPU) specifications. Resource claims can be used with multiple containers, but resource claim templates can be used with only one container. For more information, see "About configuring device allocation by using device attributes" in the _Additional Resources_ section.
 
 The example in the following procedure creates a resource claim template to assign a specific GPU to `container0` and a resource claim to share a GPU between `container1` and `container2`.
 
@@ -16,24 +17,6 @@ The example in the following procedure creates a resource claim template to assi
 //Remove for TP * The Nvidia GPU Operator is installed. For more information see "Adding Operators to a cluster" in the _Additional Resources_ section.
 * A resource slice has been created.
 * A resource claim and/or resource claim template has been created.
-* You enabled the required Technology Preview features for your cluster by editing the `FeatureGate` CR named `cluster`:
-+
-.Example `FeatureGate` CR
-[source,yaml]
-----
-apiVersion: config.openshift.io/v1
-kind: FeatureGate
-metadata:
-  name: cluster
-spec:
-  featureSet: TechPreviewNoUpgrade <1>
-----
-<1> Enables the required features.
-+
-[WARNING]
-====
-Enabling the `TechPreviewNoUpgrade` feature set on your cluster cannot be undone and prevents minor version updates. This feature set allows you to enable these Technology Preview features on test clusters, where you can fully test them. Do not enable this feature set on production clusters.
-====
 
 .Procedure
 
@@ -73,7 +56,7 @@ spec:
   resourceClaims: <2>
   - name: gpu-claim-template
     resourceClaimTemplateName: example-resource-claim-template
-  - name: gpu-claim
+  - name: gpu-devices
     resourceClaimName: example-resource-claim
 ----
 <1> Specifies one or more resource claims to use with this container.

nodes/pods/nodes-pods-allocate-dra.adoc

@@ -1,20 +1,18 @@
 :_mod-docs-content-type: ASSEMBLY
 :context: nodes-pods-allocate-dra
 [id="nodes-pods-allocate-dra"]
-= Allocating GPUs to pods
+= Allocating GPUs to pods by using DRA
 include::_attributes/common-attributes.adoc[]
 
 toc::[]
 
 // Taken from https://issues.redhat.com/browse/OCPSTRAT-1756
 // Naming taken from https://issues.redhat.com/browse/OCPSTRAT-2384. Is this correct?
-{attribute-based-full} enables fine-tuned control over graphics processing unit (GPU) resource allocation in {product-title}, allowing pods to request GPUs based on specific device attributes, including product name, GPU memory capacity, compute capability, vendor name and driver version. These attributes are exposed by a third-party Dynamic Resource Allocation (DRA) driver.
 
-// Hiding until GA. The driver is not integrated in the TP version
-// This attribute-based resource allocation is achieved through the integration of the NVIDIA Kubernetes DRA driver into OpenShift. 
+[role="_abstract"]
+You can use {attribute-based-full} to enable fine-tuned control over graphics processing unit (GPU) resource allocation in {product-title}, allowing pods to request GPUs based on specific device attributes, including product name, GPU memory capacity, compute capability, vendor name and driver version. Having access to these attributes, which are exposed by a third-party Dynamic Resource Allocation (DRA) drive, allows {product-title} to schedule a workload on a node that has the specific devices it needs. 
 
-:FeatureName: {attribute-based-full}
-include::snippets/technology-preview.adoc[]
+This workflow provides significant improvement in the device allocation workflow when compared to device plugins, which require per-container device requests, do not support device sharing, and do not support expression-based device filtering.
 
 // The following include statements pull in the module files that comprise
 // the assembly. Include any combination of concept, procedure, or reference
@@ -32,6 +30,5 @@ include::modules/nodes-pods-allocate-dra-configure.adoc[leveloffset=+1]
 
 [role="_additional-resources"]
 .Additional resources
-// Hiding until GA link:https://catalog.ngc.nvidia.com/orgs/nvidia/helm-charts/nvidia-dra-driver-gpu?version=25.3.2[NVIDIA DRA Driver for GPUs]
-// Hiding until GA * xref:../../operators/admin/olm-adding-operators-to-cluster.adoc#olm-adding-operators-to-a-cluster[Adding Operators to a cluster]
+
 * xref:../../nodes/pods/nodes-pods-allocate-dra.adoc#nodes-pods-allocate-dra-configure-about_nodes-pods-allocate-dra[About configuring device allocation by using device attributes]