`nvidia-device-plugin` failed to run on GPU nodes created by Node Auto-Provisioning

[hongchaodeng]

How to reproduce

• Create a GKE cluster in Standard mode
• Enable Node Auto-Provisioning with L4 GPU capacity
• Try to create a pod with nvidia.com/gpu resource request
• The pod will be stuck in PodInitializing state

Analysis

Using GPU nodes with Node-Auto-Provisioning in GKE will try to run nvidia-gpu-device-plugin job on GPU nodes. It's only after this job finishes successfully that the node will have allocatable nvidia.com/gpu resources. However, this job is stuck, and no pod with GPU requests can be scheduled onto it.

$ kubectl -n kube-system get pod

kube-system       nvidia-gpu-device-plugin-small-cos-4fpl4                         0/2     Init:0/2   0          16m

$ kubectl -n kube-system logs nvidia-gpu-device-plugin-small-cos-hzvds -c nvidia-driver-installer

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   598  100   598    0     0   273k      0 --:--:-- --:--:-- --:--:--  291k
GPU driver auto installation is disabled.
Waiting for GPU driver libraries to be available.

[MeCode4Food]

#402 might be related

[raushan2016]

Based on the doc here, while enabling Node-Auto-Provisioning the default GPU driver installation mode is disabled.
When creating the workload(pod/deployment) you can add the driver-version in the nodeSelector and the new nodel pool will be created with the right driver version.

spec:
  nodeSelector:
    cloud.google.com/gke-gpu-driver-version: "latest"

https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning#choose-driver-version