Running an AI model locally involves more than installing Python and downloading a model.

Before deploying an LLM server, a RAG application or an AI agent, we must confirm that the infrastructure underneath it works correctly.

In this first stage, we will verify that:

• Linux detects the physical GPU;
• the NVIDIA driver is working;
• the NVIDIA container runtime is installed;
• K3s knows how to start GPU containers;
• Kubernetes sees the available GPU;
• the NVIDIA device plugin is healthy;
• a test container can access the GPU.

This is Stage 0 of our local AI learning project.

We will not deploy an AI model yet. The objective is to prove that the platform is ready.

The lab environment

The example lab contains four K3s nodes:

control-plane-01
control-plane-02
gpu-worker-01
gpu-worker-02

Only the two worker nodes contain NVIDIA GPUs.

All real hostnames, internal domains, IP addresses and organization-specific values have been replaced with generic examples.

The same procedure can also be used with:

• one GPU worker;
• more than two GPU workers;
• a small home lab;
• an on-premises Kubernetes cluster;
• virtual machines with GPU passthrough.

---

What happens when Kubernetes runs a GPU container?

A GPU application passes through several software layers:

Kubernetes scheduler
        ↓
NVIDIA device plugin
        ↓
Kubernetes RuntimeClass
        ↓
NVIDIA container runtime
        ↓
Linux NVIDIA driver
        ↓
Physical NVIDIA GPU

Each component has a different role.

The Linux driver communicates with the physical graphics card.

The NVIDIA container runtime makes the GPU and the required driver libraries available inside a container.

The NVIDIA device plugin tells Kubernetes how many GPUs exist and whether they are healthy.

The Kubernetes scheduler decides which node can run a pod that requests a GPU.

Kubernetes supports GPU scheduling through vendor device plugins. After the NVIDIA plugin registers successfully, the GPU appears as a schedulable resource named nvidia.com/gpu.

---

Prerequisites

This guide assumes that:

• K3s is already installed;
• kubectl can connect to the cluster;
• NVIDIA drivers are installed on the GPU nodes;
• NVIDIA Container Toolkit is installed;
• the NVIDIA device plugin has already been deployed.

The purpose of this article is verification and troubleshooting, not the initial installation of those components.

The NVIDIA device plugin requires working NVIDIA drivers and a configured NVIDIA container runtime. The plugin is normally deployed as a DaemonSet and exposes GPUs to Kubernetes workloads.

---

A note about Windows and Linux commands

In this lab, Kubernetes commands are executed from Windows PowerShell, while hardware checks are executed directly on the Linux GPU nodes.

The command-line continuation character differs between shells.

PowerShell uses a backtick:

kubectl get pods `
  -n kube-system

Linux Bash uses a backslash:

kubectl get pods \
  -n kube-system

Windows Command Prompt uses a caret:

kubectl get pods ^
  -n kube-system

Using the wrong continuation character may cause confusing errors.

For example, this command is valid in Linux Bash but not in Windows Command Prompt:

kubectl get nodes \

Windows may interpret \ as the name of a Kubernetes resource.

For beginners, running short commands on a single line is usually safest.

---

Step 1: Check the Kubernetes nodes

Run this command from the computer where kubectl is configured:

kubectl get nodes -o wide

Example output:

NAME               STATUS   ROLES                 INTERNAL-IP
gpu-worker-01      Ready    <none>                192.168.10.21
gpu-worker-02      Ready    <none>                192.168.10.22
control-plane-01   Ready    control-plane,etcd    192.168.10.23
control-plane-02   Ready    control-plane,etcd    192.168.10.24

Check the STATUS column.

Every node required by the lab should display:

Ready

This command also shows:

• the K3s version;
• the node operating system;
• the Linux kernel;
• the container runtime;
• the internal node IP address.

At this stage, we are only confirming that the cluster is operational.

---

Step 2: Check the NVIDIA RuntimeClass

Run:

kubectl get runtimeclass

Look for:

NAME      HANDLER
nvidia    nvidia

A RuntimeClass tells Kubernetes which container runtime handler should start a pod.

A GPU pod can request the NVIDIA runtime with:

runtimeClassName: nvidia

When the NVIDIA runtime is not configured as the default runtime, NVIDIA documents using a RuntimeClass with the nvidia handler.

If the nvidia RuntimeClass does not exist, stop here and inspect the K3s container runtime configuration before deploying GPU applications.

---

Step 3: Verify the physical GPU

Connect directly to the first Linux GPU node and run:

nvidia-smi

Repeat the command on every GPU node.

A healthy result should show:

• the NVIDIA GPU model;
• the installed driver version;
• the supported CUDA version;
• GPU temperature;
• total GPU memory;
• current GPU utilization;
• processes using GPU memory.

Example:

NVIDIA-SMI
Driver Version: 5xx.xx
CUDA Version: 12.x or 13.x

GPU Name: NVIDIA GeForce RTX
Memory Usage: 0 MiB / 16 GB
GPU Utilization: 0%

The exact driver and CUDA versions depend on the hardware and operating system.

The CUDA Version displayed by nvidia-smi represents the maximum CUDA level supported by the installed driver. It does not necessarily match the CUDA libraries inside a container.

If nvidia-smi fails on the Linux host, Kubernetes will not be able to repair the problem. Fix the host driver before continuing.

---

Step 4: Verify the NVIDIA container runtime

On every GPU worker, run:

which nvidia-container-runtime

Expected output:

/usr/bin/nvidia-container-runtime

Display its version:

nvidia-container-runtime --version

The NVIDIA container runtime allows a container to access:

/dev/nvidia0
/dev/nvidiactl
/dev/nvidia-uvm

It also injects the NVIDIA driver libraries required by CUDA applications.

Without this runtime, a pod could be scheduled on a GPU node but still fail to use the GPU.

---

Step 5: Check the K3s containerd configuration

K3s uses containerd to start containers.

On each GPU worker, search the generated containerd configuration:

sudo grep -R "nvidia" \
  /var/lib/rancher/k3s/agent/etc/containerd/ \
  2>/dev/null

A working configuration should contain an NVIDIA runtime section similar to:

runtimes.'nvidia'
BinaryName = "/usr/bin/nvidia-container-runtime"

This connects:

runtimeClassName: nvidia

from the pod definition to:

/usr/bin/nvidia-container-runtime

on the Linux host.

---

Step 6: Check GPU capacity in Kubernetes

Run:

kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU-CAPACITY:.status.capacity.nvidia\.com/gpu,GPU-ALLOCATABLE:.status.allocatable.nvidia\.com/gpu"

Example output:

NAME               GPU-CAPACITY   GPU-ALLOCATABLE
gpu-worker-01      1              1
gpu-worker-02      1              1
control-plane-01   <none>         <none>
control-plane-02   <none>         <none>

What does capacity mean?

GPU-CAPACITY: 1

means the node reports one physical GPU resource.

What does allocatable mean?

GPU-ALLOCATABLE: 1

means Kubernetes is allowed to assign one GPU to workloads on that node.

Allocatable does not necessarily mean that the GPU is currently unused.

To see whether a pod already reserves the GPU, run:

kubectl describe node gpu-worker-01

Find the Allocated resources section.

A free GPU may show:

nvidia.com/gpu    0    0

An allocated GPU may show:

nvidia.com/gpu    1    1

Kubernetes exposes NVIDIA GPUs as an extended resource called nvidia.com/gpu. GPU resources are requested as whole integers and cannot normally be overcommitted like CPU.

---

Step 7: Check the NVIDIA device plugin

List all DaemonSets:

kubectl get daemonsets -A

Look for a resource similar to:

kube-system   nvidia-device-plugin-daemonset

Next, list the NVIDIA pods:

kubectl get pods -A -o wide |
  Select-String -Pattern "nvidia"

On each GPU node, the device-plugin pod should be:

Running

The NVIDIA device plugin is responsible for:

• discovering GPUs;
• reporting GPU health;
• exposing the number of GPUs;
• allowing GPU-enabled containers to run.

These are the primary functions described by NVIDIA for its Kubernetes device plugin.

Read the plugin logs

First identify the pod name:

kubectl get pods -n kube-system -o wide |
  Select-String -Pattern "nvidia"

Then run:

kubectl logs \
  -n kube-system \
  NVIDIA_DEVICE_PLUGIN_POD \
  --tail=100

In PowerShell, use:

kubectl logs `
  -n kube-system `
  NVIDIA_DEVICE_PLUGIN_POD `
  --tail=100

Healthy logs typically include messages similar to:

Starting NVIDIA Device Plugin
Starting GRPC server for nvidia.com/gpu
Registered device plugin for nvidia.com/gpu with Kubelet

The important phrase is:

Registered device plugin for nvidia.com/gpu with Kubelet

After registration, kubelet publishes the GPU resource in the Kubernetes node status.

---

Step 8: Create a GPU smoke-test pod

A smoke test is a small test used to confirm that the most important components work together.

Create a file called:

gpu-smoke-test.yaml

Use the following anonymized manifest:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-smoke-test
spec:
  restartPolicy: Never
  runtimeClassName: nvidia

  nodeSelector:
    kubernetes.io/hostname: gpu-worker-02

  containers:
    - name: cuda
      image: nvidia/cuda:12.8.1-base-ubuntu24.04
      imagePullPolicy: IfNotPresent

      command:
        - /bin/bash
        - -lc
        - |
          set -e

          echo "=== POD NAME ==="
          hostname

          echo
          echo "=== NVIDIA SMI ==="
          nvidia-smi

          echo
          echo "=== NVIDIA DEVICES ==="
          ls -la /dev/nvidia* || true

          echo
          echo "=== CUDA ENVIRONMENT ==="
          env | grep -E 'NVIDIA|CUDA' | sort || true

      resources:
        limits:
          nvidia.com/gpu: 1

Replace:

gpu-worker-02

with the hostname of a free GPU node.

Kubernetes allows a GPU to be specified only in limits, because it automatically uses the same value as the request. When both requests and limits are specified, their GPU values must be equal.

Understanding the important fields

This field selects the NVIDIA runtime:

runtimeClassName: nvidia

This field forces the test onto a particular node:

nodeSelector:
  kubernetes.io/hostname: gpu-worker-02

This field reserves one GPU:

resources:
  limits:
    nvidia.com/gpu: 1

The restartPolicy prevents Kubernetes from repeatedly restarting a completed diagnostic container:

restartPolicy: Never

---

Step 9: Run the smoke test

Apply the file:

kubectl apply -f .\gpu-smoke-test.yaml

Check the selected node:

kubectl get pod gpu-smoke-test -o wide

Watch the pod:

kubectl get pod gpu-smoke-test -w

The -w option means watch.

This command does not stop automatically. Press:

Ctrl+C

when the test reaches its final state.

For this one-time pod, the expected status is:

Completed

A completed test may show:

READY   STATUS
0/1     Completed

This is normal. The container has finished its work and is no longer running.

---

Step 10: Read the test results

Display the logs:

kubectl logs gpu-smoke-test

A successful test should show:

=== NVIDIA SMI ===
NVIDIA GeForce RTX
Driver Version: ...
CUDA Version: ...

=== NVIDIA DEVICES ===
/dev/nvidia0
/dev/nvidiactl
/dev/nvidia-uvm

This proves that:

1. Kubernetes scheduled the pod;
2. the NVIDIA device plugin allocated the GPU;
3. K3s used the NVIDIA runtime;
4. the runtime injected the NVIDIA devices;
5. the container communicated with the host driver;
6. CUDA software can now use the GPU.

At this point, the cluster is ready for the first local AI model server.

---

Common problems

Error: nodes “\” not found

Example:

Error from server (NotFound): nodes "\" not found

Cause:

A Linux backslash was used for line continuation inside Windows Command Prompt.

Solution:

Run the command on one line or use the correct continuation character for the current shell.

---

Error: connection to 127.0.0.1:6443 refused

Example:

The connection to the server 127.0.0.1:6443 was refused

Cause:

kubectl was executed on a worker node without a valid kubeconfig.

Solution:

Run kubectl from a configured administration workstation or from a K3s control-plane node.

Disabling YAML validation does not fix a missing Kubernetes API connection.

---

Pod remains Pending

Describe the pod:

kubectl describe pod gpu-smoke-test

Look at the Events section.

A common message is:

Insufficient nvidia.com/gpu

Possible causes include:

• another pod already uses the GPU;
• the GPU device plugin is not running;
• the pod was forced onto a node without a GPU;
• the node does not advertise nvidia.com/gpu.

---

Pod reports ImagePullBackOff

This means Kubernetes could not download the container image.

Possible causes include:

• no internet access from the worker;
• DNS failure;
• registry rate limiting;
• an invalid image tag;
• insufficient disk space.

Check the exact reason with:

kubectl describe pod gpu-smoke-test

---

Pod reaches CreateContainerError

This usually points to:

• NVIDIA runtime configuration;
• missing driver libraries;
• invalid RuntimeClass;
• device injection problems;
• container runtime errors.

Check:

nvidia-smi
which nvidia-container-runtime

and:

kubectl get runtimeclass
kubectl describe pod gpu-smoke-test

---

Clean up the smoke test

Delete the test pod:

kubectl delete pod gpu-smoke-test

Verify that the GPU is no longer allocated:

kubectl describe node gpu-worker-02 |
  Select-String -Pattern "nvidia.com/gpu"

The Allocated resources section should return to:

nvidia.com/gpu    0    0

---

Stage 0 completion checklist

Stage 0 is complete when all of the following are true:

[OK] All required K3s nodes are Ready
[OK] Linux detects the NVIDIA GPU
[OK] nvidia-smi works on every GPU worker
[OK] NVIDIA container runtime is installed
[OK] K3s has an NVIDIA RuntimeClass
[OK] NVIDIA device-plugin pods are running
[OK] Each GPU node advertises nvidia.com/gpu
[OK] GPU capacity and allocatable values are correct
[OK] Kubernetes schedules the smoke-test pod
[OK] nvidia-smi works inside the container
[OK] /dev/nvidia* devices are visible in the container

---

What comes next?

Now that the platform can run GPU containers, the next stage is to deploy an actual language model.

In the next article, we will:

• create a dedicated Kubernetes namespace;
• create persistent storage for model files;
• deploy vLLM;
• download a small instruction model;
• expose an OpenAI-compatible API;
• send the first prompt from PowerShell and Python.

The most important lesson from Stage 0 is simple:

Do not troubleshoot an AI model until you have proved that the GPU infrastructure underneath it works.

---

Frequently asked questions

Does Kubernetes include NVIDIA drivers?

No. NVIDIA drivers must be installed on the GPU host. Kubernetes uses the NVIDIA device plugin to discover and allocate the hardware.

What is nvidia.com/gpu?

It is the extended Kubernetes resource name registered by the NVIDIA device plugin.

A pod requests one GPU with:

resources:
  limits:
    nvidia.com/gpu: 1

What is a RuntimeClass?

A RuntimeClass tells Kubernetes which container runtime handler should start a pod.

For NVIDIA workloads, it may look like:

runtimeClassName: nvidia

Why does the test pod show 0/1 Completed?

The test container completed its command and exited successfully. It is no longer running, so it is not marked Ready.

Can two pods share the same GPU?

Not by default with a basic integer GPU request. Advanced configurations such as time-slicing or NVIDIA MPS can provide sharing, but they should be introduced only after the basic setup works.

Why run a smoke test before installing vLLM?

A smoke test isolates the infrastructure. If it works, later problems are more likely to be related to the model server, storage or application configuration rather than the GPU driver or Kubernetes device allocation.

---

Further reading

Kubernetes explains GPU scheduling and the nvidia.com/gpu extended resource in its official GPU scheduling documentation.

The Kubernetes device-plugin documentation describes how specialized hardware is registered with kubelet and advertised through node status.

NVIDIA maintains the official Kubernetes device plugin and documents RuntimeClass configuration and GPU pod requests.