A Linux server equipped with an NVIDIA GPU may look perfectly healthy at first.

The operating system boots. The GPU appears in nvidia-smi. CUDA detects the device. A small GPU test succeeds.

Then a real artificial intelligence workload starts—and the entire server freezes or restarts.

At that point, it is tempting to blame the GPU immediately. However, GPU workloads depend on several tightly connected software layers:

• the motherboard firmware;
• the Linux kernel;
• the NVIDIA kernel driver;
• NVIDIA user-space libraries;
• the CUDA runtime;
• the container runtime;
• the machine-learning framework;
• the application itself.

A problem in any one of these layers may appear as a “GPU problem.”

This article explains how the NVIDIA driver interacts with the Linux kernel, how to compare two GPU servers, and how to troubleshoot compatibility problems without changing multiple variables at the same time.

What is the Linux kernel?

The Linux kernel is the central part of the operating system.

A simplified way to think about it is:

The kernel is the traffic controller between software and hardware.

It manages:

• processors;
• memory;
• disks;
• network cards;
• PCI Express devices;
• interrupts;
• power management;
• device drivers.

Applications normally do not communicate directly with the physical GPU. They ask the operating system and the NVIDIA software stack to perform that communication.

When the Linux kernel changes, some of the internal interfaces used by hardware drivers may also change.

That is why an NVIDIA driver that works correctly with one kernel may fail to build, load or behave correctly with another kernel.

What is the NVIDIA driver?

The NVIDIA driver is the software that allows Linux and applications to communicate with the graphics card.

On Linux, the NVIDIA driver contains several components.

Kernel modules

Typical NVIDIA kernel modules include:

nvidia
nvidia_uvm
nvidia_modeset
nvidia_drm

These modules run inside the Linux kernel.

They handle activities such as:

• GPU memory management;
• CUDA process access;
• hardware interrupts;
• PCI Express communication;
• display management;
• interaction with the GPU firmware.

User-space libraries

The driver also installs libraries used outside the kernel.

Examples include:

libcuda.so
libnvidia-ml.so
libnvidia-ptxjitcompiler.so

Applications, containers and CUDA frameworks use these libraries to access the driver.

For the complete stack to work correctly, the kernel modules and user-space libraries must belong to compatible driver versions.

Why the kernel and NVIDIA driver must match

The NVIDIA kernel module is not a normal application.

It is loaded directly into the running Linux kernel. Therefore, it must be compatible with:

• the exact kernel version;
• the kernel architecture;
• the compiler environment;
• the kernel configuration;
• Secure Boot settings;
• the installed kernel headers.

Suppose a server has these two kernels installed:

5.14.0-enterprise
6.12.0-vendor

The NVIDIA module normally needs to exist for each kernel separately:

/lib/modules/5.14.0-enterprise/extra/nvidia.ko
/lib/modules/6.12.0-vendor/extra/nvidia.ko

Having the NVIDIA driver installed does not automatically prove that the correct module exists for every installed kernel.

NVIDIA driver installation methods

There are several ways to install NVIDIA drivers on Linux.

Understanding which method was used is important during troubleshooting.

Precompiled kernel module

Some Linux repositories provide an NVIDIA module already compiled for a specific kernel.

Advantages:

• predictable installation;
• package-manager integration;
• no local compilation during updates.

Disadvantages:

• the module may not exist for a newly installed kernel;
• kernel and driver repositories must remain synchronized.

DKMS

DKMS stands for Dynamic Kernel Module Support.

DKMS rebuilds the NVIDIA module whenever a new kernel is installed.

You can check its state with:

dkms status

A healthy result may resemble:

nvidia/580.x, 5.14.x, x86_64: installed
nvidia/580.x, 6.12.x, x86_64: installed

This means DKMS built the NVIDIA module for both kernels.

It does not guarantee runtime stability, but it confirms that a module was generated for each kernel.

NVIDIA .run installer

NVIDIA also provides a standalone installer.

Although it can work, it may create maintenance problems when combined with distribution packages. Kernel upgrades can also require the driver module to be rebuilt manually.

For servers, package-managed installations are generally easier to audit and maintain.

Open and proprietary NVIDIA kernel modules

Modern NVIDIA driver packages may use either:

• NVIDIA’s open kernel modules;
• NVIDIA’s traditional proprietary kernel modules.

You can inspect the loaded module with:

modinfo nvidia

Useful fields include:

filename:
version:
license:

For example:

filename: /lib/modules/.../nvidia.ko
version: 580.x
license: Dual MIT/GPL

The license line can help identify an open kernel module build.

When comparing two servers, do not compare only the visible driver number. Also compare whether the systems use the same kernel-module type and installation method.

The most common misunderstanding about nvidia-smi

Running:

nvidia-smi

may show:

Driver Version: 580.x
CUDA Version: 13.0

The displayed CUDA version does not necessarily mean CUDA Toolkit 13.0 is installed on the host.

It generally indicates the newest CUDA application compatibility level supported by the installed driver.

To inspect the CUDA Toolkit compiler, use:

nvcc --version

But containers may contain their own CUDA runtime and may not include nvcc at all.

Therefore, these are different things:

• NVIDIA driver version;
• maximum CUDA compatibility reported by the driver;
• installed CUDA Toolkit version;
• CUDA runtime inside a container;
• CUDA version used to build a machine-learning framework.

They must not be treated as one version number.

A practical anonymized case study

Consider two Linux worker nodes with similar NVIDIA GPUs.

The first node was stable and used:

• an enterprise Linux 5.14 kernel;
• a newer NVIDIA driver branch;
• a working NVIDIA container runtime.

The second node experienced hard reboots during the startup of a large-language-model inference server. Initially it used:

• a vendor Linux 6.12 kernel;
• NVIDIA driver 580.x;
• NVIDIA open kernel modules.

The affected system had NVIDIA modules installed for both the 5.14 and 6.12 kernels.

The diagnostic team booted the affected node into the same 5.14 kernel family used by the stable reference node. Standalone CUDA tests passed, but the original AI workload still caused a reboot.

This was an important result.

It demonstrated that:

The newer 6.12 kernel was not, by itself, the root cause.

Changing only the kernel did not eliminate the problem. The remaining differences included the NVIDIA driver branch, firmware, motherboard, PCI Express configuration and other hardware or software components.

This is why proper troubleshooting requires changing one variable at a time.

How to compare two NVIDIA GPU servers

The best comparison uses one stable system as a reference.

Do not begin by comparing hundreds of package names. Start with the layers most closely related to the GPU.

1. Compare the operating system and kernel

Run on both servers:

cat /etc/os-release

uname -a
uname -r

Record:

• Linux distribution;
• distribution release;
• exact kernel version;
• kernel family;
• architecture.

A node running kernel 5.14 and another running kernel 6.12 are not equivalent environments, even when both use the same Linux distribution.

2. Compare NVIDIA driver information

Run:

nvidia-smi

Record:

• driver version;
• reported CUDA compatibility;
• GPU model;
• PCI bus address;
• memory size;
• power limit;
• persistence mode.

Then inspect the module itself:

modinfo nvidia |
grep -E '^(filename|version|license):'

The modinfo output is often more useful than nvidia-smi because it confirms the exact kernel-module file currently available.

3. Compare loaded NVIDIA modules

Run:

lsmod |
grep '^nvidia'

A typical result contains:

nvidia
nvidia_uvm
nvidia_modeset
nvidia_drm

For a headless CUDA server, not every display-related module is always essential. However, major differences between two otherwise similar nodes should be documented.

4. Compare DKMS status

Run:

dkms status

Check whether the driver is installed for the currently running kernel:

uname -r

If the kernel is:

6.12.0-example

but DKMS only shows:

nvidia/580.x, 5.14.0-example: installed

the NVIDIA module was not built for the active kernel.

The driver may fail to load entirely after reboot.

5. Compare installed packages

On RPM-based distributions:

rpm -qa |
grep -Ei 'nvidia|cuda|kmod' |
sort

On Debian-based distributions:

dpkg -l |
grep -Ei 'nvidia|cuda' |
sort

Look for:

• different driver branches;
• duplicated packages from multiple repositories;
• old libraries left behind;
• mixed installation methods;
• missing container-toolkit packages.

6. Compare enabled repositories

For RPM-based distributions:

dnf repolist --enabled

One server might receive NVIDIA packages from the distribution repository, while another receives them from NVIDIA’s CUDA repository or a community repository.

This can produce apparently similar but internally different driver stacks.

7. Compare PCI Express status

Run:

lspci -nnk |
grep -A3 -i nvidia

Then inspect the GPU directly:

lspci -vv -s <GPU-BUS-ID>

Look for:

LnkCap:
LnkSta:

For example:

LnkCap: Speed 32GT/s, Width x16
LnkSta: Speed 2.5GT/s, Width x16

A low idle link speed can be normal because PCIe power management reduces speed when the GPU is idle.

A reduced width is more concerning:

Width x4

when the GPU and slot should support:

Width x16

Possible causes include:

• using the wrong motherboard slot;
• lane sharing with another device;
• BIOS configuration;
• a badly seated GPU;
• slot or motherboard limitations;
• CPU lane limitations.

A safe troubleshooting workflow

Step 1: Stop automatic workload restarts

Before investigating a crashing GPU node, prevent Kubernetes or another scheduler from immediately restarting the workload.

In Kubernetes, that may mean:

kubectl cordon <node>
kubectl scale deployment <deployment> --replicas=0

This avoids repeated reboots and possible storage corruption.

Step 2: Record the baseline

Before changing anything, collect:

uname -r
nvidia-smi
modinfo nvidia
dkms status
lsmod | grep '^nvidia'

Also record a unique boot identifier:

cat /proc/sys/kernel/random/boot_id

If the server restarts, the boot ID changes.

Step 3: Confirm the module belongs to the active kernel

Run:

modinfo -n nvidia

The returned path should include the active kernel version:

/lib/modules/$(uname -r)/...

You can test this directly:

MODULE_PATH="$(modinfo -n nvidia)"

echo "Kernel: $(uname -r)"
echo "Module: $MODULE_PATH"

Step 4: Run a minimal CUDA test

Before starting a large AI model, run a small GPU operation.

A basic PyTorch test might be:

python3 - <<'PY'
import torch

print("CUDA available:", torch.cuda.is_available())

if not torch.cuda.is_available():
    raise SystemExit("CUDA is unavailable")

print("GPU:", torch.cuda.get_device_name(0))
print("PyTorch CUDA:", torch.version.cuda)

a = torch.randn((2048, 2048), device="cuda")
b = torch.randn((2048, 2048), device="cuda")
c = a @ b

torch.cuda.synchronize()

print("Test completed:", c.shape)
PY

If this fails, there is little value in testing a full inference server.

Step 5: Run sustained compute separately

A small test only proves that the GPU can execute one operation.

It does not test:

• sustained power;
• high temperature;
• large memory allocations;
• repeated synchronization;
• long-running PCIe activity.

Therefore, run separate tests for:

1. sustained GPU computation;
2. high VRAM allocation;
3. combined compute and memory activity.

If these tests pass but the application still crashes, the problem is likely more specific than a basic GPU hardware failure.

Step 6: Test the real workload

Use the same:

• model;
• container image;
• storage;
• memory limits;
• application arguments;
• GPU power limit;
• Kubernetes configuration.

Changing several settings at once makes the result difficult to interpret.

Step 7: Change one layer only

A useful comparison sequence is:

1. Kernel A + Driver A
2. Kernel B + Driver A
3. Kernel B + Driver B
4. Kernel A + Driver B

In practice, not every combination will be supported. But the principle remains important:

Change either the kernel or the driver—not both at the same time.

When the kernel and driver are upgraded together and the problem disappears, you do not know which change solved it.

Commands for collecting evidence after a crash

Previous boot kernel logs

journalctl -k -b -1 --no-pager

Search for NVIDIA and PCIe errors:

journalctl -k -b -1 --no-pager |
grep -Ei \
'NVRM|Xid|SXid|GSP|GPU has fallen off|AER|PCIe|DPC|MCE|EDAC|watchdog|panic|oops|lockup'

Reboot history

journalctl --list-boots
last -x

NVIDIA diagnostic archive

nvidia-bug-report.sh

This creates:

nvidia-bug-report.log.gz

Review the archive before sharing it because diagnostic files may contain hostnames, usernames, network information and other environment details.

An NVIDIA diagnostic archive was also generated during the anonymized investigation after the application-triggered reboot.

How to interpret common results

nvidia-smi works

This proves:

• the driver can communicate with the GPU at that moment;
• the basic NVIDIA management interface is working.

It does not prove:

• CUDA workloads are stable;
• large memory allocations work;
• PCIe remains stable under load;
• container GPU access works;
• the system will survive a real AI workload.

CUDA works, but the application crashes

Possible causes include:

• application-specific kernels;
• unsupported CUDA operations;
• high VRAM fragmentation;
• model-loading behavior;
• container-library mismatch;
• GPU firmware interaction;
• PCIe or interrupt problems;
• storage and network activity occurring at the same time;
• motherboard firmware;
• power-delivery transients.

The server crashes under two different kernels

This makes a kernel-specific bug less likely, but it does not eliminate the kernel completely.

The two kernels may still share:

• the same NVIDIA driver;
• the same firmware;
• the same hardware;
• the same BIOS;
• the same application;
• the same container image.

The correct conclusion is:

Changing the kernel did not solve the problem.

It is not automatically:

The kernel is completely unrelated.

The reference server has a different NVIDIA driver

This is a valuable discovery.

However, immediately upgrading the affected server may remove useful evidence.

First record:

• current driver packages;
• module source;
• DKMS status;
• kernel versions;
• container-toolkit version;
• repositories.

Then align the driver in a controlled maintenance window.

No NVIDIA Xid appears

NVIDIA Xid errors are valuable, but their absence does not prove that the GPU is healthy.

A sudden reset may occur before the kernel flushes the error to disk.

The root cause could also be outside the GPU, such as:

• motherboard firmware;
• power delivery;
• PCIe controller;
• CPU or memory instability;
• network or storage DMA;
• hardware watchdog.

Common mistakes to avoid

Comparing only nvidia-smi

Two systems may display similar GPU information while using different:

• kernels;
• kernel modules;
• package sources;
• container runtimes;
• firmware versions.

Updating everything at once

Updating the BIOS, kernel, driver and application simultaneously may restore stability, but it destroys the ability to identify the actual cause.

Assuming a newer kernel is always better

A newer kernel may contain important fixes, but it may also have received less testing with a particular enterprise driver stack.

“Newer” does not automatically mean “more stable for this exact workload.”

Assuming an older enterprise kernel is obsolete

Enterprise Linux kernels often include extensive backported fixes while keeping the same major version number.

A 5.14 enterprise kernel can contain many security and hardware fixes that are not obvious from the version alone.

Ignoring firmware

Kernel and driver troubleshooting should also consider:

• BIOS;
• motherboard firmware;
• GPU firmware;
• Intel or AMD platform firmware;
• network-adapter firmware.

Software cannot always compensate for outdated platform firmware.

Recommended comparison checklist

Before declaring two NVIDIA nodes equivalent, compare:

• Linux distribution and release;
• exact kernel version;
• NVIDIA driver version;
• NVIDIA module type;
• DKMS or precompiled module status;
• loaded NVIDIA modules;
• CUDA compatibility;
• container-toolkit version;
• GPU model and VRAM;
• PCIe link width;
• BIOS version;
• Secure Boot status;
• power limit;
• application container image;
• application arguments.

A difference in any one of these fields may be significant.

Frequently asked questions

Can an NVIDIA driver work with more than one Linux kernel?

Yes.

With DKMS, the same driver version can be compiled separately for multiple installed kernels. Each kernel receives its own module.

Does installing a new kernel automatically update the NVIDIA driver?

Not necessarily.

DKMS may rebuild the existing driver for the new kernel. A repository may also provide a new compatible module. But the NVIDIA driver branch itself may remain unchanged.

Why does the system boot without a working NVIDIA module?

Linux can boot without the NVIDIA driver. The GPU simply may not be usable for CUDA workloads.

Can nvidia-smi work while containers cannot access the GPU?

Yes.

The host driver may work while the NVIDIA Container Toolkit, container runtime configuration or Kubernetes device plugin is broken.

Is a CUDA version mismatch always fatal?

No.

NVIDIA drivers provide backward compatibility for many CUDA applications. The important relationship is between the application’s CUDA runtime requirements and the capabilities supported by the installed driver.

Should I copy the driver configuration from a stable server?

Use the stable server as a reference, but do not copy packages blindly.

First compare:

• GPU generation;
• operating system;
• kernel;
• repositories;
• Secure Boot;
• package installation method.

Should I update the kernel or NVIDIA driver first?

When troubleshooting, choose one variable.

If a known-good reference server uses the same operating system and hardware, aligning one layer at a time produces the clearest evidence.

Final conclusion

NVIDIA GPU stability on Linux does not depend on a single version number.

It depends on the complete relationship between:

• hardware;
• firmware;
• Linux kernel;
• NVIDIA kernel module;
• user-space libraries;
• CUDA runtime;
• container platform;
• machine-learning application.

A working nvidia-smi command is only the beginning of the validation process.

The most effective troubleshooting method is to:

1. record the complete software stack;
2. compare it with a stable reference system;
3. verify the NVIDIA module for the active kernel;
4. run simple and sustained GPU tests;
5. reproduce the real workload;
6. change only one variable at a time;
7. preserve logs after every failure.

In the anonymized case described here, switching from a newer vendor kernel to the enterprise kernel used by the stable node did not eliminate the reboot. That result prevented an incorrect conclusion and redirected the investigation toward other differences, including the NVIDIA driver branch, firmware and wider platform stability.

That is the real value of a kernel and driver comparison: not immediately proving the final cause, but systematically eliminating incorrect assumptions.