How We Diagnosed Random Linux GPU Server Reboots During vLLM Startup

Running a large language model locally can look simple on paper:

1. Install an NVIDIA GPU.
2. deploy vLLM;
3. download a model;
4. expose an API;
5. start sending prompts.

In reality, an AI workload exercises many parts of a computer at the same time:

• GPU and GPU driver;
• CPU and RAM;
• PCI Express bus;
• power supply;
• network adapter;
• Linux kernel;
• Kubernetes networking;
• distributed storage;
• container runtime;
• motherboard firmware.

When the entire machine suddenly reboots, troubleshooting becomes especially difficult. There may be no normal application error because the operating system disappears before it can save one.

This article explains, in beginner-friendly language, how we diagnosed an unstable Linux GPU worker that repeatedly restarted while loading a language model through vLLM.

All identifying information, hostnames, IP addresses, usernames and storage identifiers have been removed.

The environment

The affected machine was a Linux worker in a small Kubernetes cluster.

It included:

• a modern NVIDIA consumer GPU with approximately 16 GB of video memory;
• an Intel desktop platform;
• K3s as the Kubernetes distribution;
• Longhorn for distributed block storage;
• an OpenAI-compatible vLLM server;
• a model of approximately 7.5 GB;
• an onboard Intel Ethernet adapter;
• an NVIDIA container runtime.

The model was stored on a Kubernetes PersistentVolume backed by Longhorn.

The machine appeared stable during ordinary use. The problem occurred mainly when vLLM loaded the model and initialized the inference engine.

The original symptom

During startup, vLLM would begin normally:

• the container started;
• the model architecture was detected;
• the tokenizer was loaded;
• the GPU was initialized;
• the checkpoint shards began loading;
• GPU memory was allocated.

Then the entire Linux host rebooted.

This was not merely a pod restart. The physical machine itself restarted.

That distinction matters.

A pod restart normally points toward:

• an application crash;
• an out-of-memory condition;
• a bad container configuration;
• a failed health probe.

A host reboot points toward a deeper problem, such as:

• kernel panic;
• hardware instability;
• power interruption;
• firmware defect;
• PCIe failure;
• driver deadlock;
• watchdog-triggered reset;
• motherboard-level problem.

Why random reboots are difficult to diagnose

Most useful logs are stored in memory before being written to disk.

When a machine freezes or resets abruptly, the final seconds may never reach the journal. As a result, the last visible error is not always the real cause.

For example, a network driver may report a timeout immediately before the reboot. That does not automatically prove that the network card caused the crash.

It may instead mean that:

• the PCIe bus stopped responding;
• interrupts were no longer delivered;
• the CPU was stuck;
• DMA operations stalled;
• the motherboard firmware entered an invalid state.

The network adapter may simply be the first component that noticed the wider failure.

First rule: contain the problem

Before investigating, we stopped the production workload.

The Kubernetes worker was cordoned so normal pods could not be scheduled on it. The vLLM Deployment was scaled to zero, and the model volume was detached.

This protected:

• the model cache;
• the distributed storage replicas;
• unrelated Kubernetes workloads;
• the filesystem from repeated unclean resets.

A common troubleshooting mistake is to repeat the failing workload too many times before preserving evidence.

Every hard reboot creates some risk of:

• filesystem corruption;
• storage replica degradation;
• incomplete writes;
• damaged container state;
• lost logs.

Step 1: confirm that the machine really rebooted

Linux exposes a unique identifier for every boot:

cat /proc/sys/kernel/random/boot_id

Before each test, we saved this value. After the test, we compared it again.

If the value changed, the host had rebooted.

We also checked:

uptime
journalctl --list-boots
last -x

This is more reliable than assuming that a lost SSH session means the machine restarted. SSH can disconnect because of a network outage while the server continues running.

Step 2: collect the previous boot’s logs

After a reboot, the previous kernel journal can be inspected with:

journalctl -k -b -1

We searched for important terms:

journalctl -k -b -1 --no-pager |
grep -Ei \
'NVRM|Xid|AER|PCIe|NETDEV WATCHDOG|panic|oops|lockup|thermal|MCE|EDAC'

The logs did not show a normal NVIDIA Xid error, kernel panic, overheating event or out-of-memory kill.

However, shortly before multiple reboots, the Intel network driver reported messages similar to:

NETDEV WATCHDOG
transmit queue timed out
Reset adapter

This became an important clue, but not yet a confirmed root cause.

What does NETDEV WATCHDOG mean?

Linux network interfaces use transmission queues.

A simplified flow looks like this:

1. Linux places packets in a transmit queue.
2. The network adapter reads them.
3. The adapter sends them onto the network.
4. The driver receives confirmation that the work completed.

A NETDEV WATCHDOG message means Linux waited too long for one transmit queue to make progress.

Possible causes include:

• network-driver bugs;
• adapter firmware problems;
• PCIe communication issues;
• interrupt problems;
• DMA stalls;
• hardware failure;
• platform firmware bugs;
• extreme system lockups.

It does not necessarily mean that the Ethernet cable is bad.

In our case, cable-level counters were clean:

• no CRC errors;
• no receive errors;
• no transmit errors;
• no missed packets;
• no carrier errors.

That made a simple cable problem less likely.

Step 3: inspect the physical PCIe layout

The GPU was initially installed in a lower motherboard slot that provided fewer PCIe lanes than the card supported.

It was moved to the primary x16 slot.

After relocation, the link reported full x16 width:

PCIe capability: x16
Current width:   x16

At idle, the link speed dropped to a low-power PCIe state. This is normal. Modern GPUs reduce PCIe speed when inactive and raise it again under load.

The physical relocation removed one possible bottleneck, but the machine later rebooted again during vLLM startup. Therefore, slot placement alone was not the complete fix.

Step 4: test the GPU without vLLM

Before blaming the AI application, the GPU was tested directly.

The test:

• allocated several gigabytes of VRAM;
• performed continuous matrix multiplication;
• drove GPU utilization close to 100%;
• drew approximately 250 watts;
• ran for a sustained period.

The GPU remained stable.

Typical results included:

• approximately 99% GPU utilization;
• temperatures below 70°C;
• power draw near the configured limit;
• no NVIDIA Xid errors;
• no host reboot.

This was a major finding.

It weakened several theories:

• inadequate cooling;
• immediate GPU hardware failure;
• basic CUDA instability;
• sustained power-supply overload;
• failure caused simply by high GPU utilization.

A synthetic test cannot reproduce every AI workload, but passing it means the GPU can perform substantial work without immediately crashing the host.

Step 5: compare Linux kernels

The server initially ran a newer vendor kernel. We also tested a more conservative enterprise Linux kernel.

The NVIDIA driver loaded successfully on both.

The machine still rebooted during vLLM startup on the alternative kernel.

That allowed us to reject the simplest version of the theory:

“The problem is caused only by the original Linux kernel.”

The kernel could still influence the issue, but it was not the only condition required to trigger it.

Step 6: control the GPU power limit

The GPU’s default power limit was approximately 300 watts. For diagnostics, it was reduced to 250 watts:

nvidia-smi -pm 1
nvidia-smi -pl 250

Persistence mode was also enabled so the driver kept the GPU initialized.

This was not presented as a permanent fix. It was a controlled diagnostic measure that:

• reduced peak power demand;
• reduced one source of variability;
• made repeated tests more consistent.

Even with this limit, later inference tests reached more than 210 watts and over 90% GPU utilization without failure.

Step 7: investigate the motherboard firmware

The motherboard was running its original BIOS from early 2021.

That was a major warning sign because the server used:

• a much newer GPU generation;
• modern Linux kernels;
• current NVIDIA drivers;
• PCIe devices released years after the original firmware.

BIOS updates can contain fixes for:

• PCIe initialization;
• interrupt routing;
• device compatibility;
• CPU microcode;
• memory training;
• power-state transitions;
• Above 4G Decoding;
• Resizable BAR;
• system stability.

The BIOS was upgraded in two stages to the latest available release.

After the update, Linux confirmed the new version using:

dmidecode -s bios-version
dmidecode -s bios-release-date

A useful lesson about Intel Management Engine firmware

The BIOS release notes suggested that a newer Intel Management Engine firmware version was available.

However, Linux showed that the ME firmware had not changed:

cat /sys/class/mei/mei0/fw_ver

The system still reported the older ME release.

This is an important lesson:

Updating the BIOS does not always update the Intel ME firmware.

They may be stored in separate firmware regions and require separate vendor tools.

For this diagnostic, we continued with:

• the new BIOS;
• the old ME firmware.

This distinction later became useful because the system stabilized even though ME remained unchanged.

Therefore, the successful result could not be attributed to an ME update.

Step 8: verify the model files

A damaged model file can cause application failures, although it should not normally reboot an entire host.

The model directory contained:

• configuration files;
• tokenizer files;
• an index;
• three Safetensors checkpoint shards.

The complete model was copied from Longhorn storage to a local disk.

The copy process:

1. followed symbolic links;
2. copied every file;
3. calculated SHA-256 checksums;
4. compared source and destination;
5. verified all checkpoint shards.

All files matched.

This eliminated model corruption as a likely cause.

Step 9: isolate distributed storage

At this point, two major subsystems were still active during the failing test:

• GPU inference initialization;
• Longhorn-backed storage access.

Longhorn presents distributed storage to Kubernetes workloads. Underneath, this may involve:

• network traffic between storage replicas;
• block-device operations;
• iSCSI;
• filesystem reads;
• Kubernetes CSI components.

To isolate storage from GPU initialization, the verified model copy was placed on the worker’s local XFS filesystem.

A new vLLM pod mounted the local directory through a Kubernetes hostPath.

This pod did not use the Longhorn PersistentVolume.

Step 10: run vLLM from local storage

The local-storage test used conservative settings:

• eager execution;
• no CUDA graphs;
• FlashAttention;
• disabled FlashInfer autotuning;
• one sequence at a time;
• reduced maximum model length;
• controlled GPU memory utilization;
• 250-watt GPU limit.

vLLM started successfully.

The health endpoint returned:

HTTP 200

The model endpoint also listed the expected model.

More importantly, the host’s boot ID did not change.

The server remained alive.

Step 11: send actual inference requests

A successful startup is useful, but it is not enough. The GPU must also generate tokens under load.

We sent multiple sequential chat-completion requests.

During inference, telemetry showed:

• more than 90% GPU utilization;
• approximately 210–216 watts of power;
• temperatures around the high 50s or low 60s Celsius;
• stable GPU memory usage;
• no NVIDIA errors;
• no network watchdog;
• no reboot.

This proved that the system could run genuine inference, not merely hold the model in memory.

Step 12: reintroduce Longhorn storage

The local test changed two major variables compared with the original failures:

1. the BIOS had been updated;
2. Longhorn had been removed from the model path.

To determine which change mattered, Longhorn had to be reintroduced.

The same vLLM configuration was recreated, but this time the model was mounted again from the Longhorn PersistentVolume.

The model loaded all three checkpoint shards. vLLM completed initialization, printed Application startup complete, and returned HTTP 200 from the health endpoint.

This was the decisive isolation test.

Longhorn-backed startup now worked with the new BIOS.

Step 13: stress the Longhorn-backed instance

The same sequential inference workload was then sent to the Longhorn-backed vLLM pod.

The machine again remained stable.

Observed values included:

• 91–94% GPU utilization;
• approximately 210–216 watts;
• temperatures below 60°C;
• stable video-memory allocation;
• no kernel panic;
• no NVIDIA Xid;
• no PCIe error;
• no network watchdog;
• unchanged boot ID.

This significantly weakened the theory that Longhorn or iSCSI was the primary cause.

Understanding the tx_queue_restart counter

After the successful test, one network statistic had increased:

tx_queue_1_restart: 6

At first glance, this looked alarming because earlier failures involved a transmit queue timeout.

However, a queue restart counter is not the same thing as a full network-adapter reset.

A transmit queue can be temporarily stopped when its descriptor ring is busy. When space becomes available, Linux wakes the queue and continues transmitting.

A small restart count can therefore indicate normal queue backpressure.

The more serious counters remained clean:

tx_timeout_count: 0
tx_errors:        0
rx_errors:        0

There were also no accompanying kernel messages such as:

NETDEV WATCHDOG
transmit queue timed out
Reset adapter

Therefore, six queue wakeups were treated as something to monitor, not as a new failure.

The most likely root cause

The evidence strongly implicated the outdated motherboard BIOS.

Before the firmware update:

• vLLM startup repeatedly preceded hard host reboots;
• the final visible warning often came from the network driver;
• the problem occurred across more than one Linux kernel;
• synthetic GPU tests passed;
• no thermal or standard NVIDIA error explained the reset.

After the firmware update:

• local-storage vLLM startup passed;
• local inference passed under high GPU load;
• Longhorn-backed startup passed;
• Longhorn-backed inference passed;
• the network watchdog no longer appeared;
• the host no longer rebooted.

The Intel ME firmware remained unchanged, which further narrowed the effective change to the motherboard BIOS and related firmware configuration.

This does not prove with mathematical certainty that one specific BIOS bug caused the reboot. Firmware release notes rarely identify every low-level fix.

The most defensible conclusion is:

The old motherboard firmware was likely causing, or significantly contributing to, a platform-level PCIe, interrupt or device-coordination stall. The network timeout was probably a symptom of that wider stall rather than the original cause.

Why the network adapter looked guilty

The onboard network adapter was involved in:

• Kubernetes traffic;
• Longhorn replica traffic;
• storage operations;
• container networking;
• SSH administration.

If the platform stopped servicing interrupts or DMA correctly, the NIC transmit queue could stop progressing.

The network driver would then report a watchdog timeout because it had a built-in timer that noticed the lack of progress.

The GPU driver may not have produced an error first, especially if the machine reset before the NVIDIA timeout mechanism completed.

This is why the last error in a log is not always the root cause.

What did not solve the issue by itself

Several actions were useful but were not individually sufficient:

Moving the GPU

The GPU was moved to the correct x16 slot. This improved the hardware layout but did not alone stop the reboot.

Changing the Linux kernel

The failure occurred on more than one kernel family, so the original kernel was not the sole cause.

Limiting GPU power

The 250-watt limit reduced variability, but the server had previously failed at much lower observed power.

Disabling advanced vLLM optimizations

Eager mode and disabled autotuning simplified the workload, but the machine still rebooted before the BIOS update.

Using local storage

Local storage proved that vLLM and the GPU could work. However, Longhorn also worked after the BIOS update, so distributed storage was not ultimately proven to be the root cause.

Practical troubleshooting workflow

For similar Linux GPU server problems, the following order is useful.

1. Protect the system

• stop the workload;
• cordon the node;
• detach important storage;
• preserve logs.

2. Confirm whether the host rebooted

cat /proc/sys/kernel/random/boot_id
uptime
journalctl --list-boots

3. Inspect the previous kernel journal

journalctl -k -b -1

Search for:

• Xid;
• NVRM;
• NETDEV WATCHDOG;
• AER;
• PCIe Bus Error;
• panic;
• oops;
• lockup;
• MCE;
• EDAC;
• thermal events.

4. Test one subsystem at a time

• GPU-only stress;
• memory test;
• local storage;
• network storage;
• alternative kernel;
• reduced power limit.

5. Check firmware age

Record:

dmidecode -s baseboard-product-name
dmidecode -s bios-version
dmidecode -s bios-release-date

A BIOS several years older than the GPU deserves serious attention.

6. Verify firmware components independently

Do not assume BIOS, Intel ME, NIC NVM and device firmware are all updated together.

7. Monitor during every test

Useful tools include:

nvidia-smi
journalctl
ethtool -S
ip -s link

8. Reintroduce variables one at a time

A good isolation sequence might be:

1. GPU synthetic test;
2. vLLM with local model;
3. local inference;
4. vLLM with distributed storage;
5. distributed-storage inference;
6. production configuration.

Common mistakes to avoid

Changing too many variables at once

Updating the kernel, GPU driver, BIOS, NIC settings and vLLM image simultaneously may fix the issue, but you will not know which change mattered.

Assuming high GPU power caused the reset

A reboot during AI startup does not automatically mean the power supply failed. Measure power and reproduce the load with a controlled GPU test.

Treating every network counter as an error

Queue wakeups, packet drops and watchdog timeouts are different events. Interpret each counter in context.

Deleting storage during troubleshooting

Do not remove Longhorn volumes, replicas, PersistentVolumes or iSCSI state without proving they are stale and unrelated.

Assuming the newest kernel is always best

A newer kernel may contain useful fixes, but an enterprise kernel can sometimes provide a more stable baseline. Testing both can help isolate the issue.

Ignoring BIOS updates on Linux servers

Linux administrators often update packages while forgetting motherboard firmware. A modern GPU can expose platform bugs that ordinary server workloads never trigger.

Final result

The Linux GPU worker was eventually able to:

• mount its distributed model volume;
• load all model shards;
• initialize vLLM;
• expose an OpenAI-compatible API;
• return HTTP 200 from its health endpoint;
• generate responses repeatedly;
• sustain more than 90% GPU utilization;
• draw more than 210 watts;
• remain below safe temperatures;
• avoid network watchdog errors;
• remain online without rebooting.

The strongest corrective action was upgrading the motherboard BIOS from a very old release to the latest supported version.

Final takeaway

When a Linux AI server suddenly reboots, do not focus only on the GPU.

The real problem may live at the boundary between:

• GPU;
• PCIe;
• network adapter;
• storage;
• interrupt handling;
• motherboard firmware.

The best diagnostic strategy is not random replacement. It is controlled isolation:

Preserve evidence, change one variable, measure the result, and then reintroduce complexity step by step.

In this case, the final successful Longhorn-backed inference test showed that neither the AI model nor distributed storage was inherently broken. The platform firmware was the missing piece.

Frequently asked questions

Can an old BIOS really crash a Linux GPU server?

Yes. The BIOS configures PCIe devices, interrupt routing, memory maps, power states and CPU microcode before Linux starts. A firmware problem can appear only when several high-performance devices are active together.

Does NETDEV WATCHDOG always mean the NIC is defective?

No. It means a transmit queue stopped making progress. The cause could be the adapter, its driver, PCIe, DMA, interrupts or a wider system stall.

Why did the GPU stress test pass while vLLM failed?

A synthetic GPU test mainly stresses computation and memory. vLLM startup also involves disk reads, CPU work, network storage, container networking, GPU allocation and process coordination.

Is a tx_queue_restart counter dangerous?

Not by itself. A small count can mean that Linux temporarily stopped and later woke a busy transmit queue. It becomes concerning when paired with increasing errors, timeouts, packet loss or watchdog messages.

Should Intel ME still be updated?

Yes, as maintenance, when a supported vendor method is available. However, in this case the server became stable while the older ME firmware remained installed, so the ME update was not required to demonstrate the immediate improvement.

Should AI models be stored on local disks instead of Longhorn?

Local disks generally load models faster and reduce network traffic. Longhorn can still work correctly, as the successful test demonstrated. The right choice depends on availability, portability, recovery requirements and startup-time expectations.