A GPU power limit is often one of the first safety measures applied when a workstation or server unexpectedly reboots under artificial intelligence workloads.

The reasoning appears straightforward:

1. The GPU may normally consume up to 300 watts.
2. The system becomes unstable during heavy GPU activity.
3. The administrator reduces the limit to 250 watts.
4. Lower power consumption should prevent the reboot.

Unfortunately, a GPU power cap does not guarantee system stability.

In the incident examined in this article, the power limit was successfully reduced from 300 watts to 250 watts, persistence mode was enabled, and GPU telemetry confirmed that the new limit was active. The machine still experienced an abrupt hard reboot while an inference server was loading and optimizing a language model.

More surprisingly, the last recorded GPU sample showed substantially less than the configured limit:

Configured GPU limit: 250 W
Maximum sampled power: approximately 175 W
Maximum temperature: approximately 53°C
Maximum VRAM use: approximately 12 GB

The operating system recorded a new boot ID, while the previous session was classified as a crash rather than a clean shutdown.

This article explains why that can happen, what a GPU power limit actually controls, and how to investigate the problem safely.

All hostnames, application names, volume identifiers and organization-specific details have been removed or generalized.

The incident scenario

The affected machine was a Linux worker in a small Kubernetes cluster.

It contained:

• a modern NVIDIA GPU;
• an NVIDIA driver and CUDA-enabled container runtime;
• a Kubernetes workload running an OpenAI-compatible inference server;
• a locally cached language model;
• persistent storage for the model files;
• approximately 128 GB of system memory.

Earlier tests had established that:

• Kubernetes detected the GPU;
• CUDA was available inside containers;
• model files were accessible;
• the storage volume could attach correctly;
• the inference server could begin loading model weights;
• short and sustained matrix-multiplication tests could run.

A synthetic GPU workload allocated approximately 2 GB of VRAM and performed repeated matrix multiplication for several minutes. That test completed thousands of iterations, demonstrating that basic CUDA execution was functional.

The more complex AI workload behaved differently.

During vLLM startup, the server:

1. initialized NCCL;
2. selected a FlashAttention backend;
3. loaded three model-weight shards;
4. allocated roughly 7.56 GB for model weights;
5. initialized the key-value cache;
6. performed kernel compilation and autotuning;
7. began or completed CUDA graph capture.

The application log then stopped abruptly without a normal Python exception or graceful shutdown.

At almost the same time, the entire physical node rebooted.

What a GPU power limit actually does

An NVIDIA power limit controls the power budget enforced by the GPU’s firmware and driver.

A command such as:

sudo nvidia-smi -pl 250

requests that the GPU operate within a 250-watt software power limit.

The active value can be confirmed with:

nvidia-smi \
  --query-gpu=power.limit,power.min_limit,power.max_limit \
  --format=csv

Example:

power.limit [W], power.min_limit [W], power.max_limit [W]
250.00 W, 250.00 W, 300.00 W

In the investigated system, the GPU reported that the requested limit had been applied successfully.

However, this limit applies only to the GPU’s managed power domain. It is not a complete server-wide power limit.

It does not directly cap:

• CPU package power;
• motherboard power consumption;
• memory power;
• storage devices;
• fans and pumps;
• PCIe slot power transients;
• PSU input current;
• transient behavior in connectors and cables;
• momentary system-wide load changes.

A 250-watt GPU limit therefore does not mean that the entire machine will remain below a safe total power threshold.

Why the server can reboot below the configured GPU limit

There are several reasons a machine can reset even when telemetry reports power consumption below the configured cap.

1. Telemetry samples can miss very short transients

A common monitoring command is:

nvidia-smi \
  --query-gpu=timestamp,pstate,temperature.gpu,power.draw,power.limit,utilization.gpu,memory.used \
  --format=csv \
  -l 1

The -l 1 option records one sample per second.

That is useful for observing sustained behavior, but one second is a long period in electrical terms. A problematic power transient may last only milliseconds.

A telemetry file might therefore show:

06:21:16  123.74 W
06:21:17  175.17 W
server resets

without ever recording the actual short-lived peak that occurred between samples. The final recorded value is not necessarily the maximum instantaneous electrical demand.

In this incident, telemetry reached approximately 175 watts and 53°C immediately before logging stopped. The configured limit remained 250 watts.

That evidence rules out a sustained 250-watt thermal overload, but it does not rule out a fast transient or a fault in the power path.

2. GPU power and total system power are different measurements

AI startup can stress several components simultaneously:

• GPU cores;
• GPU memory;
• CPU cores compiling kernels;
• system RAM;
• PCIe transfers;
• storage reads;
• container runtime;
• background Kubernetes services.

A synthetic GPU-only test may draw heavy GPU power while leaving the CPU relatively quiet.

A model server can create a different combined load profile:

CPU compilation + RAM pressure + storage I/O + GPU allocation + CUDA warmup

The PSU must support the total instantaneous system demand, not only the GPU’s reported board power.

A machine may therefore pass a GPU stress test but fail during a mixed CPU-and-GPU AI initialization phase.

3. Power supplies react to current spikes, not only average wattage

A PSU can shut down because of protection mechanisms such as:

• over-current protection;
• over-power protection;
• under-voltage protection;
• short-circuit protection;
• thermal protection.

These protections respond to electrical conditions that may not be visible in software telemetry.

A system may average well below the PSU’s advertised wattage and still trigger protection because of:

• fast load transitions;
• aging capacitors;
• poor transient response;
• an overloaded rail;
• an inadequate cable;
• a loose connector;
• an unsuitable adapter;
• shared or daisy-chained PCIe power cables.

A successful nvidia-smi -pl command does not test the PSU or cabling.

4. The GPU power limit may not cover every electrical path

Depending on the card and platform, power can be supplied through:

• the auxiliary GPU power connector;
• the motherboard PCIe slot;
• multiple voltage rails;
• memory and board-level power domains.

The GPU firmware manages its own power behavior, but a fault can still exist in:

• the PCIe slot;
• the motherboard;
• a riser;
• an adapter;
• the cable;
• the PSU connector;
• the card’s voltage-regulation components.

The server may reboot because a connector loses stable contact even though the GPU’s calculated board power remains below its configured limit.

5. CUDA graph capture can create a different load pattern

Modern inference engines optimize performance during startup.

Depending on configuration, startup may include:

• kernel compilation;
• FlashAttention initialization;
• FlashInfer autotuning;
• memory profiling;
• key-value-cache allocation;
• CUDA graph capture;
• warm-up inference passes.

These operations are not equivalent to a simple matrix-multiplication loop.

In the examined logs, model loading completed successfully and startup advanced into graph-related optimization before the node reset.

That does not prove that CUDA graph capture caused the reboot. It identifies the stage at which the failure occurred and provides a useful next isolation test.

6. A low-level driver or firmware problem may reset the platform

Not every hard reboot is caused by the PSU.

Other possibilities include:

• GPU firmware failure;
• NVIDIA driver fault;
• GSP firmware issue;
• PCIe bus instability;
• motherboard firmware problem;
• kernel regression;
• hardware machine-check event;
• watchdog reset;
• BIOS power-management bug.

The difficulty is that an abrupt reset can occur before the operating system has time to save useful evidence.

In this case, post-reboot searches found no conclusive:

• NVIDIA Xid or SXid;
• PCIe AER fatal error;
• kernel panic;
• machine-check exception;
• watchdog lockup;
• thermal shutdown;
• persistent pstore crash record.

The absence of these records does not prove that the driver and hardware were healthy. It means Linux did not preserve a clear diagnostic before the reset.

The difference between an application crash and a hard reboot

This distinction is essential.

Application-level failure

An application crash normally leaves evidence such as:

Traceback
CUDA out of memory
Segmentation fault
Engine process terminated
Container exited with code 1

The Linux host remains online.

Kubernetes may restart the pod, and kubectl logs --previous may show the error.

GPU driver failure

A GPU driver problem may produce:

NVRM: Xid
GPU has fallen off the bus
CUDA error
NCCL failure

The operating system may remain online, although the GPU could become unusable.

Full platform reset

A hard reset usually presents differently:

• SSH disconnects suddenly;
• Kubernetes watch streams terminate;
• port-forward connections close;
• the node temporarily becomes NotReady;
• boot ID changes;
• last -x reports crash;
• logs end abruptly;
• no normal shutdown sequence exists.

That was the pattern observed in this incident. The application log did not end with a normal error; the whole node restarted.

Why a successful stress test does not clear the hardware

A synthetic stress test can be valuable, but it cannot reproduce every workload.

For example, this test pattern:

a = torch.randn((4096, 4096), device="cuda", dtype=torch.float16)
b = torch.randn((4096, 4096), device="cuda", dtype=torch.float16)

while time.time() < end_time:
    result = a @ b
    torch.cuda.synchronize()

primarily exercises repeated matrix multiplication.

It may produce:

• high GPU utilization;
• stable power draw;
• predictable VRAM usage;
• relatively constant clock behavior.

An inference-server startup has a more irregular pattern:

• allocate large memory regions;
• load files;
• transfer data;
• compile kernels;
• release and reallocate buffers;
• switch performance states;
• autotune kernels;
• capture graphs;
• initialize networking libraries.

A machine can pass 180 seconds of stable computation and still fail during rapid workload transitions.

The synthetic test in this incident demonstrated that the GPU was not universally unusable. It did not prove that the entire platform was stable under every AI workload.

How to confirm that the power limit is active

Before interpreting a test result, verify the configuration.

Enable persistence mode:

sudo nvidia-smi -pm 1

Set the power limit:

sudo nvidia-smi -pl 250

Confirm the current state:

nvidia-smi \
  --query-gpu=name,persistence_mode,power.limit,power.draw,temperature.gpu,memory.used \
  --format=csv

Expected example:

name, persistence_mode, power.limit [W], power.draw [W], temperature.gpu, memory.used [MiB]
NVIDIA GPU, Enabled, 250.00 W, 12.00 W, 44, 0 MiB

The incident data confirmed both persistence mode and the 250-watt limit before the final model-loading test.

Important: power limits may reset after reboot

After an unexpected reboot, verify the setting again.

Unless the configuration is applied automatically during startup, the GPU may return to its default power limit.

A systemd unit can reapply it:

[Unit]
Description=Configure NVIDIA GPU power limit
After=nvidia-persistenced.service

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -pm 1
ExecStart=/usr/bin/nvidia-smi -pl 250
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Use this only after choosing a power limit supported by the GPU.

Capture useful GPU telemetry

Create a diagnostic directory:

sudo mkdir -p /var/log/gpu-diagnostics

Start one-second monitoring:

LOG="/var/log/gpu-diagnostics/gpu-$(date +%Y%m%d-%H%M%S).csv"

nohup nvidia-smi \
  --query-gpu=timestamp,pstate,temperature.gpu,power.draw,power.limit,utilization.gpu,memory.used,clocks.sm \
  --format=csv \
  -l 1 \
  > "$LOG" 2>&1 < /dev/null &

echo $! | sudo tee /run/gpu-monitor.pid
echo "$LOG"

Verify that it is running:

ps -fp "$(cat /run/gpu-monitor.pid)"
tail -5 "$LOG"

Useful fields include:

• timestamp;
• performance state;
• GPU temperature;
• power draw;
• configured limit;
• utilization;
• VRAM used;
• SM clock.

For higher-resolution observations, nvidia-smi dmon may be useful:

nvidia-smi dmon -s pucvmet -d 1

Software monitoring still cannot replace electrical instrumentation for millisecond-scale events.

Record the boot identity before testing

Linux exposes a unique identifier for each boot:

cat /proc/sys/kernel/random/boot_id

Before the test:

cat /proc/sys/kernel/random/boot_id \
  > /run/test-boot-id

After the test:

echo "Before: $(cat /run/test-boot-id)"
echo "After:  $(cat /proc/sys/kernel/random/boot_id)"

If the values differ, the machine rebooted.

This is more reliable than relying only on SSH disconnection or Kubernetes errors.

In the incident, the boot ID changed from one value to another, confirming that the physical node had restarted.

Enable persistent Linux logging

Without persistent journaling, the most important evidence may disappear during a hard reboot.

Check available boots:

journalctl --list-boots

Create persistent storage when necessary:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Verify:

journalctl --list-boots

After another reboot, inspect the previous boot:

journalctl -b -1 --no-pager

Kernel-only messages:

journalctl -k -b -1 --no-pager

Serious warnings and errors:

journalctl -b -1 -p warning..alert --no-pager

Search for GPU, PCIe and platform errors

Use a targeted filter:

journalctl -k -b -1 --no-pager |
grep -Ei \
'NVRM|Xid|SXid|GPU has fallen off|AER|PCIe|DPC|MCE|EDAC|hardware error|watchdog|thermal|overheat|power|panic|oops|lockup'

Check persistent crash storage:

find /sys/fs/pstore /var/lib/systemd/pstore \
  -maxdepth 2 \
  -type f \
  -print \
  -exec sed -n '1,240p' {} \; \
  2>/dev/null

Check boot history:

last -x -F | head -30

A line containing:

crash

indicates that the previous session did not close normally.

Capture an NVIDIA diagnostic report

Before reproducing the issue again, create a baseline report:

sudo mkdir -p /var/log/gpu-diagnostics
cd /var/log/gpu-diagnostics

sudo nvidia-bug-report.sh

The report usually produces a compressed file similar to:

nvidia-bug-report.log.gz

Preserve one report:

• while idle;
• after a successful synthetic test;
• after recovering from a failed test.

Search it:

zgrep -Ei \
'NVRM|Xid|SXid|GSP|PCIe Bus Error|AER|fatal|hardware error|power|thermal' \
nvidia-bug-report.log.gz

A report collected after reboot cannot always recover events that were never written, but it provides valuable information about:

• driver version;
• firmware;
• GPU state;
• PCIe topology;
• power settings;
• module configuration.

Check PCIe link and slot health

Inspect the GPU:

lspci -nn | grep -i nvidia

Then:

sudo lspci -vv -s <gpu-pci-address>

Review:

LnkCap
LnkSta
DevSta
AER
DPC
Correctable Error
Non-Fatal Error
Fatal Error
Unsupported Request

Also inspect the upstream root port:

sudo lspci -vv -s <root-port-address>

Important questions include:

• Is the expected PCIe width negotiated?
• Is the GPU behind a riser?
• Are correctable errors accumulating?
• Is the link frequently retraining?
• Is the GPU installed in the intended slot?
• Does the motherboard share lanes with another device?

A reduced link width is not automatically the cause of a reboot, but unusual topology should be included in the investigation.

Physical checks that software cannot perform

Before running another high-load test, shut down the machine completely.

Disconnect mains power and inspect:

GPU power connector

Check for:

• incomplete insertion;
• visible gaps;
• melted plastic;
• discoloration;
• damaged pins;
• unusual smell;
• cable tension;
• sharp bending near the connector.

PSU cables

Confirm that:

• original PSU cables are used;
• modular cables belong to that exact PSU model;
• dedicated cables are used where required;
• no unsupported splitter is present;
• no daisy-chain is overloaded;
• adapters are properly seated.

Modular PSU cables are not universally interchangeable, even when the connectors appear identical.

GPU seating

Confirm that:

• the GPU is fully inserted;
• the retention latch is engaged;
• the card is mechanically supported;
• the bracket is not pulling the card out of alignment.

PCIe risers and extenders

Temporarily remove risers or extensions when possible.

A marginal riser may work at idle and fail during link activity or power-state transitions.

Motherboard power

Verify:

• 24-pin motherboard connector;
• CPU EPS connectors;
• PCIe slot;
• no signs of overheating;
• no loose connection.

A safer isolation strategy

Do not immediately repeat the exact workload after a hard reboot.

Use controlled, incremental tests.

Stage 1: idle validation

Confirm:

nvidia-smi
uptime
journalctl -k -b --no-pager |
grep -Ei 'NVRM|Xid|AER|MCE|hardware error'

Stage 2: VRAM allocation only

Allocate memory without sustained compute.

Example:

import torch
import time

buffers = []

for _ in range(8):
    buffers.append(
        torch.empty(
            (256, 1024, 1024),
            device="cuda",
            dtype=torch.float16,
        )
    )
    print(torch.cuda.memory_allocated() / 1024**3)

time.sleep(120)

Stage 3: short compute load

Run for 10 seconds.

Stage 4: longer compute load

Increase gradually:

10 seconds
30 seconds
60 seconds
180 seconds

Stage 5: combined CPU and GPU load

Only after GPU-only tests succeed.

Stage 6: inference server without CUDA graphs

For vLLM, a useful isolation option is:

--enforce-eager

This disables CUDA graph execution and keeps the model in eager mode.

Example:

python3 -m vllm.entrypoints.openai.api_server \
  --model /models/model-snapshot \
  --served-model-name local-model \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.70 \
  --enforce-eager

If eager mode works while graph capture consistently triggers a reset, the test has isolated an important difference.

It still does not prove whether the cause is:

• software;
• driver;
• firmware;
• hardware;
• power delivery.

Consider a lower GPU limit for diagnosis

A 250-watt limit may be the minimum supported value on some high-power cards. In other cases, lower values may be available.

Check:

nvidia-smi \
  --query-gpu=power.min_limit,power.max_limit \
  --format=csv

Use only supported values.

A lower limit can reduce risk during testing, but it is not a repair.

If the machine still resets well below the normal GPU limit, that strengthens the case for investigating:

• PSU quality and transient response;
• cabling;
• connector condition;
• motherboard;
• PCIe topology;
• driver and firmware;
• GPU hardware.

Kubernetes containment during hardware investigation

When the affected machine is a Kubernetes worker, prevent workloads from returning automatically.

Cordon it:

kubectl cordon <gpu-node>

Scale the inference deployment to zero:

kubectl scale deployment <inference-deployment> \
  -n <namespace> \
  --replicas=0

Delete standalone test pods:

kubectl delete pod <test-pod> \
  -n <namespace> \
  --ignore-not-found \
  --wait=false

Verify:

kubectl get node <gpu-node>
kubectl get pods -A -o wide |
grep <gpu-node>

Expected node status:

Ready,SchedulingDisabled

This prevents the scheduler from restarting the heavy workload while diagnostics are still underway.

What not to conclude from the evidence

“The GPU never reached 250 watts, so power is not the problem”

Incorrect.

A one-second sample can miss a transient, and the failure may involve the PSU, connector, motherboard or total system power rather than GPU average power.

“The temperature was only 53°C, so the hardware is healthy”

A low GPU core temperature rules out ordinary sustained GPU overheating at that moment.

It does not rule out:

• connector overheating;
• VRM fault;
• PSU protection;
• PCIe fault;
• firmware problem;
• electrical transient.

“The stress test passed, so the GPU is good”

A passing stress test is encouraging, but not conclusive.

Different workloads exercise different:

• memory patterns;
• instruction paths;
• clocks;
• power transitions;
• driver features;
• kernel compilation stages.

“The inference application caused the physical reboot”

The application triggered the workload that exposed the failure.

That is different from proving the application itself was defective.

“There was no Xid, so the GPU cannot be involved”

A full reset can occur before an Xid is saved.

Absence of evidence is not proof of absence.

Building an evidence table

A structured table helps prevent premature conclusions.

Recommended troubleshooting order

A sensible order is:

1. Contain the Kubernetes workload.
2. Confirm the power limit and persistence mode.
3. Enable persistent system logging.
4. Record the boot ID.
5. Capture NVIDIA and PCIe baselines.
6. Inspect the physical power path.
7. Verify the PSU model, wattage, age and cabling.
8. Run short, incremental GPU tests.
9. Test combined CPU and GPU load separately.
10. Retry inference with eager execution and conservative memory use.
11. Test the GPU in another machine or install a known-good PSU.
12. Escalate with collected logs if the reset persists.

The most informative hardware-isolation tests are often:

• known-good PSU in the same machine;
• same GPU in another machine;
• different GPU in the affected machine;
• removal of any PCIe riser;
• dedicated power cables;
• updated BIOS and stable driver/kernel combination.

Change only one major variable at a time.

Frequently asked questions

Does nvidia-smi -pl protect the entire server?

No.

It limits the GPU’s managed power target. It does not directly cap CPU, memory, storage, fans, motherboard power or PSU input power.

Why did the machine reboot when telemetry showed only 175 watts?

Possible explanations include:

• a transient between one-second samples;
• combined CPU and GPU demand;
• PSU protection;
• cable or connector instability;
• motherboard or PCIe power issue;
• driver or firmware fault;
• GPU hardware problem.

The recorded 175 watts is the last sampled value, not guaranteed peak instantaneous consumption.

Can CUDA graph capture reboot a server?

CUDA graph capture should not normally reboot a healthy server.

However, it can create a distinct workload involving memory allocation, kernel execution and rapid state changes. If the platform has a latent hardware, firmware or driver problem, that phase may expose it.

Does a low GPU temperature eliminate a power-supply problem?

No.

PSU and connector problems can occur while the GPU core remains cool.

Should I reduce the power limit further?

A lower supported limit can be useful for diagnosis, but it should not be treated as the final repair.

Persistent resets at reduced power require hardware and platform investigation.

Should I replace the PSU first?

A known-good, appropriately sized PSU is one of the most valuable tests when abrupt resets occur without kernel evidence.

Before replacing it, also inspect:

• cable routing;
• dedicated GPU cables;
• connector seating;
• risers;
• motherboard slot;
• modular cable compatibility.

Why are there no useful logs after the reboot?

A sudden electrical reset can stop the CPU before Linux flushes journal buffers.

Persistent journaling and pstore improve the chance of retaining evidence, but neither guarantees that an abrupt hardware reset will produce a useful record.

Is the GPU definitely defective?

No.

The evidence identifies a platform-level instability correlated with a complex GPU workload.

Possible causes include the GPU, PSU, cable, motherboard, PCIe path, firmware, driver or interaction among several components.

Conclusion

Reducing an NVIDIA GPU from a 300-watt limit to 250 watts is a sensible diagnostic and risk-reduction step, but it is not a guarantee against hard server reboots.

In the anonymized incident described here:

• the 250-watt limit was confirmed active;
• GPU temperatures remained moderate;
• sampled power stayed below the configured limit;
• basic CUDA workloads passed;
• the AI model loaded successfully;
• the machine rebooted during advanced inference-server initialization;
• no conclusive kernel panic, NVIDIA Xid or persistent crash record was preserved.

The correct conclusion is not that the power cap failed technically. The cap did what it was designed to do: constrain the GPU’s managed power target.

The broader lesson is that GPU power limiting is only one layer of system stability.

A reliable diagnosis must also consider:

• short electrical transients;
• total-system power;
• PSU protections;
• power cables and connectors;
• motherboard and PCIe delivery;
• GPU firmware and drivers;
• workload-specific initialization phases;
• logging limitations during abrupt resets.

When a node reboots without a clean shutdown, treat it as a platform incident rather than an ordinary application crash. Contain the workload, preserve evidence, inspect the physical power path and test one variable at a time.