A graphics card can pass a basic test and still fail when a real artificial intelligence workload starts.
A small CUDA calculation may use only a few hundred megabytes of GPU memory. A large language model, image generator or scientific application may use 8 GB, 12 GB or nearly all available VRAM.
That difference matters.
A high-VRAM GPU test helps answer a specific diagnostic question:
Can the GPU allocate, write to and retain a large amount of video memory without errors, crashes or system resets?
This article explains what VRAM is, why high-memory tests are useful, how to perform one safely and how to interpret the results correctly.
What is VRAM?
VRAM stands for Video Random Access Memory.
It is the memory physically attached to the GPU. Applications use it to store:
- neural-network weights;
- tensors;
- textures;
- image data;
- model activations;
- attention caches;
- temporary computation buffers;
- compiled GPU kernels.
System RAM and VRAM are different.
System RAM is primarily used by the CPU. VRAM is used directly by the GPU and is designed to provide very high bandwidth.
A GPU may have:
8 GB VRAM
12 GB VRAM
16 GB VRAM
24 GB VRAM
48 GB VRAM
For AI inference, VRAM capacity is often one of the most important limitations.
Why run a high-VRAM GPU test?
A normal GPU test might perform one matrix multiplication and exit successfully.
That proves only that:
- the GPU is visible;
- CUDA works;
- the driver can execute a basic operation;
- a small amount of VRAM can be used.
It does not prove that:
- most of the VRAM is usable;
- large allocations are stable;
- the GPU remains stable for several minutes;
- memory errors appear only at certain addresses;
- power and temperature remain controlled;
- the driver handles sustained allocations correctly.
A high-VRAM test is useful when a machine experiences:
- crashes while loading an AI model;
- system resets during model initialization;
CUDA out of memoryerrors;- unexplained container termination;
- NVIDIA Xid errors;
- GPU disappearance;
- application crashes after allocating several gigabytes;
- instability that does not appear in small tests.
What a high-VRAM test actually checks
A useful high-VRAM GPU test should do more than call an allocation function.
It should perform three stages.
Stage 1: Allocate memory
The program asks the GPU driver to reserve several gigabytes of VRAM.
This tests whether a large allocation is possible.
Stage 2: Touch the allocated memory
The program writes real values into the allocated tensors.
This is important because merely creating an empty tensor does not exercise the memory as thoroughly as writing data throughout it.
Stage 3: Compute while the memory remains allocated
The test performs repeated GPU calculations without releasing the large allocation.
This creates a more realistic workload involving:
- high memory occupancy;
- CUDA kernel execution;
- memory reads and writes;
- synchronization;
- sustained GPU activity.
What the test does not prove
Passing a high-VRAM test does not prove that the entire system is perfect.
It does not fully test:
- a specific AI framework;
- model-loading code;
- CUDA graph capture;
- FlashAttention;
- JIT compilation;
- distributed communication;
- network storage;
- container networking;
- PCIe edge cases;
- application-specific kernels;
- GPU firmware interactions.
A passing test means:
Large VRAM allocation and generic CUDA computation appear stable under the tested conditions.
That is valuable evidence, but it is not a complete hardware certification.
Preparing the system
Before starting, verify that the GPU is visible:
nvidia-smi
You should see information such as:
GPU name
Driver version
Memory usage
Temperature
Power usage
Verify CUDA through PyTorch:
python3 - <<'PY'
import torch
print("CUDA available:", torch.cuda.is_available())
if torch.cuda.is_available():
print("GPU:", torch.cuda.get_device_name(0))
print("CUDA runtime:", torch.version.cuda)
PY
A healthy result should show:
CUDA available: True
GPU: NVIDIA ...
Stop other GPU workloads
A high-VRAM test should normally run on an otherwise idle GPU.
Check active processes:
nvidia-smi
Look at the Processes section.
If another model server or application is using several gigabytes of VRAM, stop it before testing. Otherwise, the test may fail simply because insufficient memory is available.
In Kubernetes, cordon the test node if necessary:
kubectl cordon <gpu-node>
Also scale down GPU workloads:
kubectl scale deployment <gpu-workload> \
--replicas=0 \
-n <namespace>
A safe PyTorch high-VRAM test
The following script allocates GPU memory gradually in 256 MiB chunks, writes data into each chunk and then runs matrix multiplications while the memory stays allocated.
Save it as:
high_vram_test.py
import os
import sys
import time
import torch
GIB = 1024**3
MIB = 1024**2
target_gib = float(os.getenv("TARGET_VRAM_GIB", "8"))
duration_seconds = int(os.getenv("TEST_DURATION_SECONDS", "180"))
chunk_mib = int(os.getenv("CHUNK_SIZE_MIB", "256"))
safety_margin_gib = float(os.getenv("SAFETY_MARGIN_GIB", "1"))
def gib(value: int) -> float:
return value / GIB
def print_memory(label: str) -> None:
free_bytes, total_bytes = torch.cuda.mem_get_info()
allocated = torch.cuda.memory_allocated()
reserved = torch.cuda.memory_reserved()
print(
f"{label}: "
f"allocated={gib(allocated):.2f} GiB, "
f"reserved={gib(reserved):.2f} GiB, "
f"free={gib(free_bytes):.2f} GiB, "
f"total={gib(total_bytes):.2f} GiB",
flush=True,
)
if not torch.cuda.is_available():
raise RuntimeError("CUDA is unavailable")
device = torch.device("cuda:0")
print("GPU:", torch.cuda.get_device_name(device), flush=True)
print("PyTorch CUDA:", torch.version.cuda, flush=True)
free_bytes, total_bytes = torch.cuda.mem_get_info()
target_bytes = int(target_gib * GIB)
safety_bytes = int(safety_margin_gib * GIB)
print_memory("Initial state")
if target_bytes + safety_bytes > free_bytes:
safe_target = max(0, free_bytes - safety_bytes)
raise RuntimeError(
f"Requested target is too large. "
f"Requested={target_gib:.2f} GiB, "
f"safe available target={gib(safe_target):.2f} GiB"
)
chunk_bytes = chunk_mib * MIB
elements_per_chunk = chunk_bytes // 2 # float16 uses 2 bytes
allocations: list[torch.Tensor] = []
try:
print(
f"Allocating approximately {target_gib:.2f} GiB "
f"in {chunk_mib} MiB chunks",
flush=True,
)
allocated_bytes = 0
chunk_number = 0
while allocated_bytes < target_bytes:
remaining = target_bytes - allocated_bytes
current_chunk_bytes = min(chunk_bytes, remaining)
current_elements = current_chunk_bytes // 2
tensor = torch.empty(
current_elements,
device=device,
dtype=torch.float16,
)
# Writing data ensures that the allocation is actually exercised.
tensor.uniform_()
allocations.append(tensor)
torch.cuda.synchronize()
allocated_bytes += tensor.numel() * tensor.element_size()
chunk_number += 1
print_memory(f"After chunk {chunk_number}")
print("Target allocation completed", flush=True)
print("Creating computation tensors", flush=True)
matrix_a = torch.randn(
(4096, 4096),
device=device,
dtype=torch.float16,
)
matrix_b = torch.randn(
(4096, 4096),
device=device,
dtype=torch.float16,
)
torch.cuda.synchronize()
print_memory("Before computation")
end_time = time.time() + duration_seconds
iterations = 0
print(
f"Starting sustained computation for "
f"{duration_seconds} seconds",
flush=True,
)
while time.time() < end_time:
result = matrix_a @ matrix_b
torch.cuda.synchronize()
iterations += 1
if iterations % 20 == 0:
print(
f"Iterations completed: {iterations}",
flush=True,
)
print_memory("Current state")
print("High-VRAM GPU test completed successfully", flush=True)
print("Total iterations:", iterations, flush=True)
print_memory("Final test state")
except torch.OutOfMemoryError as exc:
print("CUDA out-of-memory error:", exc, file=sys.stderr)
print_memory("State at failure")
raise
finally:
allocations.clear()
if "matrix_a" in locals():
del matrix_a
if "matrix_b" in locals():
del matrix_b
if "result" in locals():
del result
torch.cuda.empty_cache()
torch.cuda.synchronize()
print_memory("After cleanup")
Running the test
For an 8 GiB allocation and a three-minute computation phase:
TARGET_VRAM_GIB=8 \
TEST_DURATION_SECONDS=180 \
python3 high_vram_test.py
For a smaller test:
TARGET_VRAM_GIB=4 \
TEST_DURATION_SECONDS=120 \
python3 high_vram_test.py
For a larger GPU:
TARGET_VRAM_GIB=16 \
TEST_DURATION_SECONDS=300 \
python3 high_vram_test.py
Never target 100% of the available VRAM.
Leave at least 1–2 GiB free for:
- the CUDA context;
- computation tensors;
- driver overhead;
- PyTorch memory reservations;
- temporary kernel buffers.
Monitoring the GPU during the test
Open another terminal and run:
watch -n 1 nvidia-smi
This gives a quick live view.
For a more useful diagnostic record, capture telemetry to a CSV file:
mkdir -p ~/gpu-diagnostics
LOG="$HOME/gpu-diagnostics/high-vram-$(date +%Y%m%d-%H%M%S).csv"
nvidia-smi \
--query-gpu=timestamp,name,pstate,temperature.gpu,power.draw,power.limit,utilization.gpu,memory.used,memory.total,clocks.sm \
--format=csv \
-l 1 \
> "$LOG"
Run this in a separate terminal, or place it in the background:
nohup nvidia-smi \
--query-gpu=timestamp,pstate,temperature.gpu,power.draw,power.limit,utilization.gpu,memory.used,memory.total,clocks.sm \
--format=csv \
-l 1 \
> "$LOG" 2>&1 &
echo $! > /tmp/gpu-monitor.pid
Stop it afterward:
kill "$(cat /tmp/gpu-monitor.pid)"
Inspect the end of the file:
tail -30 "$LOG"
Important values to monitor
Memory used
This should rise toward the requested target.
For example:
8200 MiB / 16384 MiB
PyTorch and nvidia-smi may show slightly different values.
PyTorch reports memory managed by its allocator. nvidia-smi reports memory associated with the entire process, including CUDA context and driver overhead.
A small difference is normal.
GPU utilization
During allocation, utilization may be low or variable.
During matrix multiplication, it may rise close to:
90–100%
Temperature
Temperature depends on:
- GPU model;
- cooling;
- room temperature;
- fan curve;
- case airflow.
A rising temperature is normal under load. A rapidly increasing or unusually high temperature should be investigated.
Power draw
Power draw may increase significantly during computation.
High memory occupancy alone does not necessarily use much power. Computation usually consumes more power than simply keeping tensors allocated.
P-state
You may see:
P8
P2
P1
P0
Lower-numbered performance states generally represent higher performance activity.
Running a high-VRAM test in Kubernetes
GPU nodes in Kubernetes should be tested inside the same container environment used by production workloads.
The following example creates a one-time Job.
Replace:
<gpu-node-name>
<namespace>
<pytorch-cuda-image>
with values appropriate for your environment.
apiVersion: batch/v1
kind: Job
metadata:
name: high-vram-gpu-test
namespace: gpu-lab
spec:
backoffLimit: 0
activeDeadlineSeconds: 600
template:
metadata:
labels:
test: high-vram-gpu
spec:
nodeName: gpu-worker
runtimeClassName: nvidia
restartPolicy: Never
containers:
- name: test
image: pytorch/pytorch:latest
imagePullPolicy: IfNotPresent
env:
- name: TARGET_VRAM_GIB
value: "8"
- name: TEST_DURATION_SECONDS
value: "180"
- name: SAFETY_MARGIN_GIB
value: "1"
command:
- /bin/bash
- -lc
args:
- |
cat > /tmp/high_vram_test.py <<'PY'
import os
import time
import torch
GIB = 1024**3
MIB = 1024**2
target_gib = float(
os.getenv("TARGET_VRAM_GIB", "8")
)
duration = int(
os.getenv("TEST_DURATION_SECONDS", "180")
)
safety_gib = float(
os.getenv("SAFETY_MARGIN_GIB", "1")
)
if not torch.cuda.is_available():
raise RuntimeError("CUDA is unavailable")
device = torch.device("cuda:0")
free_bytes, total_bytes = torch.cuda.mem_get_info()
target_bytes = int(target_gib * GIB)
safety_bytes = int(safety_gib * GIB)
print(
"GPU:",
torch.cuda.get_device_name(device),
flush=True,
)
print(
"Total VRAM GiB:",
round(total_bytes / GIB, 2),
flush=True,
)
print(
"Free VRAM GiB:",
round(free_bytes / GIB, 2),
flush=True,
)
if target_bytes + safety_bytes > free_bytes:
raise RuntimeError(
"Requested allocation leaves "
"insufficient safety margin"
)
allocations = []
chunk_bytes = 256 * MIB
allocated = 0
while allocated < target_bytes:
current = min(
chunk_bytes,
target_bytes - allocated,
)
tensor = torch.empty(
current // 2,
device=device,
dtype=torch.float16,
)
tensor.uniform_()
allocations.append(tensor)
torch.cuda.synchronize()
allocated += (
tensor.numel() *
tensor.element_size()
)
print(
"Allocated GiB:",
round(
torch.cuda.memory_allocated() /
GIB,
2,
),
flush=True,
)
a = torch.randn(
(4096, 4096),
device=device,
dtype=torch.float16,
)
b = torch.randn(
(4096, 4096),
device=device,
dtype=torch.float16,
)
end_time = time.time() + duration
iterations = 0
while time.time() < end_time:
result = a @ b
torch.cuda.synchronize()
iterations += 1
if iterations % 20 == 0:
print(
"Iterations:",
iterations,
"VRAM GiB:",
round(
torch.cuda.memory_allocated() /
GIB,
2,
),
flush=True,
)
print(
"High-VRAM test completed",
flush=True,
)
PY
python3 /tmp/high_vram_test.py
resources:
requests:
nvidia.com/gpu: 1
limits:
nvidia.com/gpu: 1
Create it:
kubectl apply -f high-vram-gpu-test.yaml
Follow the logs:
kubectl logs \
-n gpu-lab \
job/high-vram-gpu-test \
--follow
Check the Job result:
kubectl get job \
-n gpu-lab \
high-vram-gpu-test
A successful result should eventually show:
COMPLETIONS: 1/1
How to interpret the result
Result 1: The test passes
A passing test means:
- the requested amount of VRAM was allocated;
- the allocated memory was written;
- repeated CUDA computation completed;
- no visible CUDA error occurred;
- the GPU stayed available for the duration.
This weakens theories involving:
- immediate VRAM allocation failure;
- simple GPU overheating;
- obvious sustained-power instability;
- generic CUDA matrix-computation failure.
It does not eliminate application-specific problems.
Result 2: CUDA reports out of memory
An error such as:
CUDA out of memory
does not automatically indicate defective hardware.
Possible explanations include:
- the target was too close to total VRAM;
- another process was using the GPU;
- PyTorch reserved extra memory;
- computation tensors needed additional space;
- the CUDA context consumed more memory than expected.
Reduce the target:
TARGET_VRAM_GIB=6 python3 high_vram_test.py
Also check:
nvidia-smi
for other active processes.
Result 3: The process crashes with an NVIDIA Xid
Check the kernel journal:
journalctl -k --since "-10 minutes" |
grep -Ei 'NVRM|Xid|SXid'
An Xid may indicate:
- GPU memory error;
- driver fault;
- invalid application command;
- PCIe communication problem;
- GPU firmware issue;
- hardware instability.
Preserve the logs before rebooting.
Generate an NVIDIA diagnostic archive:
nvidia-bug-report.sh
Result 4: The entire machine reboots
A high-VRAM test should not normally reboot the entire computer.
If it does, investigate:
- power supply;
- GPU power cabling;
- motherboard BIOS;
- PCIe slot;
- GPU seating;
- system RAM;
- CPU stability;
- NVIDIA driver;
- kernel compatibility;
- motherboard power delivery;
- thermal protection.
After the machine returns, inspect the previous boot:
journalctl -k -b -1 --no-pager
Search for:
journalctl -k -b -1 --no-pager |
grep -Ei \
'NVRM|Xid|SXid|AER|PCIe|MCE|EDAC|watchdog|panic|oops|thermal|power'
Also inspect reboot history:
journalctl --list-boots
last -x
An anonymized diagnostic example
In one laboratory investigation, a modern GPU with approximately 16 GB of VRAM was being used for AI inference.
Basic CUDA tests passed. Sustained matrix multiplication also completed successfully while keeping several gigabytes allocated.
The real application later loaded approximately 7.56 GiB of model weights.
Telemetry showed the application eventually using approximately:
10,443 MiB VRAM
At that point, the GPU temperature was only about 39°C. Across the captured run, the highest recorded temperature was 52°C and the maximum power draw was approximately 74.7 W.
This produced an important diagnostic conclusion:
High VRAM occupancy alone did not explain the machine instability.
The GPU could hold a large allocation without overheating or approaching its power limit. Therefore, the investigation had to continue into other areas such as:
- application-specific GPU kernels;
- driver behavior;
- motherboard firmware;
- PCIe and interrupt handling;
- network and storage activity;
- combined system-level workload.
This illustrates why a synthetic test should be used to eliminate possibilities, not to declare the entire system healthy.
Common high-VRAM testing mistakes
Allocating memory without writing to it
Always initialize the tensors:
tensor.uniform_()
or:
tensor.zero_()
Writing data makes the test more meaningful.
Allocating all available memory
Do not target 100% of VRAM.
Leave room for CUDA context, temporary buffers and computation.
Running only for a few seconds
Some problems need time to appear.
A reasonable progression is:
30 seconds
2 minutes
5 minutes
15 minutes
Increase duration gradually.
Running multiple tests at once
Do not combine:
- GPU stress;
- CPU stress;
- network stress;
- disk stress;
during the first test.
Start with one variable. Combined tests can be added later.
Ignoring system logs
A process can fail while the useful explanation appears only in:
journalctl -k
Always capture both application output and kernel logs.
Treating a pass as proof of perfect hardware
A synthetic test exercises a controlled pattern. A real AI application may use completely different CUDA kernels and memory-access patterns.
Recommended testing sequence
For reliable troubleshooting, use the following order:
- Confirm
nvidia-smiworks. - Run a minimal CUDA calculation.
- Run a low-VRAM sustained-compute test.
- Run a high-VRAM allocation test.
- Run high VRAM plus sustained computation.
- Monitor temperature, power and utilization.
- Check kernel logs.
- Run the real AI workload.
- Compare the synthetic and real-workload results.
This progression helps identify exactly which stage introduces instability.
Frequently asked questions
Is a high-VRAM test dangerous?
A properly designed software test should not physically damage a healthy GPU.
The GPU includes thermal and power protections.
However, the test may:
- produce significant heat;
- consume substantial power;
- trigger an application OOM;
- expose an existing hardware or firmware problem;
- crash an already unstable machine.
Monitor the system and begin with conservative targets.
How much VRAM should I test?
A reasonable starting target is approximately 50–70% of total VRAM.
For a 16 GB GPU:
Initial test: 6–8 GB
Higher test: 10–12 GB
Leave sufficient free memory for computation.
How long should the test run?
Start with 2–3 minutes.
For greater confidence, extend it to:
10 minutes
30 minutes
1 hour
Long tests are more useful after shorter tests pass.
Why does nvidia-smi show more memory than PyTorch?
nvidia-smi includes:
- CUDA context;
- driver allocations;
- library overhead;
- non-PyTorch allocations.
PyTorch primarily reports memory managed by its own allocator.
Can bad VRAM cause a complete reboot?
It is possible, but many memory problems produce:
- CUDA errors;
- application crashes;
- NVIDIA Xid messages;
- GPU resets.
A full machine reboot also raises suspicion about the PSU, motherboard, PCIe, BIOS or wider platform stability.
Does passing the test prove an AI model will work?
No.
It proves that the tested allocation and computation pattern worked.
The AI application may still use:
- different kernels;
- compilation;
- graph capture;
- attention backends;
- larger temporary buffers;
- network and storage activity.
Final conclusion
A high-VRAM GPU test is one of the most useful steps when diagnosing AI workloads that fail during model loading or initialization.
A good test should:
- allocate memory gradually;
- write data into every allocation;
- retain the allocation;
- run sustained CUDA computation;
- record temperature, power and memory use;
- preserve kernel logs.
The most important lesson is how to interpret the result.
A failed test may point toward:
- insufficient memory;
- GPU instability;
- driver problems;
- thermal or power issues;
- PCIe or platform faults.
A successful test shows that large generic CUDA allocations are stable, but it does not prove that every AI framework or GPU kernel will behave correctly.
Use the high-VRAM test as one part of a structured diagnostic process—not as the final verdict.
