A graphics card can pass a basic test and still fail when a real artificial intelligence workload starts.

A small CUDA calculation may use only a few hundred megabytes of GPU memory. A large language model, image generator or scientific application may use 8 GB, 12 GB or nearly all available VRAM.

That difference matters.

A high-VRAM GPU test helps answer a specific diagnostic question:

Can the GPU allocate, write to and retain a large amount of video memory without errors, crashes or system resets?

This article explains what VRAM is, why high-memory tests are useful, how to perform one safely and how to interpret the results correctly.

What is VRAM?

VRAM stands for Video Random Access Memory.

It is the memory physically attached to the GPU. Applications use it to store:

• neural-network weights;
• tensors;
• textures;
• image data;
• model activations;
• attention caches;
• temporary computation buffers;
• compiled GPU kernels.

System RAM and VRAM are different.

System RAM is primarily used by the CPU. VRAM is used directly by the GPU and is designed to provide very high bandwidth.

A GPU may have:

8 GB VRAM
12 GB VRAM
16 GB VRAM
24 GB VRAM
48 GB VRAM

For AI inference, VRAM capacity is often one of the most important limitations.

Why run a high-VRAM GPU test?

A normal GPU test might perform one matrix multiplication and exit successfully.

That proves only that:

• the GPU is visible;
• CUDA works;
• the driver can execute a basic operation;
• a small amount of VRAM can be used.

It does not prove that:

• most of the VRAM is usable;
• large allocations are stable;
• the GPU remains stable for several minutes;
• memory errors appear only at certain addresses;
• power and temperature remain controlled;
• the driver handles sustained allocations correctly.

A high-VRAM test is useful when a machine experiences:

• crashes while loading an AI model;
• system resets during model initialization;
• CUDA out of memory errors;
• unexplained container termination;
• NVIDIA Xid errors;
• GPU disappearance;
• application crashes after allocating several gigabytes;
• instability that does not appear in small tests.

What a high-VRAM test actually checks

A useful high-VRAM GPU test should do more than call an allocation function.

It should perform three stages.

Stage 1: Allocate memory

The program asks the GPU driver to reserve several gigabytes of VRAM.

This tests whether a large allocation is possible.

Stage 2: Touch the allocated memory

The program writes real values into the allocated tensors.

This is important because merely creating an empty tensor does not exercise the memory as thoroughly as writing data throughout it.

Stage 3: Compute while the memory remains allocated

The test performs repeated GPU calculations without releasing the large allocation.

This creates a more realistic workload involving:

• high memory occupancy;
• CUDA kernel execution;
• memory reads and writes;
• synchronization;
• sustained GPU activity.

What the test does not prove

Passing a high-VRAM test does not prove that the entire system is perfect.

It does not fully test:

• a specific AI framework;
• model-loading code;
• CUDA graph capture;
• FlashAttention;
• JIT compilation;
• distributed communication;
• network storage;
• container networking;
• PCIe edge cases;
• application-specific kernels;
• GPU firmware interactions.

A passing test means:

Large VRAM allocation and generic CUDA computation appear stable under the tested conditions.

That is valuable evidence, but it is not a complete hardware certification.

Preparing the system

Before starting, verify that the GPU is visible:

nvidia-smi

You should see information such as:

GPU name
Driver version
Memory usage
Temperature
Power usage

Verify CUDA through PyTorch:

python3 - <<'PY'
import torch

print("CUDA available:", torch.cuda.is_available())

if torch.cuda.is_available():
    print("GPU:", torch.cuda.get_device_name(0))
    print("CUDA runtime:", torch.version.cuda)
PY

A healthy result should show:

CUDA available: True
GPU: NVIDIA ...

Stop other GPU workloads

A high-VRAM test should normally run on an otherwise idle GPU.

Check active processes:

nvidia-smi

Look at the Processes section.

If another model server or application is using several gigabytes of VRAM, stop it before testing. Otherwise, the test may fail simply because insufficient memory is available.

In Kubernetes, cordon the test node if necessary:

kubectl cordon <gpu-node>

Also scale down GPU workloads:

kubectl scale deployment <gpu-workload> \
  --replicas=0 \
  -n <namespace>

A safe PyTorch high-VRAM test

The following script allocates GPU memory gradually in 256 MiB chunks, writes data into each chunk and then runs matrix multiplications while the memory stays allocated.

Save it as:

high_vram_test.py

import os
import sys
import time

import torch


GIB = 1024**3
MIB = 1024**2

target_gib = float(os.getenv("TARGET_VRAM_GIB", "8"))
duration_seconds = int(os.getenv("TEST_DURATION_SECONDS", "180"))
chunk_mib = int(os.getenv("CHUNK_SIZE_MIB", "256"))
safety_margin_gib = float(os.getenv("SAFETY_MARGIN_GIB", "1"))


def gib(value: int) -> float:
    return value / GIB


def print_memory(label: str) -> None:
    free_bytes, total_bytes = torch.cuda.mem_get_info()
    allocated = torch.cuda.memory_allocated()
    reserved = torch.cuda.memory_reserved()

    print(
        f"{label}: "
        f"allocated={gib(allocated):.2f} GiB, "
        f"reserved={gib(reserved):.2f} GiB, "
        f"free={gib(free_bytes):.2f} GiB, "
        f"total={gib(total_bytes):.2f} GiB",
        flush=True,
    )


if not torch.cuda.is_available():
    raise RuntimeError("CUDA is unavailable")

device = torch.device("cuda:0")

print("GPU:", torch.cuda.get_device_name(device), flush=True)
print("PyTorch CUDA:", torch.version.cuda, flush=True)

free_bytes, total_bytes = torch.cuda.mem_get_info()

target_bytes = int(target_gib * GIB)
safety_bytes = int(safety_margin_gib * GIB)

print_memory("Initial state")

if target_bytes + safety_bytes > free_bytes:
    safe_target = max(0, free_bytes - safety_bytes)

    raise RuntimeError(
        f"Requested target is too large. "
        f"Requested={target_gib:.2f} GiB, "
        f"safe available target={gib(safe_target):.2f} GiB"
    )

chunk_bytes = chunk_mib * MIB
elements_per_chunk = chunk_bytes // 2  # float16 uses 2 bytes

allocations: list[torch.Tensor] = []

try:
    print(
        f"Allocating approximately {target_gib:.2f} GiB "
        f"in {chunk_mib} MiB chunks",
        flush=True,
    )

    allocated_bytes = 0
    chunk_number = 0

    while allocated_bytes < target_bytes:
        remaining = target_bytes - allocated_bytes
        current_chunk_bytes = min(chunk_bytes, remaining)
        current_elements = current_chunk_bytes // 2

        tensor = torch.empty(
            current_elements,
            device=device,
            dtype=torch.float16,
        )

        # Writing data ensures that the allocation is actually exercised.
        tensor.uniform_()

        allocations.append(tensor)

        torch.cuda.synchronize()

        allocated_bytes += tensor.numel() * tensor.element_size()
        chunk_number += 1

        print_memory(f"After chunk {chunk_number}")

    print("Target allocation completed", flush=True)

    print("Creating computation tensors", flush=True)

    matrix_a = torch.randn(
        (4096, 4096),
        device=device,
        dtype=torch.float16,
    )

    matrix_b = torch.randn(
        (4096, 4096),
        device=device,
        dtype=torch.float16,
    )

    torch.cuda.synchronize()

    print_memory("Before computation")

    end_time = time.time() + duration_seconds
    iterations = 0

    print(
        f"Starting sustained computation for "
        f"{duration_seconds} seconds",
        flush=True,
    )

    while time.time() < end_time:
        result = matrix_a @ matrix_b
        torch.cuda.synchronize()

        iterations += 1

        if iterations % 20 == 0:
            print(
                f"Iterations completed: {iterations}",
                flush=True,
            )
            print_memory("Current state")

    print("High-VRAM GPU test completed successfully", flush=True)
    print("Total iterations:", iterations, flush=True)
    print_memory("Final test state")

except torch.OutOfMemoryError as exc:
    print("CUDA out-of-memory error:", exc, file=sys.stderr)
    print_memory("State at failure")
    raise

finally:
    allocations.clear()

    if "matrix_a" in locals():
        del matrix_a

    if "matrix_b" in locals():
        del matrix_b

    if "result" in locals():
        del result

    torch.cuda.empty_cache()
    torch.cuda.synchronize()

    print_memory("After cleanup")

Running the test

For an 8 GiB allocation and a three-minute computation phase:

TARGET_VRAM_GIB=8 \
TEST_DURATION_SECONDS=180 \
python3 high_vram_test.py

For a smaller test:

TARGET_VRAM_GIB=4 \
TEST_DURATION_SECONDS=120 \
python3 high_vram_test.py

For a larger GPU:

TARGET_VRAM_GIB=16 \
TEST_DURATION_SECONDS=300 \
python3 high_vram_test.py

Never target 100% of the available VRAM.

Leave at least 1–2 GiB free for:

• the CUDA context;
• computation tensors;
• driver overhead;
• PyTorch memory reservations;
• temporary kernel buffers.

Monitoring the GPU during the test

Open another terminal and run:

watch -n 1 nvidia-smi

This gives a quick live view.

For a more useful diagnostic record, capture telemetry to a CSV file:

mkdir -p ~/gpu-diagnostics

LOG="$HOME/gpu-diagnostics/high-vram-$(date +%Y%m%d-%H%M%S).csv"

nvidia-smi \
  --query-gpu=timestamp,name,pstate,temperature.gpu,power.draw,power.limit,utilization.gpu,memory.used,memory.total,clocks.sm \
  --format=csv \
  -l 1 \
  > "$LOG"

Run this in a separate terminal, or place it in the background:

nohup nvidia-smi \
  --query-gpu=timestamp,pstate,temperature.gpu,power.draw,power.limit,utilization.gpu,memory.used,memory.total,clocks.sm \
  --format=csv \
  -l 1 \
  > "$LOG" 2>&1 &

echo $! > /tmp/gpu-monitor.pid

Stop it afterward:

kill "$(cat /tmp/gpu-monitor.pid)"

Inspect the end of the file:

tail -30 "$LOG"

Important values to monitor

Memory used

This should rise toward the requested target.

For example:

8200 MiB / 16384 MiB

PyTorch and nvidia-smi may show slightly different values.

PyTorch reports memory managed by its allocator. nvidia-smi reports memory associated with the entire process, including CUDA context and driver overhead.

A small difference is normal.

GPU utilization

During allocation, utilization may be low or variable.

During matrix multiplication, it may rise close to:

90–100%

Temperature

Temperature depends on:

• GPU model;
• cooling;
• room temperature;
• fan curve;
• case airflow.

A rising temperature is normal under load. A rapidly increasing or unusually high temperature should be investigated.

Power draw

Power draw may increase significantly during computation.

High memory occupancy alone does not necessarily use much power. Computation usually consumes more power than simply keeping tensors allocated.

P-state

You may see:

P8
P2
P1
P0

Lower-numbered performance states generally represent higher performance activity.

Running a high-VRAM test in Kubernetes

GPU nodes in Kubernetes should be tested inside the same container environment used by production workloads.

The following example creates a one-time Job.

Replace:

<gpu-node-name>
<namespace>
<pytorch-cuda-image>

with values appropriate for your environment.

apiVersion: batch/v1
kind: Job
metadata:
  name: high-vram-gpu-test
  namespace: gpu-lab

spec:
  backoffLimit: 0
  activeDeadlineSeconds: 600

  template:
    metadata:
      labels:
        test: high-vram-gpu

    spec:
      nodeName: gpu-worker
      runtimeClassName: nvidia
      restartPolicy: Never

      containers:
        - name: test
          image: pytorch/pytorch:latest
          imagePullPolicy: IfNotPresent

          env:
            - name: TARGET_VRAM_GIB
              value: "8"

            - name: TEST_DURATION_SECONDS
              value: "180"

            - name: SAFETY_MARGIN_GIB
              value: "1"

          command:
            - /bin/bash
            - -lc

          args:
            - |
              cat > /tmp/high_vram_test.py <<'PY'
              import os
              import time
              import torch

              GIB = 1024**3
              MIB = 1024**2

              target_gib = float(
                  os.getenv("TARGET_VRAM_GIB", "8")
              )

              duration = int(
                  os.getenv("TEST_DURATION_SECONDS", "180")
              )

              safety_gib = float(
                  os.getenv("SAFETY_MARGIN_GIB", "1")
              )

              if not torch.cuda.is_available():
                  raise RuntimeError("CUDA is unavailable")

              device = torch.device("cuda:0")

              free_bytes, total_bytes = torch.cuda.mem_get_info()

              target_bytes = int(target_gib * GIB)
              safety_bytes = int(safety_gib * GIB)

              print(
                  "GPU:",
                  torch.cuda.get_device_name(device),
                  flush=True,
              )

              print(
                  "Total VRAM GiB:",
                  round(total_bytes / GIB, 2),
                  flush=True,
              )

              print(
                  "Free VRAM GiB:",
                  round(free_bytes / GIB, 2),
                  flush=True,
              )

              if target_bytes + safety_bytes > free_bytes:
                  raise RuntimeError(
                      "Requested allocation leaves "
                      "insufficient safety margin"
                  )

              allocations = []
              chunk_bytes = 256 * MIB
              allocated = 0

              while allocated < target_bytes:
                  current = min(
                      chunk_bytes,
                      target_bytes - allocated,
                  )

                  tensor = torch.empty(
                      current // 2,
                      device=device,
                      dtype=torch.float16,
                  )

                  tensor.uniform_()
                  allocations.append(tensor)

                  torch.cuda.synchronize()

                  allocated += (
                      tensor.numel() *
                      tensor.element_size()
                  )

                  print(
                      "Allocated GiB:",
                      round(
                          torch.cuda.memory_allocated() /
                          GIB,
                          2,
                      ),
                      flush=True,
                  )

              a = torch.randn(
                  (4096, 4096),
                  device=device,
                  dtype=torch.float16,
              )

              b = torch.randn(
                  (4096, 4096),
                  device=device,
                  dtype=torch.float16,
              )

              end_time = time.time() + duration
              iterations = 0

              while time.time() < end_time:
                  result = a @ b
                  torch.cuda.synchronize()
                  iterations += 1

                  if iterations % 20 == 0:
                      print(
                          "Iterations:",
                          iterations,
                          "VRAM GiB:",
                          round(
                              torch.cuda.memory_allocated() /
                              GIB,
                              2,
                          ),
                          flush=True,
                      )

              print(
                  "High-VRAM test completed",
                  flush=True,
              )
              PY

              python3 /tmp/high_vram_test.py

          resources:
            requests:
              nvidia.com/gpu: 1

            limits:
              nvidia.com/gpu: 1

Create it:

kubectl apply -f high-vram-gpu-test.yaml

Follow the logs:

kubectl logs \
  -n gpu-lab \
  job/high-vram-gpu-test \
  --follow

Check the Job result:

kubectl get job \
  -n gpu-lab \
  high-vram-gpu-test

A successful result should eventually show:

COMPLETIONS: 1/1

How to interpret the result

Result 1: The test passes

A passing test means:

• the requested amount of VRAM was allocated;
• the allocated memory was written;
• repeated CUDA computation completed;
• no visible CUDA error occurred;
• the GPU stayed available for the duration.

This weakens theories involving:

• immediate VRAM allocation failure;
• simple GPU overheating;
• obvious sustained-power instability;
• generic CUDA matrix-computation failure.

It does not eliminate application-specific problems.

Result 2: CUDA reports out of memory

An error such as:

CUDA out of memory

does not automatically indicate defective hardware.

Possible explanations include:

• the target was too close to total VRAM;
• another process was using the GPU;
• PyTorch reserved extra memory;
• computation tensors needed additional space;
• the CUDA context consumed more memory than expected.

Reduce the target:

TARGET_VRAM_GIB=6 python3 high_vram_test.py

Also check:

nvidia-smi

for other active processes.

Result 3: The process crashes with an NVIDIA Xid

Check the kernel journal:

journalctl -k --since "-10 minutes" |
grep -Ei 'NVRM|Xid|SXid'

An Xid may indicate:

• GPU memory error;
• driver fault;
• invalid application command;
• PCIe communication problem;
• GPU firmware issue;
• hardware instability.

Preserve the logs before rebooting.

Generate an NVIDIA diagnostic archive:

nvidia-bug-report.sh

Result 4: The entire machine reboots

A high-VRAM test should not normally reboot the entire computer.

If it does, investigate:

• power supply;
• GPU power cabling;
• motherboard BIOS;
• PCIe slot;
• GPU seating;
• system RAM;
• CPU stability;
• NVIDIA driver;
• kernel compatibility;
• motherboard power delivery;
• thermal protection.

After the machine returns, inspect the previous boot:

journalctl -k -b -1 --no-pager

Search for:

journalctl -k -b -1 --no-pager |
grep -Ei \
'NVRM|Xid|SXid|AER|PCIe|MCE|EDAC|watchdog|panic|oops|thermal|power'

Also inspect reboot history:

journalctl --list-boots
last -x

An anonymized diagnostic example

In one laboratory investigation, a modern GPU with approximately 16 GB of VRAM was being used for AI inference.

Basic CUDA tests passed. Sustained matrix multiplication also completed successfully while keeping several gigabytes allocated.

The real application later loaded approximately 7.56 GiB of model weights.

Telemetry showed the application eventually using approximately:

10,443 MiB VRAM

At that point, the GPU temperature was only about 39°C. Across the captured run, the highest recorded temperature was 52°C and the maximum power draw was approximately 74.7 W.

This produced an important diagnostic conclusion:

High VRAM occupancy alone did not explain the machine instability.

The GPU could hold a large allocation without overheating or approaching its power limit. Therefore, the investigation had to continue into other areas such as:

• application-specific GPU kernels;
• driver behavior;
• motherboard firmware;
• PCIe and interrupt handling;
• network and storage activity;
• combined system-level workload.

This illustrates why a synthetic test should be used to eliminate possibilities, not to declare the entire system healthy.

Common high-VRAM testing mistakes

Allocating memory without writing to it

Always initialize the tensors:

tensor.uniform_()

or:

tensor.zero_()

Writing data makes the test more meaningful.

Allocating all available memory

Do not target 100% of VRAM.

Leave room for CUDA context, temporary buffers and computation.

Running only for a few seconds

Some problems need time to appear.

A reasonable progression is:

30 seconds
2 minutes
5 minutes
15 minutes

Increase duration gradually.

Running multiple tests at once

Do not combine:

• GPU stress;
• CPU stress;
• network stress;
• disk stress;

during the first test.

Start with one variable. Combined tests can be added later.

Ignoring system logs

A process can fail while the useful explanation appears only in:

journalctl -k

Always capture both application output and kernel logs.

Treating a pass as proof of perfect hardware

A synthetic test exercises a controlled pattern. A real AI application may use completely different CUDA kernels and memory-access patterns.

Recommended testing sequence

For reliable troubleshooting, use the following order:

1. Confirm nvidia-smi works.
2. Run a minimal CUDA calculation.
3. Run a low-VRAM sustained-compute test.
4. Run a high-VRAM allocation test.
5. Run high VRAM plus sustained computation.
6. Monitor temperature, power and utilization.
7. Check kernel logs.
8. Run the real AI workload.
9. Compare the synthetic and real-workload results.

This progression helps identify exactly which stage introduces instability.

Frequently asked questions

Is a high-VRAM test dangerous?

A properly designed software test should not physically damage a healthy GPU.

The GPU includes thermal and power protections.

However, the test may:

• produce significant heat;
• consume substantial power;
• trigger an application OOM;
• expose an existing hardware or firmware problem;
• crash an already unstable machine.

Monitor the system and begin with conservative targets.

How much VRAM should I test?

A reasonable starting target is approximately 50–70% of total VRAM.

For a 16 GB GPU:

Initial test: 6–8 GB
Higher test: 10–12 GB

Leave sufficient free memory for computation.

How long should the test run?

Start with 2–3 minutes.

For greater confidence, extend it to:

10 minutes
30 minutes
1 hour

Long tests are more useful after shorter tests pass.

Why does nvidia-smi show more memory than PyTorch?

nvidia-smi includes:

• CUDA context;
• driver allocations;
• library overhead;
• non-PyTorch allocations.

PyTorch primarily reports memory managed by its own allocator.

Can bad VRAM cause a complete reboot?

It is possible, but many memory problems produce:

• CUDA errors;
• application crashes;
• NVIDIA Xid messages;
• GPU resets.

A full machine reboot also raises suspicion about the PSU, motherboard, PCIe, BIOS or wider platform stability.

Does passing the test prove an AI model will work?

No.

It proves that the tested allocation and computation pattern worked.

The AI application may still use:

• different kernels;
• compilation;
• graph capture;
• attention backends;
• larger temporary buffers;
• network and storage activity.

Final conclusion

A high-VRAM GPU test is one of the most useful steps when diagnosing AI workloads that fail during model loading or initialization.

A good test should:

• allocate memory gradually;
• write data into every allocation;
• retain the allocation;
• run sustained CUDA computation;
• record temperature, power and memory use;
• preserve kernel logs.

The most important lesson is how to interpret the result.

A failed test may point toward:

• insufficient memory;
• GPU instability;
• driver problems;
• thermal or power issues;
• PCIe or platform faults.

A successful test shows that large generic CUDA allocations are stable, but it does not prove that every AI framework or GPU kernel will behave correctly.

Use the high-VRAM test as one part of a structured diagnostic process—not as the final verdict.