In Stage 0, we proved that the K3s cluster could run NVIDIA GPU containers.

We verified the Linux driver, NVIDIA runtime, Kubernetes RuntimeClass, NVIDIA device plugin, GPU allocation and a CUDA smoke-test pod.

The platform is now ready for an actual AI workload.

In Stage 1, we will:

• create a dedicated Kubernetes namespace;
• create persistent storage for model files;
• deploy a small instruction-tuned language model with vLLM;
• expose an OpenAI-compatible API;
• send the first prompts to the model;
• build a simple FastAPI application;
• add a browser-based chat interface;
• containerize the application;
• automate deployment with GitLab CI/CD;
• expose the application through DNS and Kubernetes Ingress.

This is Stage 1 of our local AI learning project.

Before continuing, complete Stage 0: Preparing a K3s Cluster with NVIDIA GPUs for Local AI.

Stage 0 should end with a successful GPU smoke test inside Kubernetes.

What We Are Building

The finished Stage 1 architecture looks like this:

User browser
     |
     v
DNS hostname
     |
     v
NGINX Ingress
     |
     v
FastAPI Kubernetes Service
     |
     v
FastAPI application pod
     |
     v
vLLM Kubernetes Service
     |
     v
vLLM model server
     |
     v
NVIDIA GPU

The browser does not communicate directly with the language model.

Instead:

1. the browser sends a message to FastAPI;
2. FastAPI validates the request;
3. FastAPI adds conversation history and application instructions;
4. FastAPI sends an OpenAI-compatible request to vLLM;
5. vLLM runs the language model on the GPU;
6. the generated answer returns to the browser.

This separation will become important in later stages when we add documents, embeddings, RAG, authentication and AI agents.

The Anonymized Lab Environment

The example lab contains:

control-plane-01
control-plane-02
gpu-worker-01
gpu-worker-02

The model server is scheduled on one GPU worker.

The application uses:

• K3s;
• an NVIDIA GPU;
• the NVIDIA RuntimeClass;
• Longhorn persistent storage;
• vLLM;
• the Qwen3-4B language model;
• FastAPI;
• Uvicorn;
• a simple HTML and JavaScript interface;
• Podman;
• GitLab CI/CD;
• a private container registry;
• NGINX Ingress Controller;
• internal DNS.

All real hostnames, domains, addresses, registry names, credentials and organization-specific values have been replaced with generic examples.

Prerequisites

This guide assumes that:

• Stage 0 is complete;
• all required Kubernetes nodes are Ready;
• the GPU worker advertises nvidia.com/gpu;
• runtimeClassName: nvidia works;
• kubectl can connect to the cluster;
• a Kubernetes storage class is available;
• an NGINX Ingress Controller is installed;
• a container registry is available for the application image;
• GitLab Runner can deploy to the cluster.

Check the nodes:

kubectl get nodes -o wide

Confirm that the selected GPU worker still advertises a GPU:

kubectl get nodes `
  -o custom-columns="NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

Example:

NAME               GPU
gpu-worker-01      1
gpu-worker-02      1
control-plane-01   <none>
control-plane-02   <none>

Do not repeat the full Stage 0 diagnostics unless one of these checks fails.

A Note About the Example Versions

The project was tested with specific versions of vLLM, Python, FastAPI and other dependencies.

Software versions change over time, so treat the versions in this article as tested examples rather than permanent recommendations.

For a reproducible deployment:

• pin the container image version;
• pin Python dependencies;
• test upgrades in a separate branch;
• avoid relying only on an unversioned latest tag.

Step 1: Create a Dedicated Namespace

Create a namespace for the AI project:

kubectl create namespace ai-lab

If it already exists, Kubernetes reports:

Error from server (AlreadyExists)

That is harmless.

Verify it:

kubectl get namespace ai-lab

Expected status:

NAME      STATUS
ai-lab    Active

A separate namespace helps organize:

• model deployments;
• application deployments;
• services;
• persistent storage;
• secrets;
• Ingress resources;
• future RAG components.

For this article, commands explicitly use:

-n ai-lab

This avoids accidentally creating resources in the wrong namespace.

Step 2: Create Persistent Model Storage

Model files can occupy several gigabytes.

Without persistent storage, Kubernetes may download the model again whenever the pod is recreated on a clean node.

Create a file called:

model-cache-pvc.yaml

Use an anonymized manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache
  namespace: ai-lab
spec:
  accessModes:
    - ReadWriteOnce

  storageClassName: longhorn

  resources:
    requests:
      storage: 30Gi

Replace:

longhorn

with the storage class available in your cluster.

List storage classes:

kubectl get storageclass

Apply the claim:

kubectl apply -f .\model-cache-pvc.yaml

Check it:

kubectl get pvc -n ai-lab

A healthy result should eventually show:

NAME          STATUS   CAPACITY
model-cache   Bound    30Gi

What does Bound mean?

Bound means Kubernetes connected the PersistentVolumeClaim to a storage volume.

It does not yet prove that a pod can successfully mount and use it.

That will happen when the model pod starts.

Step 3: Understand What vLLM Does

vLLM is the model-serving layer.

Its main responsibilities are:

• loading the model;
• moving model weights into GPU memory;
• managing the attention cache;
• processing prompts;
• generating tokens;
• serving requests through HTTP.

A major advantage of vLLM is its OpenAI-compatible API.

Instead of inventing a custom request format, our application can send requests to endpoints such as:

/v1/models
/v1/chat/completions

The application can therefore use the OpenAI Python client while communicating with a model hosted entirely inside the local cluster.

No OpenAI cloud API is required for this configuration.

Step 4: Create the vLLM Deployment

Create:

vllm-deployment.yaml

Example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-model
  namespace: ai-lab
spec:
  replicas: 1

  selector:
    matchLabels:
      app: vllm-model

  template:
    metadata:
      labels:
        app: vllm-model

    spec:
      runtimeClassName: nvidia

      nodeSelector:
        kubernetes.io/hostname: gpu-worker-02

      containers:
        - name: vllm
          image: vllm/vllm-openai:TESTED_VERSION
          imagePullPolicy: IfNotPresent

          args:
            - Qwen/Qwen3-4B
            - --served-model-name
            - qwen3-4b
            - --host
            - 0.0.0.0
            - --port
            - "8000"
            - --dtype
            - bfloat16
            - --max-model-len
            - "8192"
            - --gpu-memory-utilization
            - "0.85"
            - --enable-prefix-caching
            - --generation-config
            - vllm

          env:
            - name: HF_HOME
              value: /models/huggingface

            - name: TRANSFORMERS_CACHE
              value: /models/huggingface

            - name: PYTHONUNBUFFERED
              value: "1"

          ports:
            - name: http
              containerPort: 8000

          resources:
            requests:
              cpu: "4"
              memory: 12Gi
              nvidia.com/gpu: "1"

            limits:
              cpu: "12"
              memory: 24Gi
              nvidia.com/gpu: "1"

          startupProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 180

          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 6

          livenessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 30
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 3

          volumeMounts:
            - name: model-cache
              mountPath: /models

            - name: shared-memory
              mountPath: /dev/shm

      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: model-cache

        - name: shared-memory
          emptyDir:
            medium: Memory
            sizeLimit: 8Gi

Replace:

TESTED_VERSION
gpu-worker-02

with values appropriate for your cluster.

Understanding the Important vLLM Settings

Select the NVIDIA runtime

runtimeClassName: nvidia

This tells K3s to start the pod through the NVIDIA container runtime configured in Stage 0.

Select a GPU worker

nodeSelector:
  kubernetes.io/hostname: gpu-worker-02

This forces the model onto a specific GPU node.

For a larger environment, labels, affinities and taints are usually more flexible than hard-coded hostnames.

Reserve one GPU

resources:
  limits:
    nvidia.com/gpu: "1"

Kubernetes assigns one complete GPU to the container.

Set the public model name

--served-model-name
qwen3-4b

The application will use:

qwen3-4b

in API requests.

This is separate from the model repository name:

Qwen/Qwen3-4B

Limit the context length

--max-model-len
8192

The context length includes:

• the system prompt;
• previous messages;
• the new user message;
• generated output.

A larger context length requires more GPU memory.

Control GPU memory reservation

--gpu-memory-utilization
0.85

This allows vLLM to use approximately 85% of available GPU memory.

Do not automatically set this to 1.0. The CUDA runtime and model server also require memory overhead.

Enable prefix caching

--enable-prefix-caching

Prefix caching can improve repeated requests that share the same beginning, such as a common system prompt.

Step 5: Deploy the Model

Apply the manifest:

kubectl apply -f .\vllm-deployment.yaml

Check the pod:

kubectl get pods -n ai-lab -o wide

The first startup may remain in:

ContainerCreating
Running
0/1 Running

for several minutes.

This can be normal.

The model server may need to:

1. attach the persistent volume;
2. download model metadata;
3. download several checkpoint files;
4. load the model into RAM;
5. transfer weights to the GPU;
6. compile optimized operations;
7. allocate its key-value cache;
8. perform warm-up operations;
9. start the HTTP server.

Step 6: Watch the Model Logs

Find the pod:

$pod = kubectl get pods `
  -n ai-lab `
  -l app=vllm-model `
  -o jsonpath="{.items[0].metadata.name}"

Display its name:

Write-Host "Pod: $pod"

Follow the logs:

kubectl logs `
  -n ai-lab `
  $pod `
  --tail=200 `
  -f

During a successful startup, you may see messages about:

• resolving the model architecture;
• loading checkpoint shards;
• GPU memory usage;
• compiling graphs;
• creating the key-value cache;
• capturing CUDA graphs;
• starting the API server.

The critical final message is similar to:

Starting vLLM server on http://0.0.0.0:8000

Press:

Ctrl+C

to stop following the logs. This does not stop the pod.

Check readiness:

kubectl get pods -n ai-lab

The final state should be:

READY   STATUS
1/1     Running

Step 7: Create the vLLM Service

The pod IP can change whenever Kubernetes recreates the pod.

A Service provides a stable internal address.

Create:

vllm-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: vllm-model
  namespace: ai-lab
spec:
  type: ClusterIP

  selector:
    app: vllm-model

  ports:
    - name: http
      port: 8000
      targetPort: http

Apply it:

kubectl apply -f .\vllm-service.yaml

Check it:

kubectl get service vllm-model `
  -n ai-lab `
  -o wide

Inside the same namespace, applications can reach the model at:

http://vllm-model:8000

The full Kubernetes DNS name is:

http://vllm-model.ai-lab.svc.cluster.local:8000

Step 8: Test the OpenAI-Compatible API

For the first test, forward the internal service to the workstation:

kubectl port-forward `
  -n ai-lab `
  service/vllm-model `
  8000:8000

Keep this PowerShell window open.

Check the health endpoint

From another PowerShell window:

Invoke-RestMethod `
  -Uri "http://127.0.0.1:8000/health"

A healthy request may return an empty successful response.

List available models

Invoke-RestMethod `
  -Uri "http://127.0.0.1:8000/v1/models" |
  ConvertTo-Json -Depth 10

The result should include:

qwen3-4b

Send the first chat prompt

$body = @{
    model = "qwen3-4b"

    messages = @(
        @{
            role    = "system"
            content = "You are a helpful assistant."
        },
        @{
            role    = "user"
            content = "Explain what Kubernetes is in simple terms."
        }
    )

    temperature = 0.2
    max_tokens  = 300
} | ConvertTo-Json -Depth 10

Send it:

$response = Invoke-RestMethod `
  -Method Post `
  -Uri "http://127.0.0.1:8000/v1/chat/completions" `
  -ContentType "application/json" `
  -Body $body

Display the answer:

$response.choices[0].message.content

At this point, Stage 1 has already delivered the main promise made at the end of Stage 0:

The K3s cluster is serving a real language model through an OpenAI-compatible API.

The next steps turn that API into an actual application.

Step 9: Create the FastAPI Project

Create a local project directory:

basic-gen-ai

Suggested structure:

basic-gen-ai/
├── app/
│   ├── __init__.py
│   ├── main.py
│   └── static/
│       ├── index.html
│       ├── app.js
│       └── style.css
├── k8s/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── ingress.yaml
├── ci/
├── requirements.txt
├── Dockerfile
├── .dockerignore
├── .gitignore
└── .gitlab-ci.yml

Create a virtual environment:

python -m venv .venv

Activate it:

Set-ExecutionPolicy `
  -Scope Process `
  -ExecutionPolicy Bypass

.\.venv\Scripts\Activate.ps1

Install the main dependencies:

python -m pip install `
  fastapi `
  "uvicorn[standard]" `
  openai

Create requirements.txt with pinned versions that you have tested.

Example:

fastapi==TESTED_VERSION
openai==TESTED_VERSION
uvicorn[standard]==TESTED_VERSION

Step 10: Build the FastAPI Backend

The backend needs four main endpoints:

GET  /
GET  /api/live
GET  /api/health
POST /api/chat

/

Serves the browser interface.

/api/live

Confirms that FastAPI itself is running.

It should not depend on vLLM.

/api/health

Checks whether:

• FastAPI is running;
• vLLM is reachable;
• the configured model exists.

/api/chat

Accepts:

• the user message;
• previous conversation messages;
• temperature;
• maximum output tokens.

It then calls vLLM.

A simplified OpenAI client configuration is:

import os

from openai import AsyncOpenAI

VLLM_BASE_URL = os.getenv(
    "VLLM_BASE_URL",
    "http://127.0.0.1:8000/v1",
)

VLLM_MODEL = os.getenv(
    "VLLM_MODEL",
    "qwen3-4b",
)

client = AsyncOpenAI(
    base_url=VLLM_BASE_URL,
    api_key="not-required",
    timeout=180.0,
)

The API key value is a placeholder required by the client library.

The local vLLM deployment in this lab does not validate it.

A chat request can then use:

response = await client.chat.completions.create(
    model=VLLM_MODEL,
    messages=messages,
    temperature=request.temperature,
    max_tokens=request.max_tokens,
)

Why Use FastAPI Instead of Calling vLLM Directly?

The application layer provides a controlled place for:

• request validation;
• authentication;
• prompt templates;
• conversation management;
• document retrieval;
• rate limiting;
• logging;
• error handling;
• model selection;
• security rules.

Exposing vLLM directly to every user would mix model-serving concerns with application concerns.

The recommended flow is:

Browser
   ↓
FastAPI
   ↓
vLLM

not:

Browser
   ↓
vLLM

Step 11: Add Conversation History

A language model does not automatically remember previous HTTP requests.

The application must resend relevant history with every new prompt.

The browser can maintain a list such as:

const history = [
    {
        role: "user",
        content: "What is a Kubernetes pod?"
    },
    {
        role: "assistant",
        content: "A pod is the smallest deployable unit..."
    }
];

When the next user message is sent, the browser includes this list in the request.

The backend should limit the amount of history:

for item in request.history[-20:]:
    messages.append(
        {
            "role": item.role,
            "content": item.content,
        }
    )

Without a limit, every conversation would grow indefinitely and eventually exceed the model context window.

In Stage 1, history is stored only in browser memory.

Refreshing the page clears the conversation.

Persistent conversations can be added later with a database.

Step 12: Create the Browser Interface

The Stage 1 interface can remain deliberately simple.

It should contain:

• a chat-message area;
• a text box;
• a Send button;
• a Clear button;
• a model-health indicator;
• a temperature field;
• a maximum-token field.

The browser sends:

POST /api/chat

Example request:

{
  "message": "Explain containers for a beginner.",
  "history": [],
  "temperature": 0.2,
  "max_tokens": 300
}

Example response:

{
  "reply": "A container is a lightweight package...",
  "model": "qwen3-4b"
}

Avoid injecting generated HTML

Display user and model text with:

element.textContent = content;

Avoid inserting untrusted content directly with:

element.innerHTML = content;

Model output should be treated as untrusted text unless it passes through a carefully configured sanitizer.

Step 13: Test the Application Locally

Keep the vLLM port-forward active:

kubectl port-forward `
  -n ai-lab `
  service/vllm-model `
  8000:8000

Set local variables:

$env:VLLM_BASE_URL = "http://127.0.0.1:8000/v1"
$env:VLLM_MODEL = "qwen3-4b"

Start the application:

python -m uvicorn `
  app.main:app `
  --host 127.0.0.1 `
  --port 8080 `
  --reload

Open:

http://127.0.0.1:8080

Test conversation history:

User: What is a Kubernetes Service?

Then:

User: Summarize your previous answer in five words.

The second response should reflect the first answer.

Step 14: Containerize the Application

Create a Dockerfile:

FROM python:3.14-slim

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PIP_NO_CACHE_DIR=1

WORKDIR /app

RUN groupadd --gid 10001 appuser \
    && useradd \
        --uid 10001 \
        --gid appuser \
        --no-create-home \
        --shell /usr/sbin/nologin \
        appuser

COPY requirements.txt ./requirements.txt

RUN python -m pip install --upgrade pip \
    && python -m pip install \
        --requirement requirements.txt

COPY --chown=appuser:appuser app ./app

USER appuser

EXPOSE 8080

CMD ["python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080", "--proxy-headers"]

Important points:

• the application runs as a non-root user;
• Python output is unbuffered;
• dependencies are installed from a pinned file;
• only the required project files are copied;
• the Uvicorn command listens on all container interfaces;
• the JSON-form CMD is kept on one line.

Create .dockerignore:

.git
.gitlab-ci.yml
.venv
__pycache__
*.pyc
.env
README.md

Do not copy local secrets into the image.

Step 15: Build the Application with GitLab CI/CD

The pipeline follows this flow:

Git commit
    ↓
Build image with Podman
    ↓
Validate application
    ↓
Push image to private registry
    ↓
Deploy manifests to K3s
    ↓
Wait for rollout
    ↓
Run health check

Suggested stages:

stages:
  - build
  - push
  - deploy

Use an immutable image tag based on the Git commit:

registry.example.net/team/basic-gen-ai:1a2b3c4d

This is more traceable than deploying only:

latest

A Podman build command can look like:

sudo podman build \
  --tag "$IMAGE_TAG" \
  --file "$DOCKER_FILE" \
  .

An additional Python validation can run inside the image:

sudo podman run \
  --rm \
  "$IMAGE_TAG" \
  python -m py_compile /app/app/main.py

The push stage authenticates to the registry and uploads the immutable image.

The deploy stage uses kubectl to apply the Kubernetes resources.

Step 16: Create the FastAPI Deployment

A simplified application Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: basic-gen-ai
  namespace: ai-lab
  labels:
    app.kubernetes.io/name: basic-gen-ai
    app.kubernetes.io/part-of: local-ai
spec:
  replicas: 1

  selector:
    matchLabels:
      app.kubernetes.io/name: basic-gen-ai

  template:
    metadata:
      labels:
        app.kubernetes.io/name: basic-gen-ai
        app.kubernetes.io/part-of: local-ai

    spec:
      imagePullSecrets:
        - name: registry-credentials

      containers:
        - name: application
          image: registry.example.net/team/basic-gen-ai:IMAGE_TAG

          ports:
            - name: http
              containerPort: 8080

          env:
            - name: VLLM_BASE_URL
              value: "http://vllm-model.ai-lab.svc.cluster.local:8000/v1"

            - name: VLLM_MODEL
              value: "qwen3-4b"

          resources:
            requests:
              cpu: 100m
              memory: 256Mi

            limits:
              cpu: "1"
              memory: 1Gi

          startupProbe:
            httpGet:
              path: /api/live
              port: http
            periodSeconds: 5
            failureThreshold: 30

          livenessProbe:
            httpGet:
              path: /api/live
              port: http
            periodSeconds: 20

          readinessProbe:
            httpGet:
              path: /api/health
              port: http
            periodSeconds: 10

Notice that this Deployment does not request:

nvidia.com/gpu

Only the vLLM pod needs the GPU.

FastAPI is a normal CPU workload.

Step 17: Create the Application Service

apiVersion: v1
kind: Service
metadata:
  name: basic-gen-ai
  namespace: ai-lab
  labels:
    app.kubernetes.io/name: basic-gen-ai
spec:
  type: ClusterIP

  selector:
    app.kubernetes.io/name: basic-gen-ai

  ports:
    - name: http
      port: 8080
      targetPort: http

Inside Kubernetes, the application is now available at:

http://basic-gen-ai:8080

The Service sends traffic only to pods matching:

app.kubernetes.io/name=basic-gen-ai

Step 18: Validate the Internal Application

Check the deployment:

kubectl get deployment basic-gen-ai `
  -n ai-lab `
  -o wide

Check the pod:

kubectl get pods `
  -n ai-lab `
  -l "app.kubernetes.io/name=basic-gen-ai" `
  -o wide

Check the service:

kubectl get service basic-gen-ai `
  -n ai-lab `
  -o wide

A healthy application should show:

Deployment: 1/1 available
Pod:        1/1 Running
Restarts:   0
Service:    ClusterIP

Test from inside Kubernetes

Create a temporary curl pod:

kubectl run application-health-check `
  -n ai-lab `
  --image=curlimages/curl `
  --restart=Never `
  --command -- sleep 300

Call the service:

kubectl exec `
  -n ai-lab `
  application-health-check `
  -- curl -sS `
  http://basic-gen-ai:8080/api/health

Expected response:

{
  "status": "ok",
  "vllm": "reachable",
  "configured_model": "qwen3-4b",
  "available_models": [
    "qwen3-4b"
  ]
}

Delete the test pod:

kubectl delete pod application-health-check `
  -n ai-lab

This test proves the complete internal path:

Test pod
   ↓
Application Service
   ↓
FastAPI
   ↓
vLLM Service
   ↓
Language model
   ↓
GPU

Step 19: Check the NGINX Ingress Controller

List available Ingress classes:

kubectl get ingressclass

A working NGINX installation may show:

NAME    CONTROLLER
nginx   k8s.io/ingress-nginx

The Ingress resource must use the matching class:

ingressClassName: nginx

Step 20: Create the Application Ingress

Create:

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: basic-gen-ai
  namespace: ai-lab
  labels:
    app.kubernetes.io/name: basic-gen-ai
spec:
  ingressClassName: nginx

  rules:
    - host: basic.ai-lab.example.net

      http:
        paths:
          - path: /
            pathType: Prefix

            backend:
              service:
                name: basic-gen-ai

                port:
                  number: 8080

Apply it:

kubectl apply -f .\ingress.yaml

Check it:

kubectl get ingress `
  -n ai-lab `
  -o wide

Describe the route:

kubectl describe ingress basic-gen-ai `
  -n ai-lab

Look for:

Host:     basic.ai-lab.example.net
Backend:  basic-gen-ai:8080

Step 21: Configure DNS

A wildcard DNS record can route all lab applications toward the Ingress Controller.

Example:

*.ai-lab.example.net
    CNAME
ingress.example.net

The application hostname is:

basic.ai-lab.example.net

Verify DNS:

Resolve-DnsName basic.ai-lab.example.net

The result should lead to the IP address used by the Ingress Controller.

Test the application:

Invoke-RestMethod `
  -Uri "http://basic.ai-lab.example.net/api/health" |
  ConvertTo-Json -Depth 10

Open the browser interface:

http://basic.ai-lab.example.net

Port forwarding is no longer required for normal access.

Understanding the Final Request Path

When a user sends a message, the request passes through:

Browser
   ↓
DNS
   ↓
NGINX Ingress
   ↓
FastAPI Service
   ↓
FastAPI pod
   ↓
vLLM Service
   ↓
vLLM pod
   ↓
NVIDIA runtime
   ↓
NVIDIA GPU

Each layer has a separate responsibility.

This makes troubleshooting easier because we can test each boundary independently.

Common Problems

Pod Remains Pending with Insufficient GPU

Check the pod events:

kubectl describe pod `
  -n ai-lab `
  VLLM_POD_NAME

A common message is:

Insufficient nvidia.com/gpu

Possible causes:

• another pod already reserves the GPU;
• a completed test pod was not deleted;
• the selected node does not advertise a GPU;
• the node is cordoned;
• the NVIDIA device plugin is unavailable.

Check allocation:

kubectl describe node gpu-worker-02

Look under:

Allocated resources

Node Is Cordoned

Check:

kubectl get nodes

A cordoned node may show:

SchedulingDisabled

When safe, restore scheduling:

kubectl uncordon gpu-worker-02

PVC Is Stuck Attaching

Inspect the pod:

kubectl describe pod `
  -n ai-lab `
  VLLM_POD_NAME

Then inspect:

kubectl get pvc -n ai-lab
kubectl get volumeattachments

A storage problem may appear as:

ContainerCreating

even though the container image is not the real problem.

The actual failure may occur lower in the stack:

PVC
 ↓
CSI driver
 ↓
Longhorn engine
 ↓
iSCSI initiator
 ↓
Linux block device

Read the complete error before restarting random components.

Corrupted Model Cache

A partial download may create an invalid tokenizer or model configuration file.

Symptoms can include:

JSONDecodeError

or parsing failures while loading tokenizer metadata.

A safe troubleshooting approach is:

1. scale the model Deployment to zero;
2. mount the persistent volume in a maintenance pod;
3. move the suspicious model directory into a quarantine folder;
4. restart the model Deployment;
5. allow a clean download;
6. delete the quarantine only after the model works.

Moving the damaged cache is safer than immediately deleting all evidence.

vLLM Takes Several Minutes to Start

This is often normal during the first startup.

Logs may show:

• checkpoint downloads;
• shard loading;
• compilation;
• CUDA graph capture;
• cache allocation;
• warm-up.

Do not use an aggressive startup probe.

The startup probe should allow enough time for the model to initialize.

Pod Is Running but Not Ready

Check:

kubectl describe pod `
  -n ai-lab `
  VLLM_POD_NAME

Then test the model health endpoint from inside the pod or through the service.

A pod can be Running while the model server is still loading.

Running describes the container process.

Ready means Kubernetes considers it prepared to receive traffic.

Dockerfile Build Reports an Unknown Instruction

A multiline JSON-form CMD may be interpreted incorrectly by some build configurations.

Use one line:

CMD ["python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]

Do not write each JSON argument as if it were a separate Dockerfile instruction.

Label Filter Finds No Resources

This command:

kubectl get pods `
  -n ai-lab `
  -l app=basic-gen-ai

returns nothing when the pod actually uses:

app.kubernetes.io/name=basic-gen-ai

Display real labels:

kubectl get deployments,pods,services `
  -n ai-lab `
  --show-labels

Then use the correct selector:

kubectl get pods `
  -n ai-lab `
  -l "app.kubernetes.io/name=basic-gen-ai"

kubectl Reports Unauthorized

Example:

The server has asked for the client to provide credentials

This normally means the current kubeconfig contains an expired or invalid credential.

Check the active context:

kubectl config current-context
kubectl config get-contexts

Do not permanently switch between unrelated configuration files without understanding which contexts they contain.

A better long-term solution is to update the valid cluster, user and token entries inside the standard kubeconfig while preserving existing contexts.

Always create a backup first.

DNS Works but the Application Does Not Open

Test each layer separately.

Check DNS

Resolve-DnsName basic.ai-lab.example.net

Check the Ingress

kubectl get ingress -n ai-lab -o wide

Check the route

kubectl describe ingress basic-gen-ai `
  -n ai-lab

Check the Service

kubectl get endpoints basic-gen-ai `
  -n ai-lab

Check FastAPI internally

kubectl run temporary-curl `
  -n ai-lab `
  --image=curlimages/curl `
  --restart=Never `
  --command -- sleep 300

Then:

kubectl exec `
  -n ai-lab `
  temporary-curl `
  -- curl -sS `
  http://basic-gen-ai:8080/api/health

Troubleshoot from the inside outward.

Security Limitations of Stage 1

This is a learning environment.

The initial application does not yet include:

• user authentication;
• role-based application authorization;
• HTTPS;
• rate limiting;
• prompt quotas;
• content moderation;
• persistent conversation storage;
• encrypted application secrets;
• network policies;
• audit logging.

Do not expose the Stage 1 application directly to the public internet without additional protection.

At minimum, a production-oriented version should add:

• TLS;
• authentication;
• authorization;
• network restrictions;
• rate limits;
• request-size limits;
• application monitoring;
• secret management;
• backup and recovery procedures.

Stage 1 Completion Checklist

Stage 1 is complete when all of the following are true:

[OK] Stage 0 GPU smoke test is complete
[OK] The ai-lab namespace exists
[OK] The model-cache PVC is Bound
[OK] The vLLM pod mounts persistent storage
[OK] The vLLM pod reserves one NVIDIA GPU
[OK] The model checkpoint loads successfully
[OK] The vLLM health endpoint responds
[OK] /v1/models lists the configured model
[OK] A PowerShell prompt receives an answer
[OK] FastAPI can call the vLLM service
[OK] The browser chat interface works
[OK] Conversation history works within the browser tab
[OK] The application image builds successfully
[OK] The image is pushed to a private registry
[OK] GitLab CI/CD deploys the application
[OK] The FastAPI pod reaches Ready
[OK] The internal application health check succeeds
[OK] NGINX Ingress routes to the FastAPI Service
[OK] DNS resolves the application hostname
[OK] The application is accessible without port forwarding

What Have We Learned?

Stage 0 proved that Kubernetes could use the physical GPU.

Stage 1 connected that GPU to a complete application:

GPU infrastructure
      ↓
Model server
      ↓
OpenAI-compatible API
      ↓
FastAPI backend
      ↓
Browser interface
      ↓
Container image
      ↓
CI/CD pipeline
      ↓
Kubernetes deployment
      ↓
Ingress and DNS

The most important lesson from Stage 1 is:

A language model is only one component of a usable AI system.

A practical AI application also requires:

• storage;
• networking;
• health checks;
• application logic;
• user-interface code;
• containerization;
• deployment automation;
• operational troubleshooting.

What Comes Next?

The current application can answer questions using knowledge already contained in the language model.

It cannot yet answer accurately from private documents.

Stage 2 will introduce Retrieval-Augmented Generation, commonly called RAG.

In the next stage, we will:

• upload a document;
• extract its text;
• divide the text into smaller chunks;
• generate embeddings;
• store vectors;
• search for relevant passages;
• add those passages to the model prompt;
• generate an answer based on the retrieved context;
• display the sources used for the answer.

The Stage 2 architecture will add:

User question
      ↓
Embedding model
      ↓
Vector database
      ↓
Relevant document chunks
      ↓
Prompt construction
      ↓
vLLM
      ↓
Answer with sources

Stage 1 gives us the application platform needed to build that system.

Frequently Asked Questions

Is vLLM the language model?

No.

vLLM is the inference server.

The model in this example is Qwen3-4B.

A simple comparison is:

Qwen3-4B = the model
vLLM     = the server that runs the model
FastAPI  = the application that uses the server

Why use a 4-billion-parameter model?

A smaller model is easier to run on a single consumer GPU.

It downloads faster, requires less VRAM and is suitable for learning the full deployment process.

Larger models can be tested after the platform is stable.

Why use persistent storage?

The model files are large.

Persistent storage prevents unnecessary re-downloads after routine pod recreation.

Why use an OpenAI-compatible API?

It allows applications to use familiar client libraries and request formats.

The application can later switch between compatible model endpoints with fewer code changes.

Why does FastAPI not request a GPU?

FastAPI processes HTTP requests and application logic.

vLLM performs model inference and needs the GPU.

Keeping them separate avoids wasting GPU resources.

Does the model remember earlier messages automatically?

No.

The application must send relevant conversation history with each new request.

Why use a Kubernetes Service?

Pod IP addresses are temporary.

A Service gives the application a stable internal DNS name.

Why use an Ingress?

A ClusterIP Service is available only inside Kubernetes.

Ingress provides hostname-based access through the cluster’s HTTP entry point.

Does a wildcard DNS record provide HTTPS?

No.

DNS maps a hostname to an address.

HTTPS also requires a valid TLS certificate and an Ingress TLS configuration.

Is this already a RAG application?

No.

Stage 1 is a basic chat application backed by a local model.

RAG will be added in Stage 2.

Further Reading

For the previous part of this project, read:

Stage 0: Preparing a K3s Cluster with NVIDIA GPUs for Local AI

The official Kubernetes documentation explains:

• GPU scheduling;
• extended resources;
• Services;
• Deployments;
• probes;
• Ingress.

The vLLM documentation describes:

• supported models;
• OpenAI-compatible endpoints;
• model-serving arguments;
• memory configuration;
• chat templates.

FastAPI documentation covers:

• request models;
• asynchronous endpoints;
• static files;
• application health checks.

GitLab documentation covers:

• CI/CD stages;
• registry authentication;
• protected variables;
• deployment jobs.

Conclusion

Stage 0 answered this question:

Can Kubernetes start a container that uses the NVIDIA GPU?

Stage 1 answered a much larger question:

Can that GPU power a complete, deployable local AI application?

The answer is yes.

We now have:

• a model running on the GPU;
• an OpenAI-compatible API;
• persistent model storage;
• a FastAPI backend;
• a browser chat interface;
• short conversation history;
• a container image;
• automated GitLab deployment;
• a Kubernetes Service;
• an NGINX Ingress;
• DNS-based access.

The foundation is now ready for the next major capability: asking questions about private documents through Retrieval-Augmented Generation.