A Kubernetes pod that remains in ContainerCreating can be frustrating to diagnose.

The pod has been scheduled. Its persistent volume claim may report Bound. The container image may already be available. CPU and memory may be sufficient.

However, the container never starts.

In many cases, the problem is not the application container at all. It is somewhere in the storage chain between Kubernetes, the CSI driver, the storage engine and the Linux host.

This article explains how to diagnose one such incident involving:

• Kubernetes;
• K3s;
• Longhorn;
• CSI volume attachments;
• persistent volume claims;
• Longhorn engines and replicas;
• Open-iSCSI;
• Linux node configuration.

The case study is based on a real technical incident, but all names, namespaces, hostnames, IP addresses, identifiers and organization-specific details have been anonymized.

The final root cause was unexpected: an old parameter inside a persistent iSCSI node record caused iscsiadm to reject its configuration database. Longhorn could create its replicas, but it could not start the block-device frontend required by the pod.

The initial symptom

A test pod was created with two important requirements:

1. Access to an NVIDIA GPU.
2. Access to a persistent volume managed by Longhorn.

The pod remained in this state:

NAME             READY   STATUS              RESTARTS   IP
storage-test     0/1     ContainerCreating   0          <none>

Trying to read its logs returned:

container is waiting to start: ContainerCreating

This response is normal when the application container has not actually started. There are no application logs yet because Kubernetes is still preparing the pod.

The absence of logs does not mean there is nothing to investigate. It means the investigation must begin outside the container.

What does ContainerCreating mean?

ContainerCreating is a status displayed while Kubernetes prepares the environment required by a container.

During this stage, Kubernetes may be:

• creating the pod sandbox;
• configuring pod networking;
• pulling the container image;
• mounting ConfigMaps and Secrets;
• attaching persistent volumes;
• formatting storage devices;
• mounting filesystems;
• preparing runtime devices such as GPUs.

A pod may therefore remain in ContainerCreating even when the application image and application code are perfectly valid.

For CSI-backed storage, Kubernetes invokes a series of storage operations to attach, stage and publish the volume before the container can use it. These operations include controller-level attachment and node-level mounting.

Understand the complete storage chain

A beginner may look only at the PVC:

PVC: Bound

and assume storage is working.

That is not enough.

The complete path looks more like this:

Pod
  ↓
PersistentVolumeClaim
  ↓
PersistentVolume
  ↓
Kubernetes VolumeAttachment
  ↓
CSI driver
  ↓
Longhorn VolumeAttachment
  ↓
Longhorn volume
  ↓
Longhorn engine
  ↓
Longhorn replicas
  ↓
iSCSI frontend
  ↓
Linux block device
  ↓
Filesystem mount
  ↓
Application container

A failure at any level can prevent the container from starting.

A Bound PVC proves only that Kubernetes matched the claim to a persistent volume. It does not prove that the volume was successfully attached to the selected node, exposed as a block device or mounted inside the pod.

Kubernetes represents external CSI attachment requests through VolumeAttachment objects. These objects identify a persistent volume, the target node and whether the attachment operation succeeded.

Step 1: Inspect the pod

Start with the basic pod status:

kubectl get pod `
  -n demo-ai `
  storage-test `
  -o wide

Example:

NAME           READY   STATUS              IP       NODE
storage-test   0/1     ContainerCreating   <none>   gpu-worker-02

Important observations include:

• the pod has been scheduled;
• a node has been selected;
• the pod has no IP yet;
• the container has not started.

Now describe the pod:

kubectl describe pod `
  -n demo-ai `
  storage-test

Read the Events section at the bottom.

Look for messages such as:

FailedAttachVolume
FailedMount
Unable to attach or mount volumes
AttachVolume.Attach failed
DeadlineExceeded

Pod events provide an initial direction, but they often contain only the final symptom rather than the deepest root cause.

Step 2: Check the PVC

List the persistent volume claim:

kubectl get pvc `
  -n demo-ai

Example:

NAME          STATUS   VOLUME                                     CAPACITY   STORAGECLASS
model-cache   Bound    pvc-11111111-2222-3333-4444-555555555555   30Gi       longhorn

The claim is Bound, but the pod is still not running.

Extract the actual persistent volume name:

$pv = kubectl get pvc model-cache `
  -n demo-ai `
  -o jsonpath="{.spec.volumeName}"

Verify the variable:

Write-Host "PV=[$pv]"

Expected:

PV=[pvc-11111111-2222-3333-4444-555555555555]

PowerShell variable warning

PowerShell variables exist only in the PowerShell session where they were created.

Opening another PowerShell window and running:

Write-Host "PV=[$pv]"

may return:

PV=[]

An empty variable can make a command unexpectedly query every volume instead of the intended one.

Always verify variables before using them in diagnostic commands.

Step 3: Inspect the Kubernetes VolumeAttachment

List the Kubernetes VolumeAttachment object associated with the PV.

A reliable PowerShell method is:

$volumeAttachments = kubectl get volumeattachments.storage.k8s.io -o json |
  ConvertFrom-Json

Filter by the persistent volume name:

$va = $volumeAttachments.items |
  Where-Object {
    $_.spec.source.persistentVolumeName -eq $pv
  }

Display its important properties:

$va |
  Select-Object `
    @{Name="Name"; Expression={$_.metadata.name}},
    @{Name="PV"; Expression={$_.spec.source.persistentVolumeName}},
    @{Name="Node"; Expression={$_.spec.nodeName}},
    @{Name="Attached"; Expression={$_.status.attached}},
    @{Name="AttachError"; Expression={$_.status.attachError.message}},
    @{Name="DetachError"; Expression={$_.status.detachError.message}} |
  Format-List

The affected case produced a result similar to:

PV          : pvc-11111111-2222-3333-4444-555555555555
Node        : gpu-worker-02
Attached    : False
AttachError : rpc error: code = DeadlineExceeded

This is an important turning point.

The pod is not waiting for its application image. Kubernetes is waiting for the CSI driver to attach the volume.

The VolumeAttachment API exists specifically to track attachment of a persistent volume to a node by an external storage attacher.

Step 4: Inspect the Longhorn volume

Longhorn stores additional state in its own custom resources.

Run:

kubectl get volumes.longhorn.io `
  -n longhorn-system `
  $pv `
  -o custom-columns="NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness,CURRENT-NODE:.status.currentNodeID,SPEC-NODE:.spec.nodeID,IMAGE:.spec.image"

The affected volume showed:

STATE       ROBUSTNESS   CURRENT-NODE   SPEC-NODE
attaching   unknown                     gpu-worker-02

This tells us:

• Longhorn received the request;
• Longhorn knows which node should receive the volume;
• the volume has not completed attachment;
• the current node remains empty;
• the volume has not reached a healthy operational state.

For more detail:

kubectl get volumes.longhorn.io `
  -n longhorn-system `
  $pv `
  -o yaml

Useful fields include:

spec:
  nodeID: gpu-worker-02
  numberOfReplicas: 3
  image: longhornio/longhorn-engine:<version>

status:
  currentNodeID: ""
  robustness: unknown
  state: attaching

Longhorn requires Open-iSCSI and a running iscsid daemon on nodes that host its version 1 data-engine volumes.

Step 5: Inspect the Longhorn attachment ticket

Longhorn has its own VolumeAttachment custom resource.

Run:

kubectl get volumeattachments.longhorn.io `
  -n longhorn-system `
  $pv `
  -o yaml

A normal CSI attachment request contains a ticket similar to:

spec:
  attachmentTickets:
    csi-example-id:
      nodeID: gpu-worker-02
      type: csi-attacher

The status may show:

status:
  attachmentTicketStatuses:
    csi-example-id:
      satisfied: false

This means:

• Kubernetes requested the attachment;
• Longhorn created the ticket;
• the ticket has not been satisfied;
• the failure is inside the Longhorn attachment workflow.

Longhorn documents its attachment-ticket mechanism and recommends carefully investigating attachment state before manually changing these resources.

Do not immediately delete attachment tickets, engine objects or replica objects. They are controller-managed resources, and manual changes can make the incident harder to understand.

Step 6: Check the Longhorn engine image

Before investigating the engine itself, verify that its image is deployed.

kubectl get engineimages.longhorn.io `
  -n longhorn-system `
  -o custom-columns="NAME:.metadata.name,IMAGE:.spec.image,STATE:.status.state,REFCOUNT:.status.refCount,INCOMPATIBLE:.status.incompatible"

Example healthy output:

IMAGE                                  STATE      INCOMPATIBLE
longhornio/longhorn-engine:<version>   deployed   false

Check the engine-image DaemonSet:

kubectl get daemonsets `
  -n longhorn-system `
  -l longhorn.io/component=engine-image `
  -o wide

Also inspect the configured default image:

kubectl get settings.longhorn.io `
  -n longhorn-system `
  default-engine-image `
  -o jsonpath="{.value}"

Write-Host

This step is useful because storage namespaces can contain errors for unrelated historical volumes.

For example, logs may mention:

engineimage.longhorn.io not found

for a different PV that still references an older Longhorn engine.

Do not assume that the first red error in the logs belongs to your pod.

Always compare:

• volume identifier;
• engine identifier;
• replica identifier;
• node name;
• engine image version.

Step 7: Inspect the Longhorn engine and replicas

A Longhorn volume normally has:

• one engine;
• one or more replicas.

The engine coordinates data access. The replicas store copies of the volume data.

Define the expected engine name:

$engine = "$pv-e-0"

Inspect it:

kubectl get engines.longhorn.io `
  -n longhorn-system `
  $engine `
  -o custom-columns="NAME:.metadata.name,DESIRED:.spec.desireState,CURRENT:.status.currentState,NODE:.spec.nodeID,INSTANCE-MANAGER:.status.instanceManagerName,ENDPOINT:.status.endpoint"

Inspect the replicas:

kubectl get replicas.longhorn.io `
  -n longhorn-system `
  -l longhornvolume=$pv `
  -o custom-columns="NAME:.metadata.name,DESIRED:.spec.desireState,CURRENT:.status.currentState,NODE:.spec.nodeID,INSTANCE-MANAGER:.status.instanceManagerName,FAILED:.spec.failedAt,HEALTHY:.spec.healthyAt"

In the incident, replicas repeatedly:

1. started;
2. opened their storage directories;
3. accepted network connections;
4. received EOF;
5. shut down;
6. restarted on new ports.

Representative log messages included:

Opening replica
failed to handle data server: EOF
Received signal interrupt to shutdown
Creating instance

The repeated SIGINT shutdowns showed that the replicas were not simply crashing unexpectedly. The Longhorn control process was cleaning them up after the engine failed to stabilize.

The replica errors were therefore symptoms, not the original cause.

Step 8: Find the correct Longhorn manager pod

List Longhorn pods on the affected node:

kubectl get pods `
  -n longhorn-system `
  -o wide |
  Select-String "gpu-worker-02"

Find the manager pod programmatically:

$longhornManager = kubectl get pods `
  -n longhorn-system `
  -l app=longhorn-manager `
  --field-selector spec.nodeName=gpu-worker-02 `
  -o jsonpath="{.items[0].metadata.name}"

Verify it:

Write-Host "Longhorn manager: $longhornManager"

Do not copy placeholder names such as:

EXACT_LONGHORN_MANAGER_POD
longhorn-manager-abc12

Those are examples, not real pod names.

Avoid overly broad log filters

A command such as this can produce enormous output:

kubectl logs `
  -n longhorn-system `
  $longhornManager `
  --since=60m |
  Select-String -Pattern "$pv|error|failed|attach|mount|iscsi"

The problem is the filter:

error|failed|attach|mount|iscsi

A busy storage namespace may contain thousands of matching lines for unrelated volumes.

Use the exact PV first:

kubectl logs `
  -n longhorn-system `
  $longhornManager `
  --since=15m `
  --tail=3000 |
  Select-String -SimpleMatch $pv

Add limited context only when necessary:

kubectl logs `
  -n longhorn-system `
  $longhornManager `
  --since=15m `
  --tail=3000 |
  Select-String -SimpleMatch $pv -Context 2,5

A good diagnostic principle is:

Filter by the exact resource identifier before filtering by generic error words.

Step 9: Inspect the instance-manager logs

Find the instance-manager pod on the affected node:

$instanceManager = kubectl get pods `
  -n longhorn-system `
  --field-selector spec.nodeName=gpu-worker-02 `
  -o jsonpath="{range .items[?(@.metadata.name)]}{.metadata.name}{'\n'}{end}" |
  Select-String "^instance-manager-" |
  Select-Object -First 1

A simpler approach is to list the pods and copy the exact name:

kubectl get pods `
  -n longhorn-system `
  -o wide |
  Select-String "instance-manager"

Then set it:

$instanceManager = "instance-manager-example"

Use a narrow filter:

kubectl logs `
  -n longhorn-system `
  $instanceManager `
  --since=10m `
  --tail=4000 |
  Select-String -Pattern "Unknown parameter name|config file.*invalid|Failed to startup frontend" `
  -Context 2,6

This command exposed the decisive error.

The real root cause

The engine log contained:

Failed to startup frontend

iSCSI ERROR:
Unknown parameter name node.session.sess_reopen_log_freq

iSCSI ERROR:
config file /var/lib/iscsi/nodes/.../default invalid

The Longhorn engine then exited:

failed to start up frontend
exit status 1

These messages established the complete causal chain:

Old iSCSI node record
        ↓
iscsiadm rejected the configuration
        ↓
Longhorn could not inspect or initialize the iSCSI target
        ↓
Longhorn frontend could not start
        ↓
Engine process exited
        ↓
Replica connections closed with EOF
        ↓
Longhorn restarted the engine and replicas
        ↓
CSI attachment timed out
        ↓
Pod remained in ContainerCreating

The original logs explicitly showed Failed to startup frontend, the unsupported node.session.sess_reopen_log_freq parameter and an invalid file under /var/lib/iscsi/nodes.

Longhorn uses Open-iSCSI to create block devices. On RHEL-derived systems, persistent iSCSI node records are stored under /var/lib/iscsi/nodes, and system components may read those records when establishing or restoring connections.

The parameter itself exists in newer upstream Open-iSCSI sources, but the installed package in the affected environment did not accept it in the stored node record.

This was therefore a compatibility problem between a persistent iSCSI node record and the iscsiadm implementation installed on the node.

Step 10: Inspect the iSCSI records on the affected node

Connect directly to the Linux node where Longhorn is trying to attach the volume.

Search for the unsupported parameter:

sudo grep -Rns \
  'node.session.sess_reopen_log_freq' \
  /var/lib/iscsi/nodes

Example:

/var/lib/iscsi/nodes/iqn.example/.../default:52:
node.session.sess_reopen_log_freq = 1

Check the installed package:

rpm -q iscsi-initiator-utils

Check the command version:

iscsiadm --version

Record these results before modifying anything.

Important safety warning

The directory:

/var/lib/iscsi/nodes

contains Open-iSCSI’s persistent node database.

Do not:

• delete the entire directory;
• delete all node records;
• blindly replace every configuration file;
• store backup files inside the active database directory;
• restart storage services without understanding active sessions;
• edit several cluster nodes at the same time.

The remediation below removes one unsupported configuration line from the files that actually contain it.

Production clusters should have current backups, storage-health checks and an approved maintenance procedure before host-level storage configuration is changed.

Step 11: Back up the affected files outside the active directory

Create a backup directory outside /var/lib/iscsi/nodes:

sudo mkdir -p /root/iscsi-node-backups

Define one affected file:

BAD_ISCSI_FILE='/var/lib/iscsi/nodes/iqn.example/10.0.0.25,3260,1/default'

Verify it:

sudo ls -l "$BAD_ISCSI_FILE"
sudo grep -n 'sess_reopen_log_freq' "$BAD_ISCSI_FILE"

Back it up outside the active iSCSI database:

sudo cp -a \
  "$BAD_ISCSI_FILE" \
  "/root/iscsi-node-backups/default-$(date +%Y%m%d-%H%M%S).bak"

Why backups must be outside /var/lib/iscsi/nodes

During the incident, a backup named similarly to:

default.bak-YYYYMMDD-HHMMSS

was initially created next to the active default file.

Running:

iscsiadm -m node

then reported the same invalid parameter—but this time it referenced the backup file.

The tool was still parsing that file as part of the node-record tree.

The lesson is simple:

Do not store ad hoc backup files inside /var/lib/iscsi/nodes.

Step 12: Remove only the unsupported parameter

Remove the matching line from the affected active file:

sudo sed -i \
  '/^[[:space:]]*node\.session\.sess_reopen_log_freq[[:space:]]*=/d' \
  "$BAD_ISCSI_FILE"

Verify the result:

sudo grep -n \
  'sess_reopen_log_freq' \
  "$BAD_ISCSI_FILE" \
  || echo "Unsupported parameter removed"

Repeat this process only for the other active default files reported by the original recursive grep.

Do not assume there is only one record. A single iSCSI target can have records associated with more than one portal address.

Step 13: Confirm that no invalid records remain

Search again:

sudo grep -Rns \
  'node.session.sess_reopen_log_freq' \
  /var/lib/iscsi/nodes \
  || echo "Unsupported parameter removed from all iSCSI node records"

Check for accidental backup files:

sudo find \
  /var/lib/iscsi/nodes \
  -type f \
  ! -name default \
  -print

Ideally, this returns nothing.

If it returns .bak, .old, .copy or similarly named files, move those files outside the iSCSI database before validating.

Step 14: Validate Open-iSCSI

Run:

sudo iscsiadm -m node

A successful result lists the known iSCSI nodes without reporting:

Unknown parameter name
config file invalid

Check the exit code:

echo $?

Expected:

0

The command may list many Longhorn targets. That is normal on a Kubernetes node that has previously mounted several Longhorn volumes.

The important result is that no configuration parsing error remains.

Step 15: Watch Longhorn recover

Return to the Kubernetes administration workstation.

Recreate the PV variable if necessary:

$pv = kubectl get pvc model-cache `
  -n demo-ai `
  -o jsonpath="{.spec.volumeName}"

Check the volume:

kubectl get volumes.longhorn.io `
  -n longhorn-system `
  $pv `
  -o custom-columns="NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness,CURRENT-NODE:.status.currentNodeID"

The expected transition is:

STATE:       attaching → attached
ROBUSTNESS:  unknown   → healthy
CURRENT-NODE:          → gpu-worker-02

In the verified incident, correcting the invalid iSCSI records allowed the Longhorn volume to change to:

attached   healthy

The application pod then moved from:

ContainerCreating

to:

Running

No application code, GPU configuration or container image change was required.

Understanding watch commands

A command ending in:

-w

watches continuously.

For example:

kubectl get pod `
  -n demo-ai `
  storage-test `
  -w

It does not normally stop by itself.

Press:

Ctrl+C

after the expected state appears.

Stopping the watch does not stop the pod or change Kubernetes resources. It only ends the local display command.

Diagnostic mistakes that can waste time

Mistake 1: Assuming Bound means mounted

This:

PVC STATUS: Bound

does not prove that the volume is attached to the node.

Check:

• Kubernetes VolumeAttachment;
• Longhorn volume state;
• Longhorn attachment ticket;
• engine state;
• node-level mount operations.

Mistake 2: Looking only at application logs

When the container has not started, application logs do not exist.

Investigate:

Pod events
→ PVC
→ PV
→ VolumeAttachment
→ CSI
→ Longhorn
→ host storage

Mistake 3: Searching every error in the storage namespace

Longhorn may manage dozens or hundreds of volumes.

Generic searches can mix:

• missing engine-image errors;
• historical volume errors;
• backup-target validation warnings;
• deleted volumes;
• unrelated replicas;
• the affected volume.

Filter by the exact PV or engine identifier first.

Mistake 4: Treating EOF as the root cause

Replica logs may show:

failed to handle data server: EOF

An EOF only means the other side closed the connection.

In this incident, the engine closed the connection because its iSCSI frontend could not start. The EOF was downstream of the real error.

Mistake 5: Deleting Longhorn controller objects too early

Deleting engines, replicas or attachment tickets may temporarily change symptoms without fixing the host-level issue.

The controllers will often recreate the same resources, and the failure loop will continue.

Mistake 6: Saving backups inside the active iSCSI database

A file named:

default.bak

under /var/lib/iscsi/nodes may still be parsed as a node record.

Back up to a separate location such as:

/root/iscsi-node-backups

Mistake 7: Force-deleting the pod first

Deleting the pod may cause the Deployment to create a replacement, but the replacement will encounter the same storage failure.

Fix the attachment problem first.

Reusable diagnostic workflow

The following order works well for pods that use CSI-backed storage.

Layer 1: Pod

kubectl get pod -n <namespace> <pod> -o wide
kubectl describe pod -n <namespace> <pod>

Questions:

• Was a node selected?
• Does the pod have an IP?
• Are there FailedAttachVolume events?
• Are there FailedMount events?

Layer 2: PVC and PV

kubectl get pvc -n <namespace>
kubectl get pv

Questions:

• Is the claim Bound?
• Which PV belongs to it?
• Which StorageClass is used?

Layer 3: Kubernetes VolumeAttachment

kubectl get volumeattachments.storage.k8s.io

Questions:

• Is status.attached true?
• Is there an attachment error?
• Is the target node correct?

Layer 4: Longhorn volume

kubectl get volumes.longhorn.io -n longhorn-system <pv-name> -o yaml

Questions:

• Is the state attached, attaching or detached?
• Is robustness healthy, degraded or unknown?
• Is currentNodeID populated?

Layer 5: Longhorn attachment ticket

kubectl get volumeattachments.longhorn.io -n longhorn-system <pv-name> -o yaml

Questions:

• Is a CSI ticket present?
• Is it satisfied?
• Does it reference the correct node?

Layer 6: Engine and replicas

kubectl get engines.longhorn.io -n longhorn-system
kubectl get replicas.longhorn.io -n longhorn-system -l longhornvolume=<pv-name>

Questions:

• Does the engine start?
• Does it immediately enter an error state?
• Do replicas repeatedly restart?
• Is the engine endpoint created?

Layer 7: Longhorn logs

Search by:

exact PV
exact engine
exact replica
exact node

Avoid beginning with every occurrence of error.

Layer 8: Linux node

Check:

systemctl status iscsid
iscsiadm --version
iscsiadm -m node
journalctl -u iscsid

Questions:

• Is Open-iSCSI installed?
• Is the daemon running?
• Can the node database be parsed?
• Are there stale or incompatible records?
• Are there SELinux or permission errors?

Longhorn also documents host-level attachment failures caused by Open-iSCSI permissions, SELinux and stale connection information, which is why the Linux node must be included in the investigation.

Troubleshooting checklist

Use this checklist when a pod remains in ContainerCreating.

[ ] Confirm the pod has been scheduled
[ ] Read the pod Events section
[ ] Confirm the PVC is Bound
[ ] Extract the exact PV name
[ ] Inspect the Kubernetes VolumeAttachment
[ ] Check whether status.attached is true
[ ] Inspect the Longhorn volume state
[ ] Check Longhorn robustness
[ ] Inspect the Longhorn attachment ticket
[ ] Verify the current engine image is deployed
[ ] Inspect the Longhorn engine
[ ] Inspect all replicas for the affected volume
[ ] Filter logs by the exact PV identifier
[ ] Inspect the instance-manager logs
[ ] Check Open-iSCSI on the selected Linux node
[ ] Validate /var/lib/iscsi/nodes
[ ] Store backups outside the active configuration tree
[ ] Run iscsiadm -m node
[ ] Confirm Longhorn becomes attached and healthy
[ ] Confirm the pod enters Running

Final lesson

The visible symptom was:

Kubernetes pod stuck in ContainerCreating

The Kubernetes PVC appeared healthy:

Bound

The actual problem was several layers lower:

Invalid persistent Open-iSCSI node configuration

The most useful lesson is not the exact sed command. It is the diagnostic method:

Start with the pod
Follow the storage dependency chain
Use exact resource identifiers
Separate symptoms from root causes
Validate each layer before modifying it

Kubernetes storage failures often look like application failures because the application container is the final component waiting to start.

By following the chain from pod to PVC, from CSI attachment to Longhorn engine, and from Longhorn to the Linux iSCSI initiator, a vague ContainerCreating status can be turned into a precise and verifiable diagnosis.

Frequently asked questions

Why was the PVC Bound if the volume could not attach?

Binding and attachment are separate operations.

Binding associates a PVC with a PV. Attachment makes that PV available on a specific Kubernetes node. A PVC can therefore remain Bound while the CSI attachment fails.

Why were there no pod logs?

The application container had not started. Kubernetes was still preparing its storage.

Why did the Longhorn replicas keep restarting?

The Longhorn engine could not start its iSCSI frontend. After the engine exited, Longhorn cleaned up and recreated the related processes.

Was the EOF error the root cause?

No. The replicas received EOF after the engine connection closed. The original failure was the invalid iSCSI configuration.

Why did iscsiadm read the backup file?

The backup had been stored inside the persistent iSCSI node-record tree. Moving it outside that directory prevented it from being treated as an active configuration record.

Was a node restart required?

No restart was required in the verified incident. Once the invalid records were corrected and iscsiadm -m node completed successfully, Longhorn recovered through its normal reconciliation process.

Should all occurrences of the parameter always be deleted?

Not automatically.

First confirm that the installed iscsiadm rejects the parameter and that the affected files are responsible for the failure. Back up the exact records outside the active directory before changing them.

Could similar symptoms have a different cause?

Yes.

Other causes include:

• unavailable storage nodes;
• missing engine images;
• broken CSI plugins;
• SELinux denials;
• multipath configuration;
• disk pressure;
• network failures;
• stale attachment tickets;
• unavailable replicas;
• a stopped iscsid service.

The workflow in this article helps identify which cause applies to a particular volume.

Further reading

Kubernetes documents how CSI operations attach and mount persistent volumes before containers can use them.

The Kubernetes API reference describes VolumeAttachment resources and their role in external storage attachment.

Longhorn’s installation documentation lists Open-iSCSI and a running iscsid daemon as node requirements for version 1 volumes.

Longhorn’s troubleshooting documentation covers storage diagnostics, logs and support bundles for attachment and mounting problems.

Longhorn also documents how RHEL-derived systems use the persistent Open-iSCSI database under /var/lib/iscsi/nodes.