A Kubernetes workload can remain stuck in ContainerCreating even when the container image, CPU, memory and node selection are all correct.

When Longhorn provides the persistent storage, one common cause is a volume that cannot complete the attach process. Kubernetes may report repeated FailedAttachVolume events, the CSI VolumeAttachment may remain at attached: false, and Longhorn may show the volume in an indefinite attaching state.

This article explains how to diagnose and recover a Longhorn PVC failed to attach incident without deleting the volume, removing replicas or risking application data.

The guide is based on a real K3s and Longhorn incident. Hostnames, namespaces, volume identifiers and application details have been changed or replaced with generic placeholders.

The incident scenario

A stateful workload was explicitly assigned to a worker node and requested a Longhorn-backed PersistentVolumeClaim.

The pod was successfully scheduled, but it never reached the point where the container could start. It remained in:

Pending
ContainerCreating

The pod events showed repeated attachment failures:

Warning  FailedAttachVolume

AttachVolume.Attach failed for volume "pvc-xxxxxxxx":
rpc error: code = DeadlineExceeded
volume failed to attach to node worker-storage-02

The corresponding CSI VolumeAttachment existed, but its status showed:

status:
  attached: false
  attachError:
    message: >
      rpc error: code = DeadlineExceeded
      volume failed to attach to node worker-storage-02

At the same time, the Longhorn volume reported:

STATE       ROBUSTNESS   SPEC-NODE
attaching   unknown      worker-storage-02

This combination demonstrated that Kubernetes had already made the scheduling decision, but the Longhorn CSI attachment workflow could not finish.

Understanding the Longhorn volume attachment process

Before troubleshooting, it helps to understand the major components involved.

When a pod uses a Longhorn PVC, the attachment process typically includes:

1. Kubernetes schedules the pod to a node.
2. The Kubernetes attach/detach controller creates a VolumeAttachment.
3. The Longhorn CSI controller receives the attachment request.
4. Longhorn selects or starts the volume engine.
5. Longhorn connects the engine to the available replicas.
6. Longhorn exposes the volume through its block-device frontend.
7. The node connects to the device, commonly through iSCSI for Longhorn’s traditional data engine.
8. Kubelet mounts the volume into the pod.
9. The container can finally start.

A failure at any of these stages may appear to Kubernetes as the same general symptom:

FailedAttachVolume

That is why deleting and recreating the pod repeatedly rarely solves the underlying problem.

Typical symptoms of a Longhorn attachment failure

The most common symptoms include:

• Pod stuck in Pending or ContainerCreating
• PodReadyToStartContainers=False
• Repeated FailedAttachVolume events
• CSI attachment error with DeadlineExceeded
• VolumeAttachment.status.attached=false
• Longhorn volume stuck in attaching
• Longhorn volume robustness shown as unknown
• Longhorn engine repeatedly starting and stopping
• Missing or failed instance-manager pod
• iSCSI discovery or login failures
• Longhorn manager logs showing an invalid instance manager
• Volume becoming attached but remaining degraded

These symptoms are related, but they do not necessarily share the same root cause.

Step 1: Contain the workload

Before making storage changes, stop the application from repeatedly retrying the attachment.

For a Deployment:

kubectl scale deployment <deployment-name> \
  -n <application-namespace> \
  --replicas=0

For a standalone pod:

kubectl delete pod <pod-name> \
  -n <application-namespace> \
  --wait=false

Verify that no application pod is still using the PVC:

kubectl get pods \
  -n <application-namespace> \
  -o wide

This prevents Kubernetes from continuously generating new attachment attempts while the storage layer is being investigated.

Do not delete the PVC or PV.

Step 2: Identify the PVC, PV and target node

Start with the pod description:

kubectl describe pod <pod-name> \
  -n <application-namespace>

Look for:

• the assigned node;
• the PVC name;
• FailedAttachVolume;
• FailedMount;
• Multi-Attach;
• CSI timeout messages.

Retrieve the PV associated with the PVC:

kubectl get pvc <pvc-name> \
  -n <application-namespace> \
  -o jsonpath='{.spec.volumeName}{"\n"}'

Store it in a shell variable:

PV=$(kubectl get pvc <pvc-name> \
  -n <application-namespace> \
  -o jsonpath='{.spec.volumeName}')

Print the result:

echo "$PV"

It will normally resemble:

pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Step 3: Inspect the Kubernetes VolumeAttachment

List attachments associated with the PV:

kubectl get volumeattachments.storage.k8s.io \
  -o custom-columns='NAME:.metadata.name,PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached'

To filter for one volume:

kubectl get volumeattachments.storage.k8s.io \
  -o custom-columns='NAME:.metadata.name,PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached' |
grep "$PV"

Inspect the relevant object:

kubectl get volumeattachment.storage.k8s.io <attachment-name> \
  -o yaml

Pay attention to:

spec:
  attacher: driver.longhorn.io
  nodeName: worker-storage-02

status:
  attached: false
  attachError:
    message: ...

How to interpret the result

attached: true

The CSI attachment succeeded. The problem may instead be:

• filesystem mount failure;
• kubelet mount path;
• permissions;
• filesystem corruption;
• stale mount;
• pod-level volume configuration.

attached: false with an error

The problem is still inside the CSI or Longhorn attachment workflow.

No VolumeAttachment exists

The pod may not yet be scheduled, the PVC may not be bound, or the CSI controller may not have processed the request.

Step 4: Inspect the Longhorn volume state

Use the PV name as the Longhorn volume name:

kubectl get volumes.longhorn.io \
  -n longhorn-system \
  "$PV" \
  -o custom-columns='NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness,SPEC-NODE:.spec.nodeID,CURRENT-NODE:.status.currentNodeID,PENDING-NODE:.status.pendingNodeID'

Possible states include:

detached
attaching
attached
detaching

detached / unknown

This is usually normal when no workload is using the volume.

attaching / unknown

Longhorn accepted the attachment request but could not finish starting the engine or frontend.

attached / degraded

The workload may be able to use the volume, but fewer replicas are healthy than configured.

attached / healthy

The volume is attached and all required replicas are available.

Describe the full Longhorn volume:

kubectl describe volumes.longhorn.io \
  -n longhorn-system \
  "$PV"

Look for conditions such as:

ReplicaSchedulingFailure
DiskNotReady
DiskFilesystemChanged
TooManySnapshots
Restore
WaitForBackingImage

Step 5: Check Longhorn components on the target node

List Longhorn pods and their nodes:

kubectl get pods \
  -n longhorn-system \
  -o wide

Focus on the target worker:

kubectl get pods \
  -n longhorn-system \
  -o wide |
grep 'worker-storage-02'

Important components include:

• longhorn-manager
• instance-manager
• engine-image
• longhorn-csi-plugin

The manager should normally be running on every participating node.

Example:

longhorn-manager-xxxxx       2/2   Running
instance-manager-xxxxxxxx    1/1   Running
longhorn-csi-plugin-xxxxx    3/3   Running

If the instance-manager is missing, failed or associated with an old pod IP, Longhorn may be unable to start the volume engine.

In the investigated incident, manager logs repeatedly reported an instance manager in an invalid error state, preventing the engine process from being retrieved or synchronized.

Step 6: Inspect Longhorn manager logs

Find the manager running on the affected node:

kubectl get pods \
  -n longhorn-system \
  -l app=longhorn-manager \
  -o wide

Then inspect its recent logs:

kubectl logs \
  -n longhorn-system \
  <longhorn-manager-pod> \
  -c longhorn-manager \
  --since=30m

Filter for the volume:

kubectl logs \
  -n longhorn-system \
  <longhorn-manager-pod> \
  -c longhorn-manager \
  --since=30m |
grep "$PV"

Useful search terms include:

grep -Ei 'error|failed|attach|frontend|iscsi|instance manager|engine'

Common errors include:

invalid instance manager
failed to sync Longhorn engine
failed to start frontend
failed to discover iSCSI target
failed to login to iSCSI target
cannot find process

Do not stop at the first generic error. Continue until you find the lowest-level command that actually failed.

The root cause: an invalid iSCSI configuration record

In this incident, the most useful log entry was not the Kubernetes timeout. It was the error produced by iscsiadm on the affected worker:

iSCSI ERROR:
Unknown parameter name node.session.sess_reopen_log_freq

iSCSI ERROR:
config file /var/lib/iscsi/nodes/.../default invalid

Longhorn attempted to start its block-device frontend and invoked iscsiadm through the host namespace. The command failed because one or more stored iSCSI node records contained a parameter that the installed open-iscsi version did not understand.

As a result:

1. the Longhorn engine started;
2. the replica connections were initialized;
3. Longhorn attempted to expose the iSCSI frontend;
4. iscsiadm read all stored node records;
5. one invalid record caused the command to fail;
6. the frontend exited;
7. the engine process entered an error state;
8. Kubernetes eventually reported DeadlineExceeded.

The logs explicitly showed the unsupported parameter and the invalid file under /var/lib/iscsi/nodes.

This is an important diagnostic lesson:

A Longhorn volume attachment may fail because of a stale iSCSI record belonging to a different, older volume.

The broken record does not always belong to the volume currently being attached. Some iscsiadm operations load the broader node database, so one invalid file can interfere with other targets.

Safely repairing invalid iSCSI node records

Run the following commands directly on the affected Linux worker.

1. Verify the iSCSI service

systemctl status iscsid --no-pager

Check the installed tools:

iscsiadm --version
rpm -q iscsi-initiator-utils

List current node records:

iscsiadm -m node

If this command fails with the same unknown parameter error, the problem is confirmed outside Longhorn.

2. Locate the unsupported parameter

grep -Rns \
  'node.session.sess_reopen_log_freq' \
  /var/lib/iscsi/nodes

Inspect every matching file:

grep -RnsC 3 \
  'node.session.sess_reopen_log_freq' \
  /var/lib/iscsi/nodes

Do not edit the files until they have been backed up.

3. Back up the iSCSI database

Store the backup outside the active iSCSI configuration directory:

mkdir -p /root/iscsi-node-backups

tar -C / \
  -czf "/root/iscsi-node-backups/iscsi-nodes-$(date +%Y%m%d-%H%M%S).tar.gz" \
  var/lib/iscsi/nodes

Confirm the backup:

ls -lh /root/iscsi-node-backups

Why the backup must be outside /var/lib/iscsi/nodes

Do not create files such as:

default.bak
default.old
default.backup

inside the active iSCSI node tree.

The iSCSI tools may attempt to parse those files as active configuration. A perfectly valid backup copy can therefore keep the original error alive even after the primary file has been corrected.

4. Remove only the unsupported line

First review the matching files manually.

Then remove only the obsolete parameter from files named default:

find /var/lib/iscsi/nodes \
  -type f \
  -name default \
  -exec sed -i \
  '/^[[:space:]]*node\.session\.sess_reopen_log_freq[[:space:]]*=/d' \
  {} +

Search again:

grep -Rns \
  'node.session.sess_reopen_log_freq' \
  /var/lib/iscsi/nodes \
  || true

There should be no remaining active occurrences.

5. Validate the iSCSI database

iscsiadm -m node

A successful result should list targets or return without the invalid-parameter error.

Also check active sessions:

iscsiadm -m session

Do not indiscriminately delete active sessions belonging to other Longhorn volumes.

Recovering a stale Longhorn instance manager

Fixing iSCSI may not immediately repair an instance manager that was already in an error state.

Check instance-manager custom resources:

kubectl get instancemanagers.longhorn.io \
  -n longhorn-system \
  -o wide

Inspect the one associated with the target node:

kubectl describe instancemanagers.longhorn.io \
  -n longhorn-system \
  <instance-manager-name>

Also list its pod:

kubectl get pods \
  -n longhorn-system \
  -o wide |
grep instance-manager

After the iSCSI configuration has been corrected, restart the Longhorn manager pod on the affected node so that its DaemonSet recreates the manager process cleanly.

First identify the exact manager pod:

kubectl get pods \
  -n longhorn-system \
  -l app=longhorn-manager \
  -o wide

Delete only the manager pod assigned to the affected node:

kubectl delete pod \
  -n longhorn-system \
  <longhorn-manager-pod>

The DaemonSet should recreate it automatically.

Monitor recovery:

kubectl get pods \
  -n longhorn-system \
  -o wide \
  --watch

Do not delete Longhorn engine, replica or volume custom resources as an initial recovery action.

How a cordoned node can affect recovery

A Kubernetes node may be cordoned intentionally:

kubectl cordon worker-storage-02

A cordoned node can remain Ready, but normal workloads are prevented from being newly scheduled.

Longhorn also has a setting named:

disable-scheduling-on-cordoned-node

Check it with:

kubectl get settings.longhorn.io \
  disable-scheduling-on-cordoned-node \
  -n longhorn-system \
  -o custom-columns='NAME:.metadata.name,VALUE:.value'

When the setting is true, Longhorn avoids scheduling certain storage components or replicas on cordoned nodes.

In the incident described here, the affected node was temporarily uncordoned during controlled maintenance. Longhorn then recreated the missing third replica and the volume progressed from degraded to healthy.

Temporary recovery sequence:

kubectl uncordon worker-storage-02

Wait for Longhorn components to stabilize:

kubectl get pods \
  -n longhorn-system \
  -o wide

After the required component or replica has been created, cordon the node again when necessary:

kubectl cordon worker-storage-02

Important warning

Do not uncordon a node that is physically unstable, under hardware investigation or unsafe for production workloads.

A pod that explicitly uses nodeName can bypass normal scheduler placement logic, but Longhorn’s own scheduling decisions may still be influenced by the cordon setting.

Test the repaired volume with a dedicated pod

Do not immediately restart the production application.

Create a simple pod that mounts the PVC and remains alive long enough for inspection.

apiVersion: v1
kind: Pod
metadata:
  name: pvc-attach-check
  namespace: application
spec:
  nodeName: worker-storage-02
  restartPolicy: Never

  containers:
    - name: check
      image: busybox:1.36.1

      command:
        - /bin/sh
        - -c
        - |
          set -eux
          echo "PVC attached successfully"
          df -h /data
          ls -la /data
          sleep 1800

      volumeMounts:
        - name: persistent-storage
          mountPath: /data

  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: application-cache

Apply it:

kubectl apply -f pvc-attach-check.yaml

Wait for readiness:

kubectl wait \
  --for=condition=Ready \
  pod/pvc-attach-check \
  -n application \
  --timeout=120s

Inspect the pod:

kubectl get pod pvc-attach-check \
  -n application \
  -o wide

Read the logs:

kubectl logs \
  -n application \
  pvc-attach-check

Expected:

PVC attached successfully
Filesystem                Size      Used Available Use% Mounted on
/dev/longhorn/...          30G       ...       ...   ... /data

A successful test confirms:

• CSI attachment works;
• the Longhorn frontend starts;
• the node can connect to the block device;
• kubelet can mount the filesystem;
• the PVC data is readable.

Verify the Longhorn volume after attachment

Check state and robustness:

kubectl get volumes.longhorn.io \
  -n longhorn-system \
  "$PV" \
  -o custom-columns='STATE:.status.state,ROBUSTNESS:.status.robustness,CURRENT-NODE:.status.currentNodeID'

Expected:

STATE      ROBUSTNESS   CURRENT-NODE
attached   healthy      worker-storage-02

List replicas:

kubectl get replicas.longhorn.io \
  -n longhorn-system \
  -o custom-columns='NAME:.metadata.name,VOLUME:.spec.volumeName,NODE:.spec.nodeID,STATE:.status.currentState,FAILED-AT:.spec.failedAt,HEALTHY-AT:.spec.healthyAt'

Filter for the PV:

kubectl get replicas.longhorn.io \
  -n longhorn-system \
  -o custom-columns='NAME:.metadata.name,VOLUME:.spec.volumeName,NODE:.spec.nodeID,STATE:.status.currentState' |
grep "$PV"

When three replicas are configured, a healthy attached volume should normally show three running replicas on eligible storage nodes.

Longhorn logs from the incident showed the manager creating the replica and engine processes, connecting the engine to three replica backends and starting the block-device frontend after reconciliation.

What if the volume attaches but remains degraded?

An attached volume may still show:

ROBUSTNESS=degraded

Check the configured replica count:

kubectl get volumes.longhorn.io \
  -n longhorn-system \
  "$PV" \
  -o jsonpath='Desired replicas: {.spec.numberOfReplicas}{"\n"}'

Count existing replicas:

kubectl get replicas.longhorn.io \
  -n longhorn-system \
  -o json |
jq --arg volume "$PV" \
  '[.items[] | select(.spec.volumeName == $volume)] | length'

A common example is:

Desired replicas: 3
Existing replicas: 2

Describe the Longhorn volume:

kubectl describe volumes.longhorn.io \
  -n longhorn-system \
  "$PV"

Look for:

ReplicaSchedulingFailure
precheck new replica failed: disks are unavailable

Possible causes include:

• eligible node is cordoned;
• Longhorn scheduling disabled on the node;
• disk marked unschedulable;
• disk free-space threshold exceeded;
• disk UUID mismatch;
• node selector or disk tag restrictions;
• replica anti-affinity rules;
• failed storage mount;
• missing instance manager.

In the investigated environment, another storage node had a DiskFilesystemChanged condition and reported zero available Longhorn storage. This meant it was not an eligible destination for the missing replica.

Do not reduce the replica count merely to make the status turn green unless two replicas are an intentional design decision.

Cleanly detach the test volume

After validation, delete the test pod:

kubectl delete pod pvc-attach-check \
  -n application \
  --wait=true

Monitor Longhorn:

kubectl get volumes.longhorn.io \
  -n longhorn-system \
  "$PV" \
  -w

Expected final state:

detached
unknown

robustness=unknown is normally expected when the volume is detached because the engine and replica processes are stopped.

Confirm that no Kubernetes attachment remains:

kubectl get volumeattachments.storage.k8s.io \
  -o custom-columns='NAME:.metadata.name,PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached' |
grep "$PV"

No result normally means the attachment was cleaned up successfully.

Common troubleshooting mistakes

Recreating the pod repeatedly

Deleting and recreating the application pod only produces more attachment attempts. It does not repair iSCSI, the Longhorn engine or the instance manager.

Deleting the PVC

Deleting the PVC can trigger destructive storage operations depending on the reclaim policy. It should never be the first response to an attachment timeout.

Deleting the VolumeAttachment immediately

A VolumeAttachment is evidence of the current CSI state. Deleting it before diagnosing the underlying problem may only cause Kubernetes to recreate it.

It may be appropriate for a genuinely stale attachment after confirming that:

• no pod uses the volume;
• the volume is detached;
• no node has the device mounted;
• the CSI controller cannot clean it up.

It should not be used as the default fix.

Deleting Longhorn replica custom resources

A failed-looking replica may contain the only usable copy of recent data. Allow Longhorn to reconcile it unless the replica is confirmed failed and a safe recovery plan exists.

Leaving backup files inside the iSCSI database

Files named .bak or .old under /var/lib/iscsi/nodes may still be parsed. Store backups outside the active directory.

Restarting every storage component simultaneously

Restarting CSI, Longhorn managers, instance managers, kubelet and iscsid together destroys useful evidence and makes the real cause harder to identify.

Change one layer at a time.

Confusing detached / unknown with data loss

For an unused volume:

STATE=detached
ROBUSTNESS=unknown

is usually normal.

Ignoring node cordon state

A node can be healthy from Kubernetes’ perspective but unavailable for new Longhorn replica scheduling because it is cordoned.

A reusable diagnostic command sequence

The following workflow can be reused whenever a Longhorn PVC fails to attach.

NS=<application-namespace>
PVC=<pvc-name>
POD=<pod-name>

PV=$(kubectl get pvc "$PVC" \
  -n "$NS" \
  -o jsonpath='{.spec.volumeName}')

echo "PVC: $PVC"
echo "PV:  $PV"

kubectl describe pod "$POD" -n "$NS"

kubectl get volumeattachments.storage.k8s.io \
  -o custom-columns='NAME:.metadata.name,PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached'

kubectl get volumes.longhorn.io \
  -n longhorn-system \
  "$PV" \
  -o custom-columns='STATE:.status.state,ROBUSTNESS:.status.robustness,SPEC-NODE:.spec.nodeID,CURRENT-NODE:.status.currentNodeID'

kubectl describe volumes.longhorn.io \
  -n longhorn-system \
  "$PV"

kubectl get pods \
  -n longhorn-system \
  -o wide

kubectl get instancemanagers.longhorn.io \
  -n longhorn-system \
  -o wide

kubectl get replicas.longhorn.io \
  -n longhorn-system \
  -o custom-columns='NAME:.metadata.name,VOLUME:.spec.volumeName,NODE:.spec.nodeID,STATE:.status.currentState' |
grep "$PV"

On the affected node:

systemctl status iscsid --no-pager
iscsiadm -m node
iscsiadm -m session
grep -Rns 'node.session.' /var/lib/iscsi/nodes

This sequence identifies which layer failed before any destructive action is taken.

Preventing future Longhorn attachment failures

Keep iSCSI packages consistent

All Longhorn storage nodes should use compatible versions of:

iscsi-initiator-utils
open-iscsi

After operating-system upgrades, validate:

iscsiadm --version
iscsiadm -m node

Audit stale iSCSI records

Periodically inspect:

/var/lib/iscsi/nodes

Look for:

• obsolete targets;
• files copied from older systems;
• unsupported parameters;
• backup files in the active tree.

Do not delete records blindly while volumes are attached.

Monitor Longhorn node and disk conditions

kubectl get nodes.longhorn.io \
  -n longhorn-system

kubectl describe nodes.longhorn.io \
  -n longhorn-system \
  <node-name>

Alert on:

DiskNotReady
DiskFilesystemChanged
ReplicaSchedulingFailure
Storage Available: 0

Monitor volume robustness

kubectl get volumes.longhorn.io \
  -n longhorn-system \
  -o custom-columns='NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness'

Investigate persistent degraded, faulted or unknown states while volumes are expected to be attached.

Understand the impact of cordoning storage nodes

Before cordoning a Longhorn node:

• verify replica distribution;
• check the configured replica count;
• confirm another eligible storage node exists;
• understand disable-scheduling-on-cordoned-node;
• verify that critical volume engines can run elsewhere.

Preserve logs across reboots

Persistent system logs help diagnose iSCSI and kubelet problems after an unexpected restart.

Check:

journalctl --list-boots

Store journals persistently under:

/var/log/journal

Use dedicated PVC test pods

A minimal BusyBox test pod isolates storage from:

• application startup errors;
• GPU initialization;
• model loading;
• database migrations;
• readiness probes;
• custom entrypoints.

This makes troubleshooting faster and safer.

Frequently asked questions

Why is my Longhorn pod stuck in ContainerCreating?

The pod may already be scheduled, but kubelet cannot start the container until the Longhorn PVC has been attached and mounted.

Check:

kubectl describe pod <pod-name> -n <namespace>

A FailedAttachVolume event confirms that the problem occurs before container startup.

What does VolumeAttachment attached: false mean?

It means Kubernetes created a CSI attachment request, but the CSI driver has not completed the operation successfully.

Inspect:

kubectl get volumeattachment <name> -o yaml

The attachError field often contains the first useful clue.

Is DeadlineExceeded the real cause?

Usually not.

DeadlineExceeded means the CSI operation did not complete within its allowed time. The actual cause is normally found in Longhorn manager, instance-manager or node-level iSCSI logs.

Why can one old iSCSI record break a different volume?

Some iscsiadm commands read multiple stored node records. A malformed record for an old target may cause the command to fail before the current target can be processed.

Should I restart iscsid?

Only after checking whether the node has active iSCSI sessions.

Use:

iscsiadm -m session

Restarting iscsid on a node with active Longhorn volumes can disrupt storage. Correcting the invalid record and validating iscsiadm -m node may be sufficient.

Why is the Longhorn volume detached with robustness unknown?

This is normally expected when no workload uses the volume. Longhorn cannot report active replica health through an engine that is not running.

Why did the volume become degraded after attachment?

The configured number of replicas may be higher than the number Longhorn could schedule.

Check:

kubectl get volumes.longhorn.io \
  -n longhorn-system \
  <volume-name> \
  -o jsonpath='{.spec.numberOfReplicas}{"\n"}'

Then inspect replica and disk scheduling conditions.

Can a cordoned node still run a pod using nodeName?

A pod explicitly assigned with nodeName may be created on a cordoned node because it bypasses normal scheduler placement.

However, Longhorn’s own storage scheduling can still reject that node when disable-scheduling-on-cordoned-node=true.

Should I delete a stale instance-manager pod?

A failed instance-manager pod may be recreated by Longhorn after the node and manager reconcile.

First fix the underlying issue, such as iSCSI configuration or node readiness. Restarting the Longhorn manager pod on the affected node may then trigger a clean instance-manager recreation.

Avoid deleting instance-manager custom resources or Longhorn replicas without understanding their state.

Conclusion

A Longhorn PVC failed to attach incident should be investigated from the outside inward:

1. Confirm that the pod is blocked by storage.
2. Identify the PVC and PV.
3. Inspect the Kubernetes VolumeAttachment.
4. Check the Longhorn volume state.
5. Verify the manager and instance-manager on the target node.
6. Read the Longhorn engine and frontend errors.
7. Validate the node’s iSCSI database.
8. Repair only the confirmed invalid configuration.
9. Reconcile Longhorn components.
10. Test the volume with a minimal pod.
11. Verify replica health before restarting the application.

In the case described here, the visible Kubernetes error was a CSI timeout. The actual root cause was a stale iSCSI record containing an unsupported parameter. A failed Longhorn instance manager and a cordoned node then complicated recovery and replica rebuilding.

The most important lesson is not to treat FailedAttachVolume as a reason to delete storage objects.

The PVC, PV, VolumeAttachment, Longhorn volume, engine, instance manager, replicas and host iSCSI configuration each represent a separate diagnostic layer. Working through those layers methodically makes it possible to recover the volume safely while preserving both data and evidence.