This article explains how to install the NetApp Trident CSI driver on a Harvester cluster. Once it's installed, VMs running in Harvester can store their volumes on a NetApp ONTAP storage system.

NetApp storage is added alongside Longhorn; it does not replace it. You can use any combination of these protocols:

• NFS (ontap-nas)
• iSCSI (ontap-san, sanType: iscsi)
• NVMe/TCP (ontap-san, sanType: nvme)
• Fibre Channel (ontap-san, sanType: fcp)

This procedure was tested with Harvester v1.9.0 and Trident 26.06.1 (Helm chart 100.2606.1), using ONTAP 9.16.1. With NFS, iSCSI and NVMe/TCP, the following worked on NetApp-backed volumes: booting VMs, live migration, online volume expansion, and Harvester VM snapshots. Trident also supports Fibre Channel, and the multipath configuration below covers FC LUNs. However, FC was not tested for this article.

Prerequisites​

• A Harvester cluster and its KubeConfig. To download it, open the Harvester UI, click Support in the lower left, then click Download KubeConfig.
• A workstation with kubectl and helm that can reach the Harvester cluster.
• An ONTAP SVM with:
  • A management LIF that the Harvester nodes can reach.
  • Data LIFs for each protocol you plan to use: NFS, iSCSI, NVMe/TCP and/or FC. NVMe/TCP requires the NVMe_oF license on the ONTAP system.
  • An ONTAP login for Trident with the http and ontapi applications enabled. The examples in this article use limitAggregateUsage, which requires a cluster-scoped login with the admin role. In that case, managementLIF is the cluster management LIF. If you use an SVM-scoped vsadmin login instead, set managementLIF to the SVM management LIF and remove limitAggregateUsage from the backends. See User permissions in the Trident documentation.
• For FC only: FC HBAs in every Harvester node that mounts NetApp volumes, and fabric zoning between those HBAs and the SVM's FC LIFs.

Network design​

Trident's node plugin runs in the host network namespace, so the host's routing table decides which NIC carries storage traffic.

How Trident discovers data LIFs differs by protocol:

• iSCSI / NVMe/TCP: Trident ignores dataLIF on SAN backends and logs Specifying data LIF is no longer supported for SAN backends. It asks ONTAP for every SAN LIF in the SVM, and each node logs in to all of them, so multipath works automatically. You only need managementLIF and svm.
• NFS: dataLIF takes a single address. If you leave it out, Trident uses the first NFS LIF it finds in the SVM. Set it explicitly to choose the network that carries NFS traffic.
• FC: traffic runs over the FC fabric, not the IP network, so the rest of this section does not apply. Only the management LIF must be reachable over IP.

Optional: a dedicated storage interface with HostNetworkConfig​

This takes two steps: create a cluster network on the storage NICs, then give each host an IP address on it with a HostNetworkConfig.

Step 1: create the cluster network. A cluster network groups the storage NICs into a bridge on each node. See Create a New Cluster Network for details.

1. In the Harvester UI, go to Networks > ClusterNetworks/Configs and click Create. Name it, for example cn-storage.
2. Click Create Network Config on the new cluster network.
3. On the Node Selector tab, select the nodes that mount NetApp volumes. Select all nodes does not work if the cluster has a witness node. Select nodes by name, or by a label such as netapp-trident=enabled (see the note further down).
4. On the Uplink tab, select the storage NICs, the bond mode, and the MTU. The MTU must match the switch ports and the ONTAP LIFs, for example 9000 for jumbo frames.
5. Click Save.

The switch ports for these NICs must be trunk ports that carry the storage VLAN tagged.

Step 2: give the hosts an IP address on it. On its own, a cluster network carries only VM traffic, because the hosts have no IP address on it. A HostNetworkConfig (Networks > Host Networks) creates a VLAN sub-interface on the cluster network's bridge on each node and gives it an IP address. Traffic to the ONTAP data LIFs on that subnet then leaves through that interface instead of the management network. The ONTAP data LIFs must be on the same VLAN and subnet.

Example with static addresses on VLAN 150:

apiVersion: network.harvesterhci.io/v1beta1
kind: HostNetworkConfig
metadata:
  name: netapp-storage
spec:
  clusterNetwork: cn-storage
  vlanID: 150
  mode: static
  nodeSelector:
    matchLabels:
      netapp-trident: enabled
  ips:
    <node-1>: 192.168.150.11/24
    <node-2>: 192.168.150.12/24

Check the result on a node with ip addr show cn-storage-br.150. Each node's state is reported in the resource's status.nodeStatus.

For iSCSI, you can add a second cluster network (or a second VLAN) with its own HostNetworkConfig and LIFs, so multipath has two independent paths.

1. Configure multipathd (iSCSI and FC)​

iSCSI and FC LUNs need multipathd, which is not enabled on Harvester by default. Both protocols present NetApp LUNs with vendor NETAPP and product LUN, so the same configuration covers them. The configuration below lets multipathd claim only NetApp LUNs. Every other device is blacklisted, so Longhorn and the local disks are never touched. Harvester's own /etc/multipath/conf.d/99-longhorn.conf is left as is.

The following CloudInit resource writes /etc/multipath/conf.d/98-netapp.conf and enables multipathd on every node. It is re-applied on every boot. Save it as 01-cloudinit-netapp-multipathd.yaml:

# Persists across reboots: writes /etc/multipath/conf.d/98-netapp.conf (multipath ONLY claims NETAPP LUNs)
apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: netapp-multipathd
spec:
  matchSelector: {}
  filename: 98_netapp_multipathd.yaml
  contents: |
    name: "enable multipathd for NetApp Trident iSCSI/FC (NETAPP LUNs only)"
    stages:
      initramfs:
        - files:
            - path: /etc/multipath/conf.d/98-netapp.conf
              permissions: 0644
              owner: 0
              group: 0
              content: |
                defaults {
                    user_friendly_names yes
                    find_multipaths no
                    # lets libmpathpersist (qemu-pr-helper) track SCSI-3 PR keys across paths, for shared disks (e.g. WSFC)
                    reservation_key file
                    # don't create kpartx maps for partitions that guests create on the LUNs
                    skip_kpartx yes
                }
                blacklist {
                    device {
                        vendor ".*"
                        product ".*"
                    }
                }
                blacklist_exceptions {
                    device {
                        vendor "NETAPP"
                        product "LUN.*"
                    }
                }
      network:
        - name: "enable and start multipathd"
          systemctl:
            enable:
              - multipathd
            start:
              - multipathd

kubectl apply -f 01-cloudinit-netapp-multipathd.yaml

The CloudInit resource takes effect the next time each node boots. To apply it now without rebooting, run the following on each node as root. It writes the same file and starts multipathd, and stops at the first failure:

set -e
mkdir -p /etc/multipath/conf.d
cat > /etc/multipath/conf.d/98-netapp.conf <<'EOF'
defaults {
    user_friendly_names yes
    find_multipaths no
    reservation_key file
    skip_kpartx yes
}
blacklist {
    device {
        vendor ".*"
        product ".*"
    }
}
blacklist_exceptions {
    device {
        vendor "NETAPP"
        product "LUN.*"
    }
}
EOF
systemctl enable --now multipathd
multipath -ll   # must not list any local disk

Check that every node has a unique iSCSI initiator name. Nodes cloned from the same image can end up with the same name in /etc/iscsi/initiatorname.iscsi. ONTAP then treats those nodes as a single host. If you change a node's IQN, restart that node's trident-node-linux pod afterwards. Trident caches the IQN, and with a stale one volume staging fails with no devices present yet.

2. Install the Trident operator​

helm repo add netapp-trident https://netapp.github.io/trident-helm-chart
helm repo update
helm search repo netapp-trident --versions | head

helm upgrade --install trident netapp-trident/trident-operator --version 100.2606.1 \
  --namespace trident --create-namespace --wait

Wait until the trident-controller pod and one trident-node-linux pod per node are Running:

kubectl -n trident get pods -o wide

kubectl label node <node-1> <node-2> netapp-trident=enabled --overwrite

helm upgrade --install trident netapp-trident/trident-operator --version 100.2606.1 \
  --namespace trident --create-namespace --wait \
  --set tridentNodePluginNodeSelector.netapp-trident=enabled

3. Create the ONTAP credentials Secret​

kubectl -n trident create secret generic netapp-creds \
  --from-literal=username=<ontap user> \
  --from-literal=password='<ontap password>'

4. Create the backends​

Create one TridentBackendConfig per protocol you use; you can drop any you don't need.

The examples below use these settings:

• Thin provisioning: spaceReserve: none and snapshotReserve: "0".
• limitAggregateUsage: "80%": Trident stops creating new volumes once the aggregate passes 80% used.

If several Kubernetes or Harvester clusters share one SVM, give each cluster a unique backendName and set storagePrefix (for example storagePrefix: hv1_). That keeps volume names from colliding and makes each cluster's volumes easy to identify in ONTAP.

For Fibre Channel, copy the iSCSI backend below, then change name and backendName (for example to netapp-fc) and set sanType: fcp. Then add a matching StorageClass in step 5 with storagePools: "netapp-fc:.*". FC was not tested for this article.

Save as 03-backends.yaml, replacing the <...> values:

# NFS
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-nas
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-nas
  backendName: netapp-nas
  managementLIF: <management LIF IP>
  dataLIF: <NFS data LIF IP>          # optional; defaults to the first NFS LIF in the SVM
  svm: <svm name>
  autoExportPolicy: true
  autoExportCIDRs: ["<storage network CIDR>"]
  nfsMountOptions: "nfsvers=4.1"
  limitAggregateUsage: "80%"
  credentials:
    name: netapp-creds
  defaults:
    spaceReserve: none
    snapshotReserve: "0"
---
# iSCSI (all iSCSI LIFs in the SVM are discovered; no dataLIF)
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-iscsi
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-san
  sanType: iscsi
  backendName: netapp-iscsi
  managementLIF: <management LIF IP>
  svm: <svm name>
  limitAggregateUsage: "80%"
  credentials:
    name: netapp-creds
  defaults:
    spaceReserve: none
    snapshotReserve: "0"
---
# NVMe/TCP (all NVMe LIFs in the SVM are discovered; no dataLIF)
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-nvme
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-san
  sanType: nvme
  backendName: netapp-nvme
  managementLIF: <management LIF IP>
  svm: <svm name>
  limitAggregateUsage: "80%"
  credentials:
    name: netapp-creds
  defaults:
    spaceReserve: none
    snapshotReserve: "0"

kubectl apply -f 03-backends.yaml
kubectl -n trident get tridentbackendconfigs

Every backend must report PHASE Bound and STATUS Success. If one doesn't, kubectl -n trident describe tridentbackendconfig <name> shows the error.

5. Create the StorageClasses and VolumeSnapshotClass​

Trident StorageClasses have no sanType parameter. A StorageClass with only backendType: ontap-san can place a volume on any SAN backend: iSCSI, NVMe/TCP or FC. To prevent that, pin each class to its backend with storagePools: "<backendName>:.*".

Save as 04-storageclasses.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-nfs
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-nas"
  storagePools: "netapp-nas:.*"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-iscsi
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-san"
  storagePools: "netapp-iscsi:.*"
  fsType: "ext4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-nvme
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-san"
  storagePools: "netapp-nvme:.*"
  fsType: "ext4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: trident-snapshotclass
driver: csi.trident.netapp.io
deletionPolicy: Delete

kubectl apply -f 04-storageclasses.yaml

Live migration requires ReadWriteMany volumes. CDI already knows Trident's defaults:

• ontap-san (iSCSI, NVMe/TCP and FC): ReadWriteMany with volumeMode: Block.
• ontap-nas: ReadWriteMany with volumeMode: Filesystem.

So these StorageClasses don't need the cdi.harvesterhci.io/storageProfileVolumeModeAccessModes annotation.

6. Register Trident for Harvester VM snapshots​

Harvester needs to know which VolumeSnapshotClass to use for Trident volumes.

1. In the Harvester UI, go to Advanced > Settings, find csi-driver-config, and select ⋮ > Edit Setting.
2. Click Add, then fill in the fields:
   • Provisioner: csi.trident.netapp.io
   • Volume Snapshot Class Name: trident-snapshotclass
3. Save.

Leave Backup Volume Snapshot Class Name empty. In Harvester v1.9, VM backups only work with Longhorn volumes. VM snapshots of Trident volumes work. See Third-Party Storage Support.

The resulting entry in the setting's JSON value is:

"csi.trident.netapp.io": {
  "volumeSnapshotClassName": "trident-snapshotclass",
  "backupVolumeSnapshotClassName": ""
}

7. Use it​

The new StorageClasses now appear when you create a volume, a VM image or a VM in the Harvester UI. A quick check:

1. Create a VM whose root disk uses netapp-iscsi, netapp-nvme or netapp-nfs.
2. Start the VM, live-migrate it to another node, and take a VM snapshot.
3. On a node running the VM, multipath -ll (iSCSI) or ls /sys/class/nvme-subsystem (NVMe) shows the NetApp devices.

Troubleshooting​

• Backends go offline after the ONTAP system was unreachable. If they don't recover on their own, restart the controller: kubectl -n trident rollout restart deploy/trident-controller.
• Deleted volumes still use space on ONTAP. ONTAP keeps deleted volumes in the volume recovery queue for 12 hours. While a deleted FlexClone (such as a VM disk cloned from an image) sits in the queue, its parent snapshot stays busy, so the parent volume and its VolumeSnapshot can't be deleted. To free them immediately, run on ONTAP: set -privilege advanced; volume recovery-queue purge-all -vserver <svm name>.
• multipath -ll lists local disks. multipathd was started without the NetApp-only configuration. Put 98-netapp.conf in place, flush the unused maps with multipath -F, and restart multipathd.
• iSCSI volumes fail to stage with no devices present yet. Check for duplicate initiator names across nodes, and for a stale IQN in Trident (see step 1).

References​

• NetApp Trident documentation
• Deploy the Trident operator using Helm
• ONTAP SAN driver and ONTAP NAS driver
• Harvester third-party storage support

Harvester v1.7.0 introduced the kubevirt-migration setting, which exposes the cluster-wide live migration configuration that previously could only be changed by editing the kubevirt object directly.

If you customized spec.configuration.migrations on the kubevirt object before upgrading, that customization is at risk. This article explains why and provides a script that converts it into the setting so it is preserved.

Who is affected​

You are affected if both of the following are true:

• You are running a version earlier than v1.7.0 and you changed spec.configuration.migrations on the kubevirt object, for example to enable allowAutoConverge.
• You plan to upgrade to v1.7.0 or later.

You are also affected if you have already upgraded to v1.7.x and have not yet changed the kubevirt-migration setting. In that case the configuration is still intact and the same script applies.

Check whether the kubevirt object carries a configuration:

kubectl get kubevirt kubevirt -n harvester-system -o jsonpath='{.spec.configuration.migrations}' | jq

If the command returns nothing, your cluster uses the KubeVirt defaults and no action is required.

What happens without this procedure​

The upgrade creates the kubevirt-migration setting without a value, so the Harvester UI displays the default values. The kubevirt object keeps your configuration, so live migration continues to behave as before.

The two are now inconsistent, and the inconsistency is not visible unless you compare them. Harvester writes the whole value of the setting to the kubevirt object, so the first time the setting is saved, all fields are written at once, including the fields that you did not change. Your configuration is replaced by the defaults, without warning.

In practice this means that changing an unrelated field such as parallelMigrationsPerCluster in the UI silently turns allowAutoConverge back off. Migrations of write-heavy virtual machines then start failing to converge, which is often noticed only during the node drains of a later upgrade.

Procedure​

Run the following script from a machine with kubectl access to the cluster. jq is required.

Run it before you upgrade to v1.7.0 or later. If you already upgraded to v1.7.x and have not yet changed the setting, run it now.

#!/bin/bash
set -euo pipefail

SETTING_NAME="kubevirt-migration"

migrations=$(kubectl get kubevirt kubevirt -n harvester-system -o json | jq -c '.spec.configuration.migrations // empty')
if [ -z "$migrations" ]; then
  echo "The kubevirt object has no migration configuration. Nothing to preserve."
  exit 0
fi

# nodeDrainTaintKey and network cannot be configured through the setting:
# nodeDrainTaintKey is used by the upgrade process, and network is owned by
# the vm-migration-network setting. The webhook rejects a value containing them.
value=$(echo "$migrations" | jq -c 'del(.nodeDrainTaintKey, .network)')
if [ "$value" = "{}" ]; then
  echo "The migration configuration only contains fields that the setting does not manage. Nothing to preserve."
  exit 0
fi

if setting_json=$(kubectl get settings.harvesterhci.io "$SETTING_NAME" -o json 2>/dev/null); then
  current_value=$(echo "$setting_json" | jq -r '.value // empty')
  if [ -n "$current_value" ]; then
    echo "The $SETTING_NAME setting already has a value. Nothing to do."
    exit 0
  fi

  echo "Setting $SETTING_NAME to: $value"
  kubectl patch settings.harvesterhci.io "$SETTING_NAME" --type merge \
    -p "$(jq -n --arg v "$value" '{"value": $v}')"
else
  echo "Creating $SETTING_NAME with: $value"
  kubectl apply -f - <<EOF
apiVersion: harvesterhci.io/v1beta1
kind: Setting
metadata:
  name: $SETTING_NAME
value: '$value'
EOF
fi

echo "Result:"
kubectl get settings.harvesterhci.io "$SETTING_NAME" -o yaml

The script does not write to the kubevirt object, so it cannot change how the cluster performs live migrations. It is idempotent: it does nothing if there is no configuration to preserve, or if the setting already has a value.

Verification​

Confirm that the setting now holds your configuration:

kubectl get settings.harvesterhci.io kubevirt-migration -o jsonpath='{.value}' | jq

Confirm that the kubevirt object is unchanged:

kubectl get kubevirt kubevirt -n harvester-system -o jsonpath='{.spec.configuration.migrations}' | jq

After you upgrade, the two remain consistent, and the Harvester UI shows the configuration that the cluster is actually using.

After the upgrade​

The kubevirt-migration setting is the source of truth. Harvester writes it to the kubevirt object whenever the setting changes.

Do not edit spec.configuration.migrations on the kubevirt object directly. Any change is overwritten the next time the setting is reconciled. Use the setting instead, either in the Harvester UI under Advanced > Settings, or with kubectl.

For the list of supported fields and their defaults, see kubevirt-migration in the Harvester documentation.

For background on the individual options and on per-virtual-machine migration policies, see VM Live Migration Policy and Configuration.

A symlink-following vulnerability was found in KubeVirt's virt-handler migration proxy.

During live migration, virt-handler dials Unix sockets inside the target virt-launcher pod using /proc/<pid>/root/<path> via net.Dial(), without symlink protection. These socket paths reside in QEMU-owned directories that are writable by the virt-launcher user.

An attacker with namespace edit and pods/exec permissions can replace a migration proxy socket with a symlink pointing to the host CRI-O socket. Because virt-handler runs as root in the host mount namespace, absolute symlink targets resolve against the host filesystem. The bidirectional io.Copy proxy then relays attacker-controlled bytes to the container runtime, enabling full node compromise.

Affected Versions​

All currently supported Harvester versions are affected, including 1.6.1 and earlier, 1.7.3 and earlier, and 1.8.2 and earlier.

Your cluster is affected by this vulnerability, if the following command returns at least one virt-launcher pods:

kubectl get po -lkubevirt.io=virt-launcher -A

Fixed Versions​

A fix is not yet available. SUSE is actively working on a resolution, targeting Harvester 1.9.

Meanwhile, you can use the following workaround to protect your clusters' virt-launcher pods against exploitation.

Workaround​

Use a ValidatingAdmissionPolicy to deny pods/exec and pods/attach operations targeting virt-launcher pods.

Apply the following resources to your Harvester cluster:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: deny-virt-launcher-exec
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   [""]
      apiVersions: ["v1"]
      operations:  ["CONNECT"]
      resources:   ["pods/exec", "pods/attach"]
  matchConditions:
    - name: is-virt-launcher
      expression: >
        request.name.startsWith("virt-launcher-")
  validations:
    - expression: >
        request.userInfo.username == "system:admin" ||
        request.userInfo.username.startsWith("system:serviceaccount:harvester-system")
      messageExpression: "'CVE-2026-13622 mitigation. Rejected exec/attach request from ' + request.userInfo.username"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: deny-virt-launcher-exec-binding
spec:
  policyName: deny-virt-launcher-exec
  validationActions: [Deny]

This policy blocks exec and attach requests to virt-launcher pods from non-admin and non-Harvester service accounts, to prevent an attacker from replacing the migration proxy sockets with symlinks.

kubectl delete validatingadmissionpolicybinding deny-virt-launcher-exec-binding
kubectl delete validatingadmissionpolicy deny-virt-launcher-exec

References​

• https://github.com/advisories/GHSA-4336-pf6g-hp94
• https://www.suse.com/security/cve/CVE-2026-13622.html

Enabling the Longhorn V2 Data Engine can cause kernel workqueue lockups on some Harvester nodes, leading to system instability. In severe cases, rke2-server terminates, Longhorn stops processing operations, and affected nodes require recovery.

This issue typically occurs when SPDK is pinned to dedicated CPUs, but the Linux kernel continues running IRQ handlers or unbound workqueue workers on those same CPUs.

Affected Scenario​

This issue can affect Harvester clusters that meet all of the following conditions:

• The Longhorn V2 Data Engine is enabled.
• The Longhorn V2 instance-manager pod runs SPDK (spdk_tgt), which busy-polls the CPUs specified in the Longhorn V2 Data Engine CPU mask.
• Host IRQ handlers or unbound kernel workqueues are permitted to run on those same SPDK-designated CPUs.

While the default Longhorn V2 CPU mask is 0x3 (maps to CPUs 0 and 1), this issue can occur with any CPU mask if kernel IRQs or unbound workqueues share the CPUs used by spdk_tgt.

This issue primarily affects bare-metal Harvester nodes running the Longhorn V2 Data Engine. Reproducibility varies depending on hardware and workload patterns.

Symptoms​

Affected nodes may exhibit one or more of the following symptoms:

• Kernel logs repeatedly report BUG: workqueue lockup.
• The lockup message references a CPU core assigned to the Longhorn V2 Data Engine CPU mask.
• rke2-server becomes unstable or terminates without a clear user-visible reason.
• Longhorn volumes, engines, replicas, or instance managers stop progressing.
• Interactive user commands, such as sudo -i, experience severe latency before opening a shell.
• The node becomes unstable or unhealthy.

Example of kernel log records:

Jul 27 07:27:55 hp-114-tink-system kernel: BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 2117s!
Jul 27 07:27:55 hp-114-tink-system kernel: Showing busy workqueues and worker pools:
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events: flags=0x0
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=7 refcnt=8
Jul 27 07:27:55 hp-114-tink-system kernel:     in-flight: 2935011:output_poll_execute ,32008:drm_fb_helper_damage_work drm_fb_helper_damage_work
Jul 27 07:27:55 hp-114-tink-system kernel:     pending: vmstat_shepherd, switchdev_deferred_process_work, rht_deferred_worker, key_garbage_collector
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events_unbound: flags=0x2
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 98: cpus=0-23 node=1 flags=0x4 nice=0 active=4 refcnt=8
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events_power_efficient: flags=0x80
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5
Jul 27 07:27:55 hp-114-tink-system kernel:     pending: neigh_managed_work, neigh_periodic_work, gc_worker [nf_conntrack], check_lifetime
Jul 27 07:27:55 hp-114-tink-system kernel: pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=2117s workers=5 idle: 1025200 1541794
Jul 27 07:27:55 hp-114-tink-system kernel: task:kworker/0:0     state:R  running task     stack:0     pid:32008 tgid:32008 ppid:2      flags:0x00004000
Jul 27 07:27:55 hp-114-tink-system kernel: Workqueue: events drm_fb_helper_damage_work
Jul 27 07:28:12 hp-114-tink-system rke2[2909320]: time="2026-07-27T07:28:12Z" level=warning msg="Proxy error: write failed: write tcp 127.0.0.1:9345->127.0.0.1:46562: write: connection reset by peer"

Root Cause​

The Longhorn V2 Data Engine relies on SPDK, whose reactor threads busy-poll on the CPUs designated by the engine's CPU mask. This continuous polling is expected behavior for high-performance storage processing.

The issue occurs when normal host kernel work continues running on those same CPUs, including the following:

• Network or storage IRQs target SPDK CPUs.
• Unbound kernel workqueues retain SPDK CPUs in /sys/devices/virtual/workqueue/cpumask.
• Per-workqueue CPU masks allow existing workers to run on SPDK CPUs.

When SPDK consumes 100% of these CPU cores, pending kernel tasks are starved, eventually triggering the kernel workqueue lockup detector. While isolated lockup warnings may not always result in an unrecoverable node, continuous task starvation cascades into network, RKE2, kubelet, and Longhorn control-plane failure.

To prevent this, all IRQs and kernel workqueues must be explicitly re-routed away from the CPUs defined in the Longhorn V2 CPU mask (for example, CPUs 0 and 1 when using the default mask 0x3).

Confirming the Issue​

Run the following checks on each affected Harvester node.

1. Check Kernel Logs​

journalctl -k --since "24 hours ago" | grep -E "BUG: workqueue lockup|soft lockup"

Look for the CPU reported in messages such as pool cpus=0, pool cpus=1, kworker/0, or kworker/1.

2. Check the Longhorn V2 CPU Mask​

kubectl -n longhorn-system get settings.longhorn.io data-engine-cpu-mask -o jsonpath='{.value}{"\n"}'

The value may be data-engine-specific. For example:

{"v2":"0x3"}

Some Longhorn versions support CPU-list input for this setting. If the value is already a CPU list, use that list directly in the checks below.

Convert the mask to CPU IDs:

python3 - <<'PY'
mask = int("0x3", 0)
print(",".join(str(cpu) for cpu in range(mask.bit_length()) if mask & (1 << cpu)))
PY

For 0x3, the output is:

0,1

3. Check SPDK Placement​

SPDK_PIDS=$(pgrep -f '[s]pdk_tgt' || true)

for pid in ${SPDK_PIDS}; do
  ps -T -p "${pid}" -o pid,tid,psr,stat,pcpu,comm,args
done

This command matches the full command line because the SPDK process or its threads may appear as reactor_<cpu> in the kernel process name. In that case, pgrep -x spdk_tgt can return empty even when spdk_tgt is running.

Example output for the default CPU mask 0x3:

    PID     TID PSR STAT %CPU COMMAND         COMMAND
2913161 2913161   0 RLl  99.9 reactor_0       spdk_tgt -L all --mem-size 2048 -m 0x3
2913161 2913166  21 SLl   0.0 dpdk-intr       spdk_tgt -L all --mem-size 2048 -m 0x3
2913161 2913220   1 RLl  99.9 reactor_1       spdk_tgt -L all --mem-size 2048 -m 0x3

The PSR column shows the CPU that each thread is running on. In the example above, reactor_0 is running on CPU 0 and reactor_1 is running on CPU 1, so the SPDK CPU list is 0,1.

To check whether running or blocked kernel workers are currently on the SPDK CPUs, use the following more targeted command. Replace SPDK_CPUS with the comma-separated CPU IDs from the reactor_* rows.

SPDK_CPUS="0,1"

ps -eLo pid,tid,psr,stat,pcpu,wchan:30,comm,args | \
  awk -v cpus="${SPDK_CPUS}" '
    BEGIN {
      printf "%7s %7s %3s %-5s %5s %-30s %-16s %s\n", "PID", "TID", "PSR", "STAT", "%CPU", "WCHAN", "COMMAND", "ARGS"
      split(cpus, cpu_list, ",")
      for (i in cpu_list) {
        spdk_cpu[cpu_list[i]] = 1
      }
    }
    NR > 1 && spdk_cpu[$3] && $4 ~ /[RD]/ && ($7 ~ /^kworker\// || $7 ~ /^ksoftirqd\//) {
      print
    }
  '

On nodes that are not affected, this command normally prints only the header line. If the header line is removed from the command, empty output is expected. This means there are no running or uninterruptible kworker/* or ksoftirqd/* threads on the SPDK CPUs at that moment.

Example output showing kernel work on an SPDK CPU:

    PID     TID PSR STAT   %CPU WCHAN                          COMMAND          ARGS
     17      17   0 R     0.0 -                              ksoftirqd/0     [ksoftirqd/0]
  32008   32008   0 R     0.0 -                              kworker/0:0+eve [kworker/0:0+events]
2935011 2935011   0 D     0.0 mgag200_ddc_algo_bit_data_pre_ kworker/0:2+eve [kworker/0:2+events]
4012518 4012518   0 R     0.0 -                              kworker/0:3+mm_ [kworker/0:3+mm_percpu_wq]

In this example, the PSR value is 0, and the process names are ksoftirqd/0 and kworker/0:*. This means kernel softirq and workqueue threads are running or blocked on CPU 0.

If the SPDK reactor threads are running on the CPUs reported in the workqueue lockup, or if the targeted command shows kworker/* or ksoftirqd/* activity on the SPDK CPUs, continue with the IRQ and workqueue checks.

4. Check IRQ Affinity​

The following example checks CPUs 0 and 1 because the reactor_* rows above are running on those CPUs. Replace SPDK_CPUS with the CPU IDs used in your environment.

SPDK_CPUS="0,1"

cpu_list_overlaps() {
  python3 - "$1" "$2" <<'PY'
import sys

target = {int(cpu) for cpu in sys.argv[1].split(",") if cpu}
seen = set()

for part in sys.argv[2].split(","):
    part = part.strip()
    if not part:
        continue
    if "-" in part:
        start, end = map(int, part.split("-", 1))
        seen.update(range(start, end + 1))
    else:
        seen.add(int(part))

sys.exit(0 if target & seen else 1)
PY
}

for irqdir in /proc/irq/[0-9]*; do
  irq=${irqdir##*/}
  eff=$(cat "${irqdir}/effective_affinity_list" 2>/dev/null || true)
  conf=$(cat "${irqdir}/smp_affinity_list" 2>/dev/null || true)

  if cpu_list_overlaps "${SPDK_CPUS}" "${eff}"; then
    echo "IRQ=${irq} configured=${conf} effective=${eff}"
    grep -w "^ *${irq}:" /proc/interrupts 2>/dev/null || true
  fi
done

Example output:

IRQ=108 configured=0-5,12-17 effective=0
 108:   25762242          0 ... IR-PCI-MSIX-0000:04:00.0    3-edge      netboot-TxRx-3
IRQ=109 configured=0-5,12-17 effective=1
 109:          0   15555565 ... IR-PCI-MSIX-0000:04:00.0    4-edge      netboot-TxRx-4
IRQ=120 configured=0-5,12-17 effective=0
 120:   10898692          0 ... IR-PCI-MSIX-0000:04:00.0   15-edge      netboot-TxRx-15
IRQ=121 configured=0-5,12-17 effective=1
 121:          0    8135324 ... IR-PCI-MSIX-0000:04:00.0   16-edge      netboot-TxRx-16
IRQ=92 configured=0 effective=0
  92:    2242087          0 ... IR-PCI-MSIX-0000:08:00.0   13-edge      nvme0q13
IRQ=93 configured=1 effective=1
  93:          0     448454 ... IR-PCI-MSIX-0000:08:00.0   14-edge      nvme0q14

The effective value shows where the IRQ is actually running. In this example, NIC queues and NVMe queues are effectively landing on CPUs 0 and 1, which are the default SPDK reactor CPUs for mask 0x3. If device IRQs, especially high-traffic NIC or storage IRQs, are effectively landing on SPDK CPUs, the node is at risk.

5. Check Kernel Workqueue Masks​

cat /sys/devices/virtual/workqueue/cpumask

for f in /sys/devices/virtual/workqueue/*/cpumask; do
  echo "${f}: $(cat "${f}")"
done

Example output:

/sys/devices/virtual/workqueue/blkcg_punt_bio/cpumask: ffffff
/sys/devices/virtual/workqueue/ib-comp-unb-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/iscsi_conn_cleanup/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-auth-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-delete-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-reset-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/scsi_tmf_0/cpumask: ffffff
/sys/devices/virtual/workqueue/writeback/cpumask: ffffff

On a 24-CPU node, ffffff means the workqueue can run on CPUs 0-23. If SPDK uses the default CPU mask 0x3, CPUs 0 and 1 are included in this workqueue mask. If the global or per-workqueue masks include the SPDK CPUs, unbound kernel work may still run on the SPDK CPUs.

Preferred Risk-Reduction Setting​

Longhorn added the data-engine-cpu-isolation-enabled setting to reduce the chance of this issue. When enabled for the V2 Data Engine, the Longhorn V2 instance-manager:

• Persists the SPDK CPU mask under /var/lib/longhorn/instance-manager/v2/spdk_cpu_mask on the host.
• Programs /proc/irq/*/smp_affinity to the inverse of the SPDK CPU mask.
• Writes the same inverse mask to /sys/devices/virtual/workqueue/cpumask.
• Updates per-workqueue CPU masks when possible.
• Reconciles stale affinity state on the next instance-manager restart if the setting is later disabled.

This setting steers IRQs and unbound workqueues away from SPDK CPUs. In some environments, this may stop the workqueue lockup messages entirely. In others, it may only reduce how often they occur because CPU-bound or per-CPU kernel workers such as kworker/0:* are tied to a specific CPU and cannot be moved by the unbound workqueue CPU mask.

Use this setting when the Longhorn version bundled with Harvester includes it. If the Longhorn setting exists but Harvester does not expose it in the UI, you can still configure it through the Longhorn setting resource. For Harvester versions earlier than v1.9.0, the bundled Longhorn version does not include the setting, so use the manual workaround in the next section.

If the setting exists in your Longhorn version, you can check it with:

kubectl -n longhorn-system get settings.longhorn.io data-engine-cpu-isolation-enabled

Enable it for the V2 Data Engine:

kubectl -n longhorn-system patch settings.longhorn.io data-engine-cpu-isolation-enabled \
  --type=merge \
  -p '{"value":"{\"v2\":\"true\"}"}'

Then wait for the V2 instance-manager pods to be recreated after all V2 volumes are detached.

Verify the instance-manager log:

kubectl -n longhorn-system logs <v2-instance-manager-pod> | \
  grep -E "Setting IRQ affinity|Setting workqueue cpumask|Applied IRQ affinity|Applied global workqueue cpumask"

Example expected messages:

Setting IRQ affinity to exclude SPDK CPUs
Applied IRQ affinity mask
Setting workqueue cpumask to exclude SPDK CPUs
Applied global workqueue cpumask
Applied per-workqueue cpumask

Manual Workaround for Versions Without the Longhorn Setting​

Use this workaround for Harvester versions earlier than v1.9.0, where the bundled Longhorn version does not include data-engine-cpu-isolation-enabled. For later versions, first check whether the Longhorn setting exists and prefer the setting-based risk-reduction path when possible.

The goal is to move IRQs and unbound workqueues away from the SPDK CPUs. For example, if SPDK uses the default CPUs 0 and 1, IRQs and unbound workqueues should use CPU 2 through the last online CPU.

This workaround is a risk-reduction step, not a guaranteed fix for every workqueue lockup. CPU-bound or per-CPU kernel workers can still run on the SPDK CPUs because they are tied to those CPUs by the kernel. The expected result is that the node remains stable and any remaining workqueue stalls recover quickly instead of hanging for a long time.

Repeat this workaround whenever the Longhorn V2 CPU mask changes. If CPU allocation is managed dynamically, the IRQ and workqueue masks must be recalculated after each placement change.

1. Identify the Non-SPDK CPUs and Mask​

The default Longhorn V2 CPU mask is 0x3, so SPDK uses CPUs 0 and 1.

On the example node, all online CPUs are 0-23:

cat /sys/devices/system/cpu/online

0-23

Use the following values for this default example:

The Linux affinity mask is a CPU bitmap written in hexadecimal. CPU 0 is bit 0, CPU 1 is bit 1, and so on. On a 24-CPU node, all CPUs enabled is ffffff. Excluding CPUs 0 and 1 clears the lowest two bits, so the mask becomes fffffc. Leading zeros are optional, so fffffc and 00fffffc are equivalent.

Do not copy these values blindly if your Longhorn V2 CPU mask or online CPU list is different. The non-SPDK CPU list must be all online CPUs except the CPUs used by the SPDK reactor threads.

2. Apply IRQ Affinity at Runtime​

Set NON_SPDK_AFFINITY_MASK to the Linux affinity mask from the previous step.

NON_SPDK_AFFINITY_MASK=fffffc

echo "${NON_SPDK_AFFINITY_MASK}" > /proc/irq/default_smp_affinity

for f in /proc/irq/[0-9]*/smp_affinity; do
  echo "${NON_SPDK_AFFINITY_MASK}" > "${f}" 2>/dev/null || true
done

Some IRQs may reject affinity updates because they are managed by the kernel. This is expected. Always verify effective_affinity_list after applying the change.

Changing IRQ affinity alone may not make existing workqueue lockup messages disappear. Kernel workqueues can still run on the SPDK CPUs until the workqueue CPU masks are updated, and workers that are already stuck may continue to be reported by the kernel. Apply the workqueue affinity change as well, and then verify whether new lockup messages stop appearing.

These runtime IRQ affinity changes are not persistent across reboot. Use the persistence step later in this section if the runtime change mitigates the issue.

3. Apply Workqueue Affinity at Runtime​

NON_SPDK_AFFINITY_MASK=fffffc

echo "${NON_SPDK_AFFINITY_MASK}" > /sys/devices/virtual/workqueue/cpumask

for f in /sys/devices/virtual/workqueue/*/cpumask; do
  echo "${NON_SPDK_AFFINITY_MASK}" > "${f}" 2>/dev/null || true
done

These runtime workqueue affinity changes are not persistent across reboot. Use the persistence step later in this section if the runtime change mitigates the issue.

4. Restart the V2 Instance-Manager Pod​

After changing both IRQ and workqueue affinity, restart the Longhorn V2 instance-manager pod on the affected node. This is required because the existing spdk_tgt process can keep the SPDK reactor threads on the busy CPUs, and already-stuck CPU-bound workers may continue to be reported by the kernel until spdk_tgt is recreated.

Check the Longhorn V2 volumes first:

kubectl -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,DATAENGINE:.spec.dataEngine,STATE:.status.state,NODE:.status.currentNodeID

Only continue after all v2 volumes whose NODE is the affected node are detached.

Find the V2 instance-manager pod on the affected node:

AFFECTED_NODE=hp-114-tink-system

kubectl -n longhorn-system get pods \
  -l longhorn.io/component=instance-manager,longhorn.io/data-engine=v2 \
  --field-selector spec.nodeName="${AFFECTED_NODE}" \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Restart the V2 instance-manager pod:

V2_IM_POD=<v2-instance-manager-pod-name>

kubectl -n longhorn-system delete pod "${V2_IM_POD}"

5. Make IRQ and Workqueue Affinity Persistent​

The previous IRQ and workqueue affinity commands only change runtime procfs and sysfs state, so they are lost after reboot. If those runtime changes reduce or stop new lockup messages, use a Harvester CloudInit resource to reapply both IRQ and workqueue affinity after reboot. This follows the same model as the Longhorn CPU isolation setting, which updates host IRQ and workqueue affinity at runtime.

The following manifest is a Harvester CloudInit resource, not a file that you manually place on each node. Save it on a machine with kubectl access to the Harvester cluster and apply it with kubectl apply -f <file-name>.yaml. The Harvester controller writes the content to /oem/99_longhorn_v2_cpu_affinity.yaml on each node matched by matchSelector.

You can add this resource after the cluster is already installed. The file is synchronized to the matched nodes after the resource is applied, but the cloud-init commands take effect only after those nodes are rebooted. After reboot, the commands are applied again on each boot.

Example CloudInit resource for the default SPDK CPUs 0 and 1 on a 24-CPU node:

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: longhorn-v2-cpu-affinity
spec:
  matchSelector:
    kubernetes.io/hostname: "hp-114-tink-system"
  filename: 99_longhorn_v2_cpu_affinity.yaml
  contents: |
    stages:
      network:
        - commands:
            - echo fffffc > /proc/irq/default_smp_affinity
            - for f in /proc/irq/[0-9]*/smp_affinity; do echo fffffc > "${f}" 2>/dev/null || true; done
            - echo fffffc > /sys/devices/virtual/workqueue/cpumask
            - for f in /sys/devices/virtual/workqueue/*/cpumask; do echo fffffc > "${f}" 2>/dev/null || true; done

Adjust the affinity mask and matchSelector for your environment. To target a different affected node, replace hp-114-tink-system with that node's kubernetes.io/hostname label value. Do not use matchSelector: {} unless every node should receive this workaround. This workaround is not dynamically reconciled. If the Longhorn V2 CPU mask changes, update the CloudInit resource and reboot the affected nodes.

After configuring the CloudInit resource, reboot the affected nodes for the commands to take effect. To apply the workaround immediately before reboot, run the runtime commands in the previous steps.

If Lockups Still Appear​

The IRQ and workqueue affinity changes reduce the chance of a workqueue lockup, but they may not eliminate it in every environment. The workqueue CPU mask mainly controls unbound workqueues. Bound or per-CPU workqueues can still run on their associated CPU, so kernel workers such as kworker/20:* may still appear on an SPDK CPU even after unbound workqueues are moved away.

If the node still reports occasional workqueue lockups after the IRQ and workqueue affinity mitigation is applied, use the following additional mitigations.

1. Change the Longhorn V2 Disk Driver to auto​

If the affected Longhorn V2 disk is using the aio disk driver, consider changing the requested disk driver to auto. For an NVMe disk, verify after reprovisioning that Longhorn reports the actual disk driver as nvme in the Longhorn Node custom resource. In recent validation, after applying IRQ affinity and confirming that the actual disk driver was nvme, no new workqueue lockup messages were observed during the test window.

Before changing the disk driver, remove the affected disk from Harvester. Follow the Remove Disks guide, and make sure the disk no longer contains active Longhorn replicas or backing images.

Find the BlockDevice resource for the disk:

kubectl -n longhorn-system get blockdevices.harvesterhci.io \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,PROVISION:.spec.provision,PHASE:.status.provisionPhase,STATE:.status.state,ENGINE:.spec.provisioner.longhorn.engineVersion,DRIVER:.spec.provisioner.longhorn.diskDriver

After the disk is removed and the BlockDevice is no longer provisioned, patch the disk driver:

BLOCKDEVICE=<blockdevice-name>

kubectl -n longhorn-system patch blockdevice.harvesterhci.io "${BLOCKDEVICE}" \
  --type=merge \
  -p '{"spec":{"provisioner":{"longhorn":{"engineVersion":"LonghornV2","diskDriver":"auto"}}}}'

Provision the disk again from the Harvester UI. Harvester will add the disk back to Longhorn V2. Then verify the requested driver in the BlockDevice resource and the actual driver in the Longhorn Node resource.

Verify the BlockDevice after provisioning:

kubectl -n longhorn-system get blockdevice.harvesterhci.io "${BLOCKDEVICE}" -o yaml

The matching BlockDevice should show:

spec:
  provision: true
  provisioner:
    longhorn:
      engineVersion: LonghornV2
      diskDriver: auto
status:
  provisionPhase: Provisioned
  state: Active

Also check the Longhorn Node custom resource to confirm the actual disk driver selected by Longhorn:

kubectl -n longhorn-system get nodes.longhorn.io \
  -o go-template='{{printf "NODE\tDISK\tTYPE\tPATH\tSPEC_DRIVER\tSTATUS_DRIVER\n"}}{{range .items}}{{ $node := . }}{{range $diskName, $disk := .spec.disks}}{{ $status := index $node.status.diskStatus $diskName }}{{printf "%s\t%s\t%s\t%s\t%s\t%s\n" $node.metadata.name $diskName $disk.diskType $disk.path $disk.diskDriver $status.diskDriver}}{{end}}{{end}}'

For the affected disk, SPEC_DRIVER should be auto, and STATUS_DRIVER should be nvme:

NODE                  DISK                                  TYPE    PATH          SPEC_DRIVER   STATUS_DRIVER
hp-161-tink-system    32f43222-1eb1-4ab6-9e65-c4f8ddad700d  block   0000:08:00.0  auto          nvme

The spec.disks.<blockdevice-name>.diskDriver value is the requested driver mode from Harvester. The status.diskStatus.<blockdevice-name>.diskDriver value is the driver Longhorn actually uses. For an NVMe disk, the status value should be nvme.

2. Enable Longhorn V2 Interrupt Mode​

If the lockup still occurs after changing the disk driver and applying IRQ affinity, consider enabling Longhorn V2 interrupt mode.

Check the setting:

kubectl -n longhorn-system get settings.longhorn.io data-engine-interrupt-mode-enabled

Enable interrupt mode for the V2 Data Engine:

kubectl -n longhorn-system patch settings.longhorn.io data-engine-interrupt-mode-enabled \
  --type=merge \
  -p '{"value":"{\"v2\":\"true\"}"}'

Wait for the V2 instance-manager pods to be recreated after all V2 volumes are detached.

Verification​

After applying the mitigation, verify the items that directly confirm the node recovered and the settings are still applied.

1. Kernel logs no longer report new workqueue lockups, or any remaining reports are short and do not keep increasing for a long time.

   journalctl -k --since "30 minutes ago" | grep -E "BUG: workqueue lockup|soft lockup" || true
   

   Copy

   If occasional messages still appear, compare the stuck for <seconds>s value over time. The mitigation is still useful if the stuck time stops growing, the node remains responsive, and RKE2, kubelet, and Longhorn keep making progress.

2. The Longhorn V2 instance-manager pod is running on the affected node.

   AFFECTED_NODE=hp-114-tink-system
   
   kubectl -n longhorn-system get pods \
     -l longhorn.io/component=instance-manager,longhorn.io/data-engine=v2 \
     --field-selector spec.nodeName="${AFFECTED_NODE}"
   

   Copy

3. If you configured the persistent workaround and rebooted the node, the IRQ and workqueue masks are still applied.

   cat /proc/irq/default_smp_affinity
   cat /sys/devices/virtual/workqueue/cpumask
   

   Copy

References​

• Harvester bug: harvester/harvester#11030
• Longhorn bug: longhorn/longhorn#13417
• Longhorn setting reference: Data Engine CPU Mask
• Longhorn setting reference: Data Engine Interrupt Mode Enabled
• Longhorn documentation: Interrupt Mode Support
• Linux kernel documentation: IRQs
• Linux kernel documentation: Workqueue

This article provides information and mitigation steps for the following vulnerabilities in Harvester:

• CVE-2026-53359

The mitigation steps involves disabling the nested virtualization feature of the KVM kernel module on your Harvester hosts.

On your Harvester hosts, use the following commands to confirm that the KVM kernel module is loaded with nested virtualization enabled:

lsmod | grep -iE "kvm_amd|kvm_intel"

sudo grep -H '' /sys/module/{kvm_amd|kvm_intel}/parameters/* 2>&1 |grep nested

If the above commands return module and parameter information about the KVM kernel module, then your host is affected by this vulnerability.

Deploy the following CloudInit configuration to disable the nested virtualization feature of the KVM kernel module on all your Harvester hosts:

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: disabled-nested-virtualization
spec:
  matchSelector:
    harvesterhci.io/managed: "true"
  filename: 99-disabled-nested-virtualization
  contents: |
    stages:
      initramfs:
      - name: "disable nested virtualization in kvm modules"
        files:
        - path: "/etc/modprobe.d/99-disabled-nested-virtualization.conf"
          content: |
            options kvm_amd nested=0
            options kvm_intel nested=0

Once the configuration is applied, reboot your Harvester hosts for the changes to take effect.

Once you have upgraded to a fixed version of Harvester, you can re-enable the nested virtualization feature by deleting the CloudInit configuration and rebooting your Harvester hosts:

kubectl delete cloudinit disabled-nested-virtualization

References​

• https://support.scc.suse.com/s/kb/Security-Vulnerability-x86-64-kvm-Guest-to-Host-escape-aka-Januscape-CVE-2026-53359
• https://www.suse.com/security/cve/CVE-2026-53359.html

When running production workloads on virtualized infrastructure like Harvester, memory management is critical. In Harvester versions prior to v1.4.0, certain workloads experienced sudden Virtual Machine (VM) terminations due to the host Linux operating system triggering Out-Of-Memory (OOM) kills.

What is KubeVirt? Harvester uses KubeVirt as its core virtualization engine. KubeVirt is an open-source technology that allows Kubernetes to run and manage traditional Virtual Machines inside standard containers, translating VM specifications directly into Pod configurations.

This article explores why these OOM events occur in a Kubernetes-native virtualization environment and how Harvester provides granular tools to eliminate them.

Anatomy of a VM OOM Event​

When a VM is terminated due to insufficient memory at the host level, the Linux kernel logs specific keywords that help pinpoint the fault. In Harvester, these logs generally fall into two distinct categories depending on which process triggered the exhaustion.

Example 1: virt-launcher invoked oom-killer

The virt-launcher process runs inside the dedicated Kubernetes Pod backing the VM. If this component or its direct sub-processes run out of the memory allocated to their cgroup, the kernel triggers a memory cgroup (memcg) OOM event.

Feb 03 19:57:08 ** kernel: virt-launcher invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=986
Feb 03 19:57:08 ** kernel: CPU: 40 PID: 40785 Comm: virt-launcher Tainted: G          I    X    5.14.21-150400.24.60-default #1 SLE15-SP4 9096397fa6646928cc6d185ba417f2af65b536f1
...

Feb 03 19:57:08 ** kernel: memory: usage 17024340kB, limit 17024340kB, failcnt 1243
Feb 03 19:57:08 ** kernel: memory+swap: usage 17024340kB, limit 9007199254740988kB, failcnt 0                                                                 
Feb 03 19:57:08 ** kernel: kmem: usage 143556kB, limit 9007199254740988kB, failcnt 0
Feb 03 19:57:08 ** kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice:
...
Feb 03 19:57:08 ** kernel: Tasks state (memory values in pages):
Feb 03 19:57:08 ** kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Feb 03 19:57:08 ** kernel: [  38886]     0 38886      243        1    28672        0          -998 pause
Feb 03 19:57:08 ** kernel: [  38917]     0 38917   310400     6921   192512        0           986 virt-launcher-m
Feb 03 19:57:08 ** kernel: [  38934]     0 38934  1200940    25126   954368        0           986 virt-launcher
Feb 03 19:57:08 ** kernel: [  38951]     0 38951   386525     8247   466944        0           986 libvirtd
Feb 03 19:57:08 ** kernel: [  38952]     0 38952    33619     3940   290816        0           986 virtlogd
Feb 03 19:57:08 ** kernel: [  39079]   107 39079  4457263  4201766 34439168        0           986 qemu-system-x86
Feb 03 19:57:08 ** kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice/cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,task=qemu-system-x86,pid=39079,uid=107

Example 2: CPU X/KVM invoked oom-killer

This occurs when a vCPU execution thread inside qemu-system-x86_64 attempts a memory operation that pushes the entire container beyond its Kubernetes memory limit.

[Thu May  9 14:52:38 2024] CPU 11/KVM invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=830
[Thu May  9 14:52:38 2024] CPU: 60 PID: 70888 Comm: CPU 11/KVM Not tainted 5.3.18-150300.59.101-default #1 SLE15-SP3
...
[Thu May  9 14:52:38 2024] memory: usage 67579904kB, limit 67579904kB, failcnt 67391
[Thu May  9 14:52:38 2024] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[Thu May  9 14:52:38 2024] kmem: usage 633636kB, limit 9007199254740988kB, failcnt 0
...
[Thu May  9 14:52:38 2024] Tasks state (memory values in pages):
[Thu May  9 14:52:38 2024] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Thu May  9 14:52:38 2024] [  70675]     0 70675      243        1    28672        0          -998 pause
[Thu May  9 14:52:38 2024] [  70728]     0 70728   310400     5467   188416        0           830 virt-launcher-m
[Thu May  9 14:52:38 2024] [  70746]     0 70746  1242373    25104  1073152        0           830 virt-launcher
[Thu May  9 14:52:38 2024] [  70762]     0 70762   455279    14110   770048        0           830 libvirtd
[Thu May  9 14:52:38 2024] [  70763]     0 70763    37704     3916   339968        0           830 virtlogd
[Thu May  9 14:52:38 2024] [  70870]   107 70870 18302464 16718510 135278592        0           830 qemu-system-x86
[Thu May  9 14:52:38 2024] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,mems_allowed=0-3,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice/cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,task=qemu-system-x86,pid=70870,uid=107
[Thu May  9 14:52:38 2024] Memory cgroup out of memory: Killed process 70870 (qemu-system-x86) total-vm:73209856kB, anon-rss:66852088kB, file-rss:21948kB, shmem-rss:4kB
[Thu May  9 14:52:38 2024] oom_reaper: reaped process 70870 (qemu-system-x86), now anon-rss:0kB, file-rss:132kB, shmem-rss:4kB

Root Cause: Native Hypervisors vs. Harvester Architecture​

Traditional Linux Host (e.g., Virtual Machine Manager)​

On a standard Linux host, a VM managed via QEMU/KVM runs inside a systemd machine.slice. The hypervisor process (qemu-system-x86_64) has access to the host's wider pool of resources, managed loosely unless strict cgroup limits are manually added.

$ systemd-cgls
Control group /:
-.slice
├─1173 bpfilter_umh
├─system.slice

└─machine.slice
  └─machine-qemu\x2d1\x2dharv41.scope 
    └─8632 /usr/bin/qemu-system-x86_64 -name guest=harv41,debug-threads=on -S -…

Harvester (Kubernetes/KubeVirt Engine)​

In Harvester, every VM is encapsulated inside a Kubernetes Pod. This introduces a strict cgroup boundary (kubepods.slice).

As shown below, multiple helper processes must live alongside the primary qemu-system-x86_64 process within the same tightly limited container memory budget:

-.slice
└─kubepods.slice
  │ ├─kubepods-burstable-pod99ee3a64_645b_4699_9384_5a3875d78b41.slice
  │ │ ├─cri-containerd-0c316eb8a4711bff1ce968b46ddb658e49378897454b4caf2a20704c808f33f1.scope …
  │ │ │ └─ 8505 /pause
  │ │ ├─cri-containerd-eb5deef29f064adfd2456d9f9c535674ac4d0c95c81b13afbdab5a89dc6a774b.scope …
  │ │ │ └─ 8590 /usr/bin/virt-tail --logfile /var/run/kubevirt-private/2ce151aa…
  │ │ └─cri-containerd-fd57a5cfc2b9b1f53eaf7b575c3273e6784f4c56a04a17d502ecfbd19e55b066.scope …
  │ │   ├─ 8542 /usr/bin/virt-launcher-monitor --qemu-timeout 301s --name vm2 -…
  │ │   ├─ 8558 /usr/bin/virt-launcher --qemu-timeout 301s --name vm2 --uid 2ce…
  │ │   ├─ 8591 /usr/sbin/virtqemud -f /var/run/libvirt/virtqemud.conf
  │ │   ├─ 8592 /usr/sbin/virtlogd -f /etc/libvirt/virtlogd.conf
  │ │   └─ 8823 /usr/bin/qemu-system-x86_64 -name guest=default_vm2,debug-threa…

The "Hidden" Memory Overhead​

Breaking Down the Memory Overhead Buffer​

When you define a virtual machine, for example, a VM configured with 4 vCPUs, 2 GiB of memory, and 1 Ethernet interface, KubeVirt does not just allocate exactly 2 GiB of memory to the container.

Instead, KubeVirt calculates an additional baseline memory overhead required to operate the virtualization stack. This overhead budget covers:

• CPU Simulators: Thread pools tracking guest state and handling context switches.
• Memory Management: Tracking structures such as QEMU page tables mapping guest RAM.
• Auxiliary Devices: Buffers for virtual network interfaces (NICs), storage queues, and video devices.

The High Stakes of VM OOM Kills​

Depending on the guest OS type, specific kernel workloads, and heavy storage/network I/O spikes, the memory consumed by these helper tasks can quickly exceed KubeVirt's default calculations. Because Kubernetes enforces a strict hard ceiling on the Pod container, the entire container triggers a CONSTRAINT_MEMCG OOM kill the moment this boundary is breached.

Unlike traditional, stateless Kubernetes workloads where a container crash is quickly mitigated by a rapid pod restart, an OOM kill on a VM pod carries severe operational consequences:

• Prolonged Downtime: A virtual machine is a stateful workload. It does not instantaneously serve traffic upon a container restart; it must undergo a full operating system boot cycle, run init scripts, and re-initialize services, drastically extending your Recovery Time Objective (RTO).

• Risk of Data Corruption: Sudden terminations during flight can abruptly cut off active storage queues. If the guest OS or database engine is in the middle of a critical write operation when the host terminates the qemu process, it can result in uncommitted journals, filesystem degradation, or severe data corruption on your persistent volumes.

The Solution: Tunable Memory Architectures​

To address this, Harvester introduced dual-layer configurations that give administrators full flexibility over how overhead buffers are calculated.

Global Adjustment: additional-guest-memory-overhead-ratio​

This cluster-wide setting functions as a multiplier for KubeVirt's automatically calculated memory overhead. For deep structural details, refer to the Harvester Advanced Documentation.

• Definition: Scales the calculated overhead buffer to accommodate heavy I/O or virtualization tasks.
• Default Value: 1.5 (Provides a 50% safety cushion above baseline calculations).
• Valid Range: 0 or 1.0 to 10.0.

💡 Important Operational Notes:

• Lifecycle Impact: Changes to this setting only apply to newly created virtual machines or existing VMs after they undergo a migration or a full power cycle.
• System Overhead: A higher ratio increases the host container's memory allocation, guaranteeing safety for heavy workloads but scaling up the overall system resource reservation footprint.
• Resource Allocation Trade-off: Setting this ratio excessively high can lock up unneeded host memory blocks, leading to predictable underutilization and significant memory waste across your compute nodes.

Per-VM Optimization: Reserved Memory​

For specific virtual machines running intensive or non-standard workloads, a global multiplier might not offer the precision required. Harvester allows administrators to define a dedicated Reserved Memory value directly on individual VMs. For complete configuration steps, see the Harvester VM Management Documentation.

⚠️ Under the Hood Memory Carving: When you configure this setting, Harvester explicitly scales down the available memory presented to the Guest OS inside the VM. For example, if a VM is configured with 2 GiB of memory and you set a Reserved Memory value of 256 MiB, the Guest OS will only see and utilize 1.75 GiB (2 GiB - 256 MiB).

Why Use Per-VM Reserved Memory?​

• Guaranteed Overhead Headroom: By restricting the Guest OS from consuming the top slice of its configured allocation, you guarantee an isolated, un-evictable memory runway for host helper tasks.

• Targeted Safety for Heavy Workloads: This mechanism is highly practical for mission-critical, high-performance, or special-purpose workloads (such as nested virtualization layers or intensive database engines). It effectively prevents the VM from running into host-level cgroup OOM termination by proactively limiting its internal usage boundaries, removing the risk of unexpected node-level kills.

• Optimized Cluster Usability: Using per-VM reservations eliminates the major disadvantage of cranking up the global additional-guest-memory-overhead-ratio for the whole cluster. Instead of forcing a massive, wasteful memory overhead reservation across every idle or lightweight VM on your hosts, you can maintain a lean global default and surgically protect only the heavy workloads—striking an ideal balance between system density and ironclad stability.

Outcome: Guaranteed Workload Stability​

By leveraging this dual-layer tunable memory architecture, Harvester fundamentally alters how host-level overhead is calculated, moving from rigid, generalized defaults to a precise, tiered enforcement model:

Total Memory Overhead = Auto-calculated Overhead * Ratio + Reserved Memory

Best Practices & Configuration Matrix​

The following matrix showcases how combinations of Reserved Memory and the Overhead Ratio change the actual layout of the Guest OS space versus what Kubernetes reserves as a hard boundary.

When optimizing your Harvester cluster to eliminate host-level container OOM events, use the following operational checklist to tailor your memory strategies:

• For General Workloads:

  • Stick to the default ratio of 1.5, or configure a slightly higher value of 2.0. This ensures that standard Guest operating systems receive exactly the memory requested while scaling out a stable, predictable background overhead buffer across the cluster.

• For High I/O and Storage-Heavy VMs:

  • If you observe periodic KVM OOM events during massive backup windows, large-scale data syncs, or intensive disk read/write cycles, increase the individual VM's allocation or implement a targeted Reserved Memory configuration to safely expand the helper overhead pool.

• For GPU Passthrough Workloads:

  • Virtual machines utilizing direct hardware acceleration or GPU passthrough are prime candidates for explicit Reserved Memory carving. The underlying host-side device drivers and memory-mapped I/O (MMIO) windows for high-performance graphics hardware require a significantly higher, specialized memory footprint outside the guest OS space. Allocating dedicated per-VM reserved memory prevents driver-instigated cgroup allocation breaches, keeping both the hardware pipeline and the hypervisor completely stable.

Quick Summary​

• The Problem: In Harvester's Kubernetes-native architecture, every virtual machine is bound by a strict Pod container limit. While this rigid cgroup boundary is essential for security, ensuring a single rogue or leaking VM can never starve neighboring workloads or crash the bare-metal host, it means heavy storage/network I/O, device drivers, or GPU passthrough can cause internal helper processes to breach this hard ceiling, triggering a sudden host-level OOM kill.

• The Solution: Harvester eliminates these crashes without losing secure resource control using a dual-layer memory tuning strategy:

  • Globally: The additional-guest-memory-overhead-ratio scales out a safety cushion cluster-wide for newly created or migrated VMs.
  • Per-VM: The Reserved Memory setting surgically carves out a chunk of the VM's configured RAM exclusively for background helper tasks—preventing wasteful memory reservations across the cluster while safely anchoring high-performance, mission-critical workloads.

Appendix: Lab Simulation — Manually Triggering the Host-Level OOM​

For engineers looking to validate this behavior safely in a staging environment, you can replicate this multi-process cgroup breach. A detailed script and case study can be found in the Harvester Development Summary: OOM Investigation.

The simulation process highlights a fundamental truth about modern virtualization boundaries:

• The Guest OS is Trustworthy: Testing shows that modern guest operating systems handle internal resource limits reliably. If a runaway application inside the guest OS eats up all available RAM, the guest kernel safely steps in and kills that specific process internally. The VM itself survives, and from the host's perspective, the virtual machine continues running normally.

• The Host Cgroup Boundary is the Weak Link: The true host-level crash only happens if processes inside the host cgroup expand unexpectedly. If an infrastructure task or helper process inside the Pod container balloons, it consumes the memory buffer that KubeVirt set aside, causing the entire cgroup, the VM's carrier, to slam into the hard Kubernetes ceiling and trigger a host-level OOM kill.

This article provides information and mitigation steps for the following vulnerabilities in Harvester:

• CVE-2026-42945
• CVE-2026-42946
• CVE-2026-40701
• CVE-2026-42934

You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:

kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'

If the command returns one of the affected versions, perform the following mitigation steps.

The primary resolution is to upgrade Harvester to one of these versions:

• 1.7.2 or newer
• 1.8.1 or newer

If upgrade is not possible, apply the following mitigation to protect your clusters.

All ingress resources with the nginx.ingress.kubernetes.io/rewrite-target annotation containing ? in the annotation value are at risk.

By default, Harvester does not include any ingress resources with this annotation. Run the following command on your clusters to identify affected custom ingress resources:

kubectl get ingress -A -o json | jq '.items[] | select(.metadata.annotations["nginx.ingress.kubernetes.io/rewrite-target"] // "" | contains("?")) | {namespace: .metadata.namespace, name: .metadata.name, rewrite: .metadata.annotations["nginx.ingress.kubernetes.io/rewrite-target"]}'

Any ingress resources reported by the above command are vulnerable. They should be updated to either remove the vulnerable annotation or change the annotation value to not contain a question mark ?.

The following validating admission policy can be applied to your cluster to reject ingress resources with the vulnerable configuration:

cat<<EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: ingress-nginx-annotation-validation-20260514
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["networking.k8s.io"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["ingresses"]
  validations:
    - expression: |
        !has(object.metadata.annotations) ||
        !object.metadata.annotations.exists(k, k == 'nginx.ingress.kubernetes.io/rewrite-target') ||
        !object.metadata.annotations['nginx.ingress.kubernetes.io/rewrite-target'].contains('?')
      message: "Ingress resources with 'nginx.ingress.kubernetes.io/rewrite-target' annotation containing '?' in the annotation value are not allowed, due to the following CVEs: CVE-2026-42945, CVE-2026-42946, CVE-2026-40701, CVE-2026-42934"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: ingress-nginx-annotation-validation-20260514
spec:
  policyName: ingress-nginx-annotation-validation-20260514
  validationActions: [Deny]
EOF

The policy can be removed once you upgrade to Harvester 1.7.2, 1.8.1 or newer:

kubectl delete validatingadmissionpolicy ingress-nginx-annotation-validation-20260514

kubectl delete validatingadmissionpolicybinding ingress-nginx-annotation-validation-20260514

References​

• https://depthfirst.com/research/nginx-rift-achieving-nginx-rce-via-an-18-year-old-vulnerability
• https://www.suse.com/security/cve/CVE-2026-42945.html
• https://www.suse.com/security/cve/CVE-2026-42946.html
• https://www.suse.com/security/cve/CVE-2026-40701.html
• https://www.suse.com/security/cve/CVE-2026-42934.html

You have lost the admin password for the Harvester GUI​

The admin password for the Harvester GUI can be reset if you can still login via ssh as the rancher user, or if you have the Harvester cluster's kubeconfig file saved locally. For details, see How can I reset the administrator password? in the documentation.

You have lost the rancher user's ssh/console login password​

But you still have the Harvester cluster's kubeconfig​

The rancher user's ssh/console login password can be reset by creating a CloudInit CRD to update the password. For details, see Password of user rancher in the documentation. Alternately you can create additional users with ssh access as described in How to create an SSH user for Harvester nodes.

You don't have a kubeconfig, but do have physical or remote console access​

In this case, the rancher user's ssh/console login password can be changed by editing /oem/90_custom.yaml on each host.

If you can boot the Harvester installer ISO​

Boot the Harvester installer, but don't proceed with the regular installation process. Instead, once the installer comes up, press CTRL-ALT-F2 to switch to VT2 and login as the rancher user with password rancher. Then proceed with the following steps:

1. Run sudo -i to become root
2. Mount the COS_OEM partition from the host:

   # mkdir /tmp/oem
   # mount -L COS_OEM /tmp/oem
   

   Copy
3. Run vim /tmp/oem/90_custom.yaml to edit 90_custom.yaml and change the password for the rancher user. You can specify either a plaintext password (not recommended) or a password hash generated with e.g. openssl passwd -6. Following is the section that you need to edit:

   users:
     rancher:
       passwd: <PASSWORD_GOES_HERE>
   

   Copy
4. Reboot the host. You should now be able to log in as the rancher user with the new password.

You can't boot the Harvester installer ISO, but can still reboot the host and access the boot menu​

If you have no other option, then during system boot, edit the grub config and add rd.break at the end of the kernel command line (the one that starts with linux). This will drop you into the dracut emergency shell, with the root partition mounted under /sysroot. Unfortunately, this shell does not contain any text editor programs. Trying to edit /oem/90_custom.yaml under the circumstances would be unwise. Instead though, you can run this command:

# sed -i 's%rancher.*%rancher:$6$j0.h3TQv8RZPHJkB$3SbV978JLT2Qeq4KSCBZitErNlZZGfrDxnGW5HS0wHzWexGyPzeQBoQmQJetUhLFfquv/X5VWL6odxtlEec1u/:20468::::::%' /sysroot/etc/shadow

Then, hit CTRL-D to continue, and once the system finishes booting, the rancher user's password will be set back to rancher for this boot only. You can then login on the console and use vim to update /oem/90_custom.yaml and permanently set the password to something more secure as described in the previous section.

This article provides information and mitigation steps for the following vulnerabilities in Harvester:

• CVE-2025-15566
• CVE-2026-1580
• CVE-2026-24512
• CVE-2026-24513
• CVE-2026-24514

You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:

kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'

If the command returns one of the affected versions, perform one of the following mitigation steps.

The primary resolution is to upgrade to Harvester 1.7.1 or newer, which includes the fixed RKE2 ingress-nginx controller.

If upgrade is not possible, deploy the following validating admission policy to your cluster to reject ingress resources with the vulnerable configuration:

cat<<EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: ingress-nginx-annotation-validation
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["networking.k8s.io"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["ingresses"]
  validations:
    - expression: |
        !('nginx.ingress.kubernetes.io/auth-proxy-set-headers' in object.metadata.annotations) &&
        !('nginx.ingress.kubernetes.io/auth-method' in object.metadata.annotations) &&
        (object.spec.rules.all(rule, rule.http.paths.all(path, path.pathType != 'ImplementationSpecific')))
      message: "Ingress resources with the vulnerable annotations are not allowed. Please remove the 'nginx.ingress.kubernetes.io/auth-proxy-set-headers' and 'nginx.ingress.kubernetes.io/auth-method' annotations, and avoid using the 'ImplementationSpecific' path type."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: ingress-nginx-annotation-validation
spec:
  policyName: ingress-nginx-annotation-validation
  validationActions: [Deny]
EOF

This validating policy rejects any ingress resources that contain the:

• nginx.ingress.kubernetes.io/auth-proxy-set-headers annotation
• nginx.ingress.kubernetes.io/auth-method annotation
• ImplementationSpecific path type

The policy can be removed once you upgrade to Harvester 1.7.1 or newer:

kubectl delete validatingadmissionpolicy ingress-nginx-annotation-validation

kubectl delete validatingadmissionpolicybinding ingress-nginx-annotation-validation

References​

• https://github.com/kubernetes/kubernetes/issues/136789
• https://github.com/kubernetes/kubernetes/issues/136677
• https://github.com/kubernetes/kubernetes/issues/136678
• https://github.com/kubernetes/kubernetes/issues/136679
• https://github.com/kubernetes/kubernetes/issues/136680
• https://support.scc.suse.com/s/kb/Upstream-Ingress-Nginx-CVEs-CVE-2025-15566-CVE-2026-1580-CVE-2026-24512-CVE-2026-24513-and-CVE-2026-24514

Issue​

Node selector constraints can prevent the scheduler from live-migrating a virtual machine to a target node. This often indicates a mismatch between the virtual machine's requirements and the node's capabilities.

A node selector may require a specific CPU feature, but the target node lacks the corresponding label (for example, cpu-feature.node.kubevirt.io/fpu: "true"). This mismatch can occur when the host-model CPU models and features computed by KubeVirt change over time.

Solutions​

You can resolve this issue using four different approaches.

• Reboot the virtual machine.

  KubeVirt automatically adds node selectors (during a previous migration or initial start) that can restrict scheduling. You can clear these node selectors by rebooting the virtual machine.

• Reboot the virtual machine and set up a common CPU model.

  You can override KubeVirt's default host-model CPU configuration by setting up a common CPU model for virtual machine migration. The model is applied to the virtual machine as its domain CPU, and to the pod as its node selector configuration.

  This is the recommended approach for environments that can tolerate restarting of virtual machines.

• Modify the node labels.

  If rebooting the virtual machine is not an option, you can manually manipulate the target node's labels to satisfy the scheduling requirements.

  1. Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.

     This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.

     kubectl annotate node <node-name> node-labeller.kubevirt.io/skip-node="true"
     

     Copy

     important

     The annotation itself does not affect the pod's node selector. It only controls the presence of specific CPU-related labels on the node, which the node selector checks against. For more information, see the References section.

  2. Identify labels that are missing from the virtual machine's node selector and add them to the target node.

     You can add the missing labels using the following command:

     kubectl label node <node-name> <key>=<value>
     

     Copy

     This circumvents the standard scheduling restrictions, allowing the virtual machine to migrate to the target node.

  If a new node that lacks the required features is added to the cluster, you must repeat these steps to allow the virtual machine to live-migrate to that node.

• Remove the node labels.

  If you want to ensure that the virtual machine does not acquire specific node selector constraints after live migration, you can remove the relevant CPU labels from the target node.

  1. Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.

     This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.

     kubectl annotate node <node-name> node-labeller.kubevirt.io/skip-node="true"
     

     Copy

     important

     This method works only if the virtual machine's pod does not have an existing node selector that contains the labels listed in the References section. Otherwise, you must reboot the virtual machine to clear the constraints.

  2. Check if the pod has a node selector.

     kubectl get pod <pod-name> -o yaml | grep nodeSelector -A 5 -B 5
     

     Copy

  3. If no node selector exists, remove the relevant CPU labels from the node.

     Performing this action prevents the pod from acquiring new node selector constraints, thus enabling its future migration to other nodes. However, the successful outcome of that migration is not guaranteed.

References​

KubeVirt Node Labels​

The KubeVirt CPU node-labeller manages the following labels:

• cpu-feature.node.kubevirt.io/*
• cpu-model-migration.node.kubevirt.io/*
• cpu-model.node.kubevirt.io/*
• host-model-cpu.node.kubevirt.io
• host-model-required-features.node.kubevirt.io

External Links​

• Labeling nodes with CPU models, CPU features and machine types