In October 2026, the private keys of eight eMudhra intermediate certificate authorities were found to be recoverable from their public certificates, because of a key generation flaw. An attacker who holds one of these keys can issue a certificate for any host name and impersonate any TLS server to a client that still trusts these intermediates.

The intermediates chain to the emSign ECC Root CA - G3 root and, through cross-signed copies, to the AAA Certificate Services root (labeled Comodo AAA Services root in the trust store). Browsers blocked them between October 17 and 20, 2026, and Mozilla removed the emSign ECC Root CA - G3 root from NSS 3.131.1. Non-browser clients generally do not check revocation and need an updated trust store.

Affected Versions​

The following is the list of affected versions:

• Harvester 1.9.0
• Harvester 1.8.2 and earlier
• Harvester 1.7.3 and earlier
• Harvester 1.6.1 and earlier

Harvester nodes ship the Mozilla trust store from the ca-certificates-mozilla package. Versions 1.8.2 and 1.9.0 include ca-certificates-mozilla-2.84, which still trusts both roots. Earlier versions ship older bundles that include the same roots.

The trust store is used by the TLS clients that run on the host, such as containerd when it pulls images, kubelet and RKE2. Container images carry their own trust store, so the components that run in pods are affected independently of the host.

Your node is affected if the following command, run on the node, returns a value other than 0:

trust list --filter=ca-anchors | grep -cE "label: (emSign ECC Root CA - G3|Comodo AAA Services root)$"

If the node has Internet access, a successful curl https://dv-g3b-emudhra.we-have-a-problem.com/ also shows that the node is affected.

Fixed Versions​

A fix is not yet available. SUSE is working on an updated ca-certificates-mozilla package, which a later Harvester release will include along with rebuilt container images.

Meanwhile, you can use the following workaround to remove the trust in both roots on the Harvester nodes.

Workaround​

Use a CloudInit resource to move the compromised trust to the system blocklist. This workaround does not require network access and is suitable for air-gapped clusters.

Apply the following resource to your Harvester cluster:

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: emudhra-blocklist
spec:
  matchSelector: {}
  filename: 99_emudhra-blocklist.yaml
  contents: |
    stages:
      boot:
        - name: "Distrust the eMudhra compromised roots"
          commands:
            - |
              d=/etc/pki/trust/blocklist; [ -d $d ] || d=/etc/pki/trust/blacklist
              trust extract --format=pem-bundle --filter='pkcs11:object=Comodo%20AAA%20Services%20root;type=cert' $d/Comodo_AAA_Services_root.pem
              trust extract --format=pem-bundle --filter='pkcs11:object=emSign%20ECC%20Root%20CA%20-%20G3;type=cert' $d/emSign_ECC_Root_CA_-_G3.pem
              update-ca-certificates

The resource is written to /oem/99_emudhra-blocklist.yaml on every node, including nodes that join the cluster later. It takes effect at the next reboot: the boot stage runs before RKE2, containerd and kubelet start, so these services load the updated trust store. Running services keep the roots that they have already loaded, so reboot the nodes one at a time, using maintenance mode.

After the workaround is applied, the check command in Affected Versions returns 0.

This workaround only covers the clients that use the host OS trust store. If your cluster runs virtual machines or pods with independent trust stores, you must update them following their base OS instructions.

kubectl delete cloudinit emudhra-blocklist

References​

• https://support.scc.suse.com/s/kb/Security-Vulnerability-Compromised-SSL-Root-CA-eMudhra-CVE-2026-114406-CVE-2026-111943-CVE-2026-113399
• https://emudhra.we-have-a-problem.com/
• https://bugzilla.mozilla.org/show_bug.cgi?id=2083449
• https://blog.mozilla.org/security/2026/10/20/protecting-firefox-users-following-the-compromise-of-emudhra-intermediate-certificates/
• https://www.suse.com/security/cve/CVE-2026-111943.html

Description​

On a cluster configured with a CPU model such as cpu-model.node.kubevirt.io/Cascadelake-Server=true, virtual machines may no longer start after an upgrade to Harvester v1.7.3.

The mitigation for CVE-2019-11135 disables TSX on affected CPUs. As a result, the hle and rtm CPU features are missing on those nodes, and virtual machines that require them cannot start.

For details about CVE-2019-11135, see the Intel Security Advisory. To check whether a processor is affected, refer to Intel's list of affected processors and search for CVE-2019-11135.

Resolution​

Following the check described in Red Hat's TAA advisory, check the TAA status on each node by running:

cat /sys/devices/system/cpu/vulnerabilities/tsx_async_abort

If the output reports Vulnerable, use the corresponding CPU model ending in -noTSX (for example, Skylake-Server-noTSX or Cascadelake-Server-noTSX) to disable TSX for the virtual machine. Interpret other output according to your operating system's security guidance. Make sure the selected model is supported by every node where the virtual machine may run.

Alternatively, configure Cascadelake-Server-v3 or a later version as the CPU model. After changing the CPU model, restart all affected virtual machines.

For more information about choosing and configuring a common CPU model, see Setting up a common CPU model for virtual machine migration.

References​

• harvester/harvester#11784
• CVE-2019-11135 - TSX Asynchronous Abort (Red Hat)
• Setting up a common CPU model for virtual machine migration

You can integrate Harvester with the DNS servers that already exist in your environment. Specifically, you can configure the DNS settings that virtual machines (VMs) receive, and you can use ExternalDNS to publish DNS records for Harvester load balancers (LBs) and for LoadBalancer services in guest clusters.

This procedure was tested with Harvester v1.9.0, ExternalDNS v0.15.0, BIND 9.20, and PowerDNS Authoritative Server 4.9. The guest cluster load balancer example was tested with an RKE2 v1.36.5 guest cluster that uses the Harvester cloud provider v0.2.8.

User Stories​

• An infrastructure team runs application VMs behind Harvester load balancers. Application owners want to reach each application by a name in the corporate DNS zone (for example, web.vm.example.com) instead of an IP address. The DNS record must be created when the load balancer is created and removed when the load balancer is deleted, without a manual request to the DNS team. The VMs must also resolve internal names by using the corporate DNS servers.

• A platform team runs RKE2 guest clusters on Harvester. Developers expose applications with LoadBalancer services and want each service to be resolvable by name in the corporate DNS zone as soon as it receives an IP address.

Architecture​

The following diagram shows the components involved and which party manages each component.

Harvester and your existing infrastructure have the following responsibilities:

ExternalDNS is a third-party component that is not installed or managed by Harvester. ExternalDNS is available in the SUSE Application Collection. You can run ExternalDNS in the Harvester cluster or outside it, provided that it can read services from the Harvester Kubernetes API. When ExternalDNS runs outside the cluster, a service account with read-only access to services is sufficient.

DNS Integration Options​

The following table outlines the available options:

Configure DNS Settings for VMs​

VMs obtain DNS settings from the network configuration that they receive. You can use any of the following methods:

• Managed DHCP: Specify the dns, domainName, and domainSearch fields in the IPPool resource. The Managed DHCP agent sends these values to VMs as DHCP options. For more information, see Managed DHCP.

• Kube-OVN DHCP: Specify the DNS server in the dhcpV4Options field of the subnet (for example, dns_server=10.0.0.53). For more information, see Virtual Private Cloud (VPC).

• External DHCP server or static configuration: Configure the DNS settings on your DHCP server, or specify the DNS servers in the cloud-init network data when you create the VM.

Limitations​

• Harvester does not automatically create DNS records for the IP addresses of individual VMs. You can use ExternalDNS to publish DNS records only for load balancers.

• The Managed DHCP add-on does not send the hostname option (option 12) and does not process the client FQDN option (option 81). Consequently, DHCP-based dynamic DNS updates are not supported for VMs that obtain IP addresses from Managed DHCP.

Publish DNS Records for VM Load Balancers​

Each VM load balancer is backed by a Kubernetes Service of type LoadBalancer. The service has the following characteristics:

• It has the same name and namespace as the load balancer.
• It has the label loadbalancer.harvesterhci.io/servicelb: "true".
• Its status.loadBalancer.ingress field contains the IP address assigned to the load balancer.

This applies to load balancers that use either the IP pool or the DHCP IPAM mode.

Because the service is a standard Kubernetes resource, ExternalDNS can watch it and automatically create or update the corresponding DNS records on your DNS server.

Prerequisites​

Before you begin, ensure that the following requirements are met:

• The DNS server supports dynamic updates (RFC 2136) and is reachable from the location where ExternalDNS runs.

• A TSIG key is configured on the DNS server, and the key is allowed to update the target zone. For a BIND configuration example, see the ExternalDNS RFC 2136 tutorial.

The examples in this section use the following settings:

• DNS server: 192.168.100.53 (BIND)
• Zone: vm.example.com
• TSIG key name: externaldns-key
• TSIG algorithm: hmac-sha256

Deploy ExternalDNS​

1. Create the namespace for ExternalDNS.

   kubectl create namespace external-dns
   

   Copy

2. Create a secret that contains the TSIG secret.

   kubectl create secret generic external-dns-tsig \
     --namespace external-dns \
     --from-literal=tsig-secret='<base64-tsig-secret>'
   

   Copy

3. Deploy ExternalDNS with the RBAC resources described in the ExternalDNS RFC 2136 tutorial, and configure the container as follows:

   containers:
   - name: external-dns
     image: registry.k8s.io/external-dns/external-dns:v0.15.0
     args:
     - --source=service
     # Watch only the services created by the Harvester load balancer
     - --label-filter=loadbalancer.harvesterhci.io/servicelb=true
     # Generate a DNS name for each load balancer
     - --fqdn-template={{ .Name }}.{{ .Namespace }}.vm.example.com
     - --domain-filter=vm.example.com
     - --provider=rfc2136
     - --rfc2136-host=192.168.100.53
     - --rfc2136-port=53
     - --rfc2136-zone=vm.example.com
     - --rfc2136-tsig-keyname=externaldns-key
     - --rfc2136-tsig-secret-alg=hmac-sha256
     - --rfc2136-tsig-axfr
     - --registry=txt
     - --txt-owner-id=harvester-cluster-1
     - --policy=upsert-only
     env:
     - name: EXTERNAL_DNS_RFC2136_TSIG_SECRET
       valueFrom:
         secretKeyRef:
           name: external-dns-tsig
           key: tsig-secret
   

   Copy

   With this configuration, ExternalDNS publishes a DNS record named <load-balancer-name>.<namespace>.vm.example.com for each VM load balancer.

4. Create a VM load balancer.

5. Verify that the DNS record resolves to the IP address of the load balancer.

   dig +short <load-balancer-name>.<namespace>.vm.example.com @192.168.100.53
   

   Copy

Assign a Custom Hostname​

To use a specific hostname instead of the name generated by --fqdn-template, add the external-dns.alpha.kubernetes.io/hostname annotation to the load balancer's service.

Example:

kubectl annotate service <load-balancer-name> --namespace <namespace> \
  external-dns.alpha.kubernetes.io/hostname=web.vm.example.com

DNS Record Lifecycle​

ExternalDNS manages the DNS records of each load balancer according to the value of the --policy flag:

In both modes, ExternalDNS only modifies the records that it owns, and does not change records that were created by other means.

Publish DNS Records for Guest Cluster Load Balancers​

LoadBalancer services in guest clusters that use the Harvester cloud provider are standard Kubernetes services. To publish DNS records for these services, deploy ExternalDNS in the guest cluster and configure it for your DNS server. You can then add the external-dns.alpha.kubernetes.io/hostname annotation to each service.

The IP address of a guest cluster load balancer appears in the service's status.loadBalancer.ingress field only after the service has at least one ready endpoint. Until then, ExternalDNS does not publish a DNS record for the service.

For more information, see the ExternalDNS documentation.

DNS Server Integrations Included in ExternalDNS​

ExternalDNS includes integrations (providers) for a wide range of DNS servers. These integrations are developed and maintained by the ExternalDNS project. The following table lists providers that are commonly used in on-premises environments:

For a complete list of providers, including public cloud DNS services, see the ExternalDNS documentation.

This article explains how to install the NetApp Trident CSI driver on a Harvester cluster. Once it's installed, VMs running in Harvester can store their volumes on a NetApp ONTAP storage system.

NetApp storage is added alongside Longhorn; it does not replace it. You can use any combination of these protocols:

• NFS (ontap-nas)
• iSCSI (ontap-san, sanType: iscsi)
• NVMe/TCP (ontap-san, sanType: nvme)
• Fibre Channel (ontap-san, sanType: fcp)

This procedure was tested with Harvester v1.9.0 and Trident 26.06.1 (Helm chart 100.2606.1), using ONTAP 9.16.1. With NFS, iSCSI and NVMe/TCP, the following worked on NetApp-backed volumes: booting VMs, live migration, online volume expansion, and Harvester VM snapshots. Trident also supports Fibre Channel, and the multipath configuration below covers FC LUNs. However, FC was not tested for this article.

Prerequisites​

• A Harvester cluster and its KubeConfig. To download it, open the Harvester UI, click Support in the lower left, then click Download KubeConfig.
• A workstation with kubectl and helm that can reach the Harvester cluster.
• An ONTAP SVM with:
  • A management LIF that the Harvester nodes can reach.
  • Data LIFs for each protocol you plan to use: NFS, iSCSI, NVMe/TCP and/or FC. NVMe/TCP requires the NVMe_oF license on the ONTAP system.
  • An ONTAP login for Trident with the http and ontapi applications enabled, using the SVM-scoped vsadmin role. Point managementLIF at the SVM's own management LIF; Trident then infers the SVM from that connection, so the backends below don't need an explicit svm field either. See User permissions in the Trident documentation.
• For FC only: FC HBAs in every Harvester node that mounts NetApp volumes, and fabric zoning between those HBAs and the SVM's FC LIFs.

Network design​

Trident's node plugin runs in the host network namespace, so the host's routing table decides which NIC carries storage traffic.

How Trident discovers data LIFs differs by protocol:

• iSCSI / NVMe/TCP: Trident ignores dataLIF on SAN backends and logs Specifying data LIF is no longer supported for SAN backends. It asks ONTAP for every SAN LIF in the SVM, and each node logs in to all of them, so multipath works automatically. You only need managementLIF.
• NFS: dataLIF takes a single address. If you leave it out, Trident uses the first NFS LIF it finds in the SVM. Set it explicitly to choose the network that carries NFS traffic.
• FC: traffic runs over the FC fabric, not the IP network, so the rest of this section does not apply. Only the management LIF must be reachable over IP.

Optional: a dedicated storage interface with HostNetworkConfig​

This takes two steps: create a cluster network on the storage NICs, then give each host an IP address on it with a HostNetworkConfig.

Step 1: create the cluster network. A cluster network groups the storage NICs into a bridge on each node. See Create a New Cluster Network for details.

1. In the Harvester UI, go to Networks > ClusterNetworks/Configs and click Create. Name it, for example cn-storage.
2. Click Create Network Config on the new cluster network.
3. On the Node Selector tab, select the nodes that mount NetApp volumes. Select all nodes does not work if the cluster has a witness node. Select nodes by name, or by a label such as netapp-trident=enabled (see the note further down).
4. On the Uplink tab, select the storage NICs, the bond mode, and the MTU. The MTU must match the switch ports and the ONTAP LIFs, for example 9000 for jumbo frames.
5. Click Save.

The switch ports for these NICs must be trunk ports that carry the storage VLAN tagged.

Step 2: give the hosts an IP address on it. On its own, a cluster network carries only VM traffic, because the hosts have no IP address on it. A HostNetworkConfig (Networks > Host Networks) creates a VLAN sub-interface on the cluster network's bridge on each node and gives it an IP address. Traffic to the ONTAP data LIFs on that subnet then leaves through that interface instead of the management network. The ONTAP data LIFs must be on the same VLAN and subnet.

Example with static addresses on VLAN 150:

apiVersion: network.harvesterhci.io/v1beta1
kind: HostNetworkConfig
metadata:
  name: netapp-storage
spec:
  clusterNetwork: cn-storage
  vlanID: 150
  mode: static
  nodeSelector:
    matchLabels:
      netapp-trident: enabled
  ips:
    <node-1>: 192.168.150.11/24
    <node-2>: 192.168.150.12/24

Check the result on a node with ip addr show cn-storage-br.150. Each node's state is reported in the resource's status.nodeStatus.

For iSCSI, you can add a second cluster network (or a second VLAN) with its own HostNetworkConfig and LIFs, so multipath has two independent paths.

1. Configure multipathd (iSCSI and FC)​

iSCSI and FC LUNs need multipathd, which is not enabled on Harvester by default. Both protocols present NetApp LUNs with vendor NETAPP and product LUN, so the same configuration covers them. The configuration below lets multipathd claim only NetApp LUNs. Every other device is blacklisted, so Longhorn and the local disks are never touched. Harvester's own /etc/multipath/conf.d/99-longhorn.conf is left as is.

The following CloudInit resource writes /etc/multipath/conf.d/98-netapp.conf and enables multipathd on every node. It is re-applied on every boot. Save it as 01-cloudinit-netapp-multipathd.yaml:

# Persists across reboots: writes /etc/multipath/conf.d/98-netapp.conf (multipath ONLY claims NETAPP LUNs)
apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: netapp-multipathd
spec:
  matchSelector: {}
  filename: 98_netapp_multipathd.yaml
  contents: |
    name: "enable multipathd for NetApp Trident iSCSI/FC (NETAPP LUNs only)"
    stages:
      initramfs:
        - files:
            - path: /etc/multipath/conf.d/98-netapp.conf
              permissions: 0644
              owner: 0
              group: 0
              content: |
                defaults {
                    user_friendly_names yes
                    find_multipaths no
                    # lets libmpathpersist (qemu-pr-helper) track SCSI-3 PR keys across paths, for shared disks (e.g. WSFC)
                    reservation_key file
                    # don't create kpartx maps for partitions that guests create on the LUNs
                    skip_kpartx yes
                }
                blacklist {
                    device {
                        vendor ".*"
                        product ".*"
                    }
                }
                blacklist_exceptions {
                    device {
                        vendor "NETAPP"
                        product "LUN.*"
                    }
                }
      network:
        - name: "enable and start multipathd"
          systemctl:
            enable:
              - multipathd
            start:
              - multipathd

kubectl apply -f 01-cloudinit-netapp-multipathd.yaml

Despite the initramfs stage name, Harvester's node CloudInit controller writes the file and runs the network stage as soon as the resource is created — no reboot needed. Confirm the file actually merged into the running daemon's config (don't rely on multipath -ll alone, since it's empty if no LUN happens to be attached yet):

multipathd -k"show config" | grep -A6 '^blacklist {'   # must show vendor ".*" / product ".*"

If you'd rather apply it by hand instead of waiting on the controller, run the following on each node as root. It writes the same file and starts multipathd, and stops at the first failure:

set -e
mkdir -p /etc/multipath/conf.d
cat > /etc/multipath/conf.d/98-netapp.conf <<'EOF'
defaults {
    user_friendly_names yes
    find_multipaths no
    reservation_key file
    skip_kpartx yes
}
blacklist {
    device {
        vendor ".*"
        product ".*"
    }
}
blacklist_exceptions {
    device {
        vendor "NETAPP"
        product "LUN.*"
    }
}
EOF
systemctl enable --now multipathd
multipath -ll   # must not list any local disk

Check that every node has a unique iSCSI initiator name. Nodes cloned from the same image can end up with the same name in /etc/iscsi/initiatorname.iscsi. ONTAP then treats those nodes as a single host. If you change a node's IQN, restart that node's trident-node-linux pod afterwards. Trident caches the IQN, and with a stale one volume staging fails with no devices present yet.

2. Install the Trident operator​

helm repo add netapp-trident https://netapp.github.io/trident-helm-chart
helm repo update
helm search repo netapp-trident --versions | head

helm upgrade --install trident netapp-trident/trident-operator --version <chart version> \
  --namespace trident --create-namespace --wait

Pick a version from the helm search output above rather than pinning one in this article; this procedure was tested with chart 100.2606.1 (Trident 26.06.1).

Wait until the trident-controller pod and one trident-node-linux pod per node are Running:

kubectl -n trident get pods -o wide

kubectl label node <node-1> <node-2> netapp-trident=enabled --overwrite

helm upgrade --install trident netapp-trident/trident-operator --version <chart version> \
  --namespace trident --create-namespace --wait \
  --set tridentNodePluginNodeSelector.netapp-trident=enabled

3. Create the ONTAP credentials Secret​

kubectl -n trident create secret generic netapp-creds \
  --from-literal=username=<ontap user> \
  --from-literal=password='<ontap password>'

4. Create the backends​

Create one TridentBackendConfig per protocol you use; you can drop any you don't need.

A few things the examples below deliberately leave out:

• spaceReserve: none and snapshotReserve: "0": these are already Trident's defaults, so setting them changes nothing. (Don't confuse snapshotReserve with thin provisioning, either — it reserves extra space on the ONTAP volume so a PVC's filesystem doesn't run out of room as Snapshot copies grow, it isn't what makes the volume thin.)
• limitAggregateUsage: this stops Trident from creating new volumes once an aggregate passes a given percent used, but aggregates are cluster-level ONTAP objects, so it only works with a cluster-scoped admin login — strictly more access than Trident needs for anything else in this article. On all-flash aggregates there's also no performance cliff near 80% full the way there was with spinning disks, so there's less reason to want it. Watch aggregate usage with your normal ONTAP monitoring instead.
• svm and dataLIF: not needed if managementLIF already points at an SVM-scoped management LIF (Trident infers the SVM from that LIF) and the SVM has only one data LIF per protocol, which is the recommended design. If your SVM has several LIFs for the same protocol, set dataLIF on the NFS backend to pick one explicitly (see Network design).
• nfsMountOptions: this is a Trident-side fallback, not the Kubernetes-native way to set mount options — it doesn't show up in the resulting PV's mountOptions field. Set mount options on the StorageClass instead (step 5).

If several Kubernetes or Harvester clusters share one SVM, give each cluster a unique backendName and set storagePrefix (for example storagePrefix: hv1_). That keeps volume names from colliding and makes each cluster's volumes easy to identify in ONTAP.

For Fibre Channel, copy the iSCSI backend below, then change name and backendName (for example to netapp-fc) and set sanType: fcp. Then add a matching StorageClass in step 5 with sanType: fcp. FC was not tested for this article.

Save as 03-backends.yaml, replacing the <...> values:

# NFS
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-nas
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-nas
  backendName: netapp-nas
  managementLIF: <SVM management LIF IP>
  autoExportPolicy: true
  autoExportCIDRs: ["<storage network CIDR>"]
  credentials:
    name: netapp-creds
---
# iSCSI (all iSCSI LIFs in the SVM are discovered; no dataLIF)
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-iscsi
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-san
  sanType: iscsi
  backendName: netapp-iscsi
  managementLIF: <SVM management LIF IP>
  credentials:
    name: netapp-creds
---
# NVMe/TCP (all NVMe LIFs in the SVM are discovered; no dataLIF)
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-nvme
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-san
  sanType: nvme
  backendName: netapp-nvme
  managementLIF: <SVM management LIF IP>
  credentials:
    name: netapp-creds

kubectl apply -f 03-backends.yaml
kubectl -n trident get tridentbackendconfigs

Every backend must report PHASE Bound and STATUS Success. If one doesn't, kubectl -n trident describe tridentbackendconfig <name> shows the error.

5. Create the StorageClasses and VolumeSnapshotClass​

A StorageClass with only backendType: ontap-san can place a volume on any SAN backend: iSCSI, NVMe/TCP or FC. Pin each class to a protocol with the sanType parameter (iscsi, nvme or fcp) instead of matching backends by name with storagePools — it's the more direct mechanism and keeps working if you ever rename a backend. storagePools is still the right tool if you run multiple SVMs and need to steer volumes to one of them.

For the NFS class, add fsType: "nfs". It's not needed to mount the volume, but without it Kubernetes skips applying a pod's fsGroup to the volume (no setgid bit, no group ownership change) — worth the one extra line. Mount options for NFS also belong here, as the StorageClass's top-level mountOptions, not on the backend.

Save as 04-storageclasses.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-nfs
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-nas"
  fsType: "nfs"
mountOptions:
  - nfsvers=4.1
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-iscsi
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-san"
  sanType: "iscsi"
  fsType: "ext4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-nvme
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-san"
  sanType: "nvme"
  fsType: "ext4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: trident-snapshotclass
driver: csi.trident.netapp.io
deletionPolicy: Delete

kubectl apply -f 04-storageclasses.yaml

Live migration requires ReadWriteMany volumes. CDI already knows Trident's defaults:

• ontap-san (iSCSI, NVMe/TCP and FC): ReadWriteMany with volumeMode: Block.
• ontap-nas: ReadWriteMany with volumeMode: Filesystem.

So these StorageClasses don't need the cdi.harvesterhci.io/storageProfileVolumeModeAccessModes annotation. None of the other cdi.harvesterhci.io/storageProfile* annotations from older third-party storage write-ups are needed either — check kubectl get storageprofile <storageclass> -o yaml after creating the class, and its status is already filled in (access modes, cloneStrategy: snapshot, snapshotClass) because CDI recognizes csi.trident.netapp.io directly. The separate storageclass.kubevirt.io/is-default-virt-class annotation is unrelated to any of this — it only matters if you want a NetApp class, instead of Longhorn, to be the default for new VM disks.

6. Register Trident for Harvester VM snapshots​

Harvester needs to know which VolumeSnapshotClass to use for Trident volumes.

1. In the Harvester UI, go to Advanced > Settings, find csi-driver-config, and select ⋮ > Edit Setting.
2. Click Add, then fill in the fields:
   • Provisioner: csi.trident.netapp.io
   • Volume Snapshot Class Name: trident-snapshotclass
3. Save.

Leave Backup Volume Snapshot Class Name empty. In Harvester v1.9, VM backups only work with Longhorn volumes. VM snapshots of Trident volumes work. See Third-Party Storage Support.

The resulting entry in the setting's JSON value is:

"csi.trident.netapp.io": {
  "volumeSnapshotClassName": "trident-snapshotclass",
  "backupVolumeSnapshotClassName": ""
}

7. Use it​

The new StorageClasses now appear when you create a volume, a VM image or a VM in the Harvester UI. A quick check:

1. Create a VM whose root disk uses netapp-iscsi, netapp-nvme or netapp-nfs.
2. Start the VM, live-migrate it to another node, and take a VM snapshot.
3. On a node running the VM, multipath -ll (iSCSI) or ls /sys/class/nvme-subsystem (NVMe) shows the NetApp devices.

Troubleshooting​

• Backends go offline after the ONTAP system was unreachable. If they don't recover on their own, restart the controller: kubectl -n trident rollout restart deploy/trident-controller.
• Deleted volumes still use space on ONTAP. ONTAP keeps deleted volumes in the volume recovery queue for 12 hours. While a deleted FlexClone (such as a VM disk cloned from an image) sits in the queue, its parent snapshot stays busy, so the parent volume and its VolumeSnapshot can't be deleted. To free them immediately, run on ONTAP: set -privilege advanced; volume recovery-queue purge-all -vserver <svm name>.
• multipath -ll lists local disks. multipathd was started without the NetApp-only configuration. Put 98-netapp.conf in place, flush the unused maps with multipath -F, and restart multipathd.
• iSCSI volumes fail to stage with no devices present yet. Check for duplicate initiator names across nodes, and for a stale IQN in Trident (see step 1).

References​

• NetApp Trident documentation
• Deploy the Trident operator using Helm
• ONTAP SAN driver and ONTAP NAS driver
• Harvester third-party storage support

Harvester v1.7.0 introduced the kubevirt-migration setting, which exposes the cluster-wide live migration configuration that previously could only be changed by editing the kubevirt object directly.

If you customized spec.configuration.migrations on the kubevirt object before upgrading, that customization is at risk. This article explains why and provides a script that converts it into the setting so it is preserved.

Who is affected​

You are affected if both of the following are true:

• You are running a version earlier than v1.7.0 and you changed spec.configuration.migrations on the kubevirt object, for example to enable allowAutoConverge.
• You plan to upgrade to v1.7.0 or later.

You are also affected if you have already upgraded to v1.7.x and have not yet changed the kubevirt-migration setting. In that case the configuration is still intact and the same script applies.

Check whether the kubevirt object carries a configuration:

kubectl get kubevirt kubevirt -n harvester-system -o jsonpath='{.spec.configuration.migrations}' | jq

If the command returns nothing, your cluster uses the KubeVirt defaults and no action is required.

What happens without this procedure​

The upgrade creates the kubevirt-migration setting without a value, so the Harvester UI displays the default values. The kubevirt object keeps your configuration, so live migration continues to behave as before.

The two are now inconsistent, and the inconsistency is not visible unless you compare them. Harvester writes the whole value of the setting to the kubevirt object, so the first time the setting is saved, all fields are written at once, including the fields that you did not change. Your configuration is replaced by the defaults, without warning.

In practice this means that changing an unrelated field such as parallelMigrationsPerCluster in the UI silently turns allowAutoConverge back off. Migrations of write-heavy virtual machines then start failing to converge, which is often noticed only during the node drains of a later upgrade.

Procedure​

Run the following script from a machine with kubectl access to the cluster. jq is required.

Run it before you upgrade to v1.7.0 or later. If you already upgraded to v1.7.x and have not yet changed the setting, run it now.

#!/bin/bash
set -euo pipefail

SETTING_NAME="kubevirt-migration"

migrations=$(kubectl get kubevirt kubevirt -n harvester-system -o json | jq -c '.spec.configuration.migrations // empty')
if [ -z "$migrations" ]; then
  echo "The kubevirt object has no migration configuration. Nothing to preserve."
  exit 0
fi

# nodeDrainTaintKey and network cannot be configured through the setting:
# nodeDrainTaintKey is used by the upgrade process, and network is owned by
# the vm-migration-network setting. The webhook rejects a value containing them.
value=$(echo "$migrations" | jq -c 'del(.nodeDrainTaintKey, .network)')
if [ "$value" = "{}" ]; then
  echo "The migration configuration only contains fields that the setting does not manage. Nothing to preserve."
  exit 0
fi

if setting_json=$(kubectl get settings.harvesterhci.io "$SETTING_NAME" -o json 2>/dev/null); then
  current_value=$(echo "$setting_json" | jq -r '.value // empty')
  if [ -n "$current_value" ]; then
    echo "The $SETTING_NAME setting already has a value. Nothing to do."
    exit 0
  fi

  echo "Setting $SETTING_NAME to: $value"
  kubectl patch settings.harvesterhci.io "$SETTING_NAME" --type merge \
    -p "$(jq -n --arg v "$value" '{"value": $v}')"
else
  echo "Creating $SETTING_NAME with: $value"
  kubectl apply -f - <<EOF
apiVersion: harvesterhci.io/v1beta1
kind: Setting
metadata:
  name: $SETTING_NAME
value: '$value'
EOF
fi

echo "Result:"
kubectl get settings.harvesterhci.io "$SETTING_NAME" -o yaml

The script does not write to the kubevirt object, so it cannot change how the cluster performs live migrations. It is idempotent: it does nothing if there is no configuration to preserve, or if the setting already has a value.

Verification​

Confirm that the setting now holds your configuration:

kubectl get settings.harvesterhci.io kubevirt-migration -o jsonpath='{.value}' | jq

Confirm that the kubevirt object is unchanged:

kubectl get kubevirt kubevirt -n harvester-system -o jsonpath='{.spec.configuration.migrations}' | jq

After you upgrade, the two remain consistent, and the Harvester UI shows the configuration that the cluster is actually using.

After the upgrade​

The kubevirt-migration setting is the source of truth. Harvester writes it to the kubevirt object whenever the setting changes.

Do not edit spec.configuration.migrations on the kubevirt object directly. Any change is overwritten the next time the setting is reconciled. Use the setting instead, either in the Harvester UI under Advanced > Settings, or with kubectl.

For the list of supported fields and their defaults, see kubevirt-migration in the Harvester documentation.

For background on the individual options and on per-virtual-machine migration policies, see VM Live Migration Policy and Configuration.

A symlink-following vulnerability was found in KubeVirt's virt-handler migration proxy.

During live migration, virt-handler dials Unix sockets inside the target virt-launcher pod using /proc/<pid>/root/<path> via net.Dial(), without symlink protection. These socket paths reside in QEMU-owned directories that are writable by the virt-launcher user.

An attacker with namespace edit and pods/exec permissions can replace a migration proxy socket with a symlink pointing to the host CRI-O socket. Because virt-handler runs as root in the host mount namespace, absolute symlink targets resolve against the host filesystem. The bidirectional io.Copy proxy then relays attacker-controlled bytes to the container runtime, enabling full node compromise.

Affected Versions​

All currently supported Harvester versions are affected, including 1.6.1 and earlier, 1.7.3 and earlier, and 1.8.2 and earlier.

Your cluster is affected by this vulnerability, if the following command returns at least one virt-launcher pods:

kubectl get po -lkubevirt.io=virt-launcher -A

Fixed Versions​

A fix is not yet available. SUSE is actively working on a resolution, targeting Harvester 1.9.

Meanwhile, you can use the following workaround to protect your clusters' virt-launcher pods against exploitation.

Workaround​

Use a ValidatingAdmissionPolicy to deny pods/exec and pods/attach operations targeting virt-launcher pods.

Apply the following resources to your Harvester cluster:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: deny-virt-launcher-exec
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   [""]
      apiVersions: ["v1"]
      operations:  ["CONNECT"]
      resources:   ["pods/exec", "pods/attach"]
  matchConditions:
    - name: is-virt-launcher
      expression: >
        request.name.startsWith("virt-launcher-")
  validations:
    - expression: >
        request.userInfo.username == "system:admin" ||
        request.userInfo.username.startsWith("system:serviceaccount:harvester-system")
      messageExpression: "'CVE-2026-13622 mitigation. Rejected exec/attach request from ' + request.userInfo.username"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: deny-virt-launcher-exec-binding
spec:
  policyName: deny-virt-launcher-exec
  validationActions: [Deny]

This policy blocks exec and attach requests to virt-launcher pods from non-admin and non-Harvester service accounts, to prevent an attacker from replacing the migration proxy sockets with symlinks.

kubectl delete validatingadmissionpolicybinding deny-virt-launcher-exec-binding
kubectl delete validatingadmissionpolicy deny-virt-launcher-exec

References​

• https://github.com/advisories/GHSA-4336-pf6g-hp94
• https://www.suse.com/security/cve/CVE-2026-13622.html

Enabling the Longhorn V2 Data Engine can cause kernel workqueue lockups on some Harvester nodes, leading to system instability. In severe cases, rke2-server terminates, Longhorn stops processing operations, and affected nodes require recovery.

This issue typically occurs when SPDK is pinned to dedicated CPUs, but the Linux kernel continues running IRQ handlers or unbound workqueue workers on those same CPUs.

Affected Scenario​

This issue can affect Harvester clusters that meet all of the following conditions:

• The Longhorn V2 Data Engine is enabled.
• The Longhorn V2 instance-manager pod runs SPDK (spdk_tgt), which busy-polls the CPUs specified in the Longhorn V2 Data Engine CPU mask.
• Host IRQ handlers or unbound kernel workqueues are permitted to run on those same SPDK-designated CPUs.

While the default Longhorn V2 CPU mask is 0x3 (maps to CPUs 0 and 1), this issue can occur with any CPU mask if kernel IRQs or unbound workqueues share the CPUs used by spdk_tgt.

This issue primarily affects bare-metal Harvester nodes running the Longhorn V2 Data Engine. Reproducibility varies depending on hardware and workload patterns.

Symptoms​

Affected nodes may exhibit one or more of the following symptoms:

• Kernel logs repeatedly report BUG: workqueue lockup.
• The lockup message references a CPU core assigned to the Longhorn V2 Data Engine CPU mask.
• rke2-server becomes unstable or terminates without a clear user-visible reason.
• Longhorn volumes, engines, replicas, or instance managers stop progressing.
• Interactive user commands, such as sudo -i, experience severe latency before opening a shell.
• The node becomes unstable or unhealthy.

Example of kernel log records:

Jul 27 07:27:55 hp-114-tink-system kernel: BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 2117s!
Jul 27 07:27:55 hp-114-tink-system kernel: Showing busy workqueues and worker pools:
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events: flags=0x0
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=7 refcnt=8
Jul 27 07:27:55 hp-114-tink-system kernel:     in-flight: 2935011:output_poll_execute ,32008:drm_fb_helper_damage_work drm_fb_helper_damage_work
Jul 27 07:27:55 hp-114-tink-system kernel:     pending: vmstat_shepherd, switchdev_deferred_process_work, rht_deferred_worker, key_garbage_collector
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events_unbound: flags=0x2
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 98: cpus=0-23 node=1 flags=0x4 nice=0 active=4 refcnt=8
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events_power_efficient: flags=0x80
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5
Jul 27 07:27:55 hp-114-tink-system kernel:     pending: neigh_managed_work, neigh_periodic_work, gc_worker [nf_conntrack], check_lifetime
Jul 27 07:27:55 hp-114-tink-system kernel: pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=2117s workers=5 idle: 1025200 1541794
Jul 27 07:27:55 hp-114-tink-system kernel: task:kworker/0:0     state:R  running task     stack:0     pid:32008 tgid:32008 ppid:2      flags:0x00004000
Jul 27 07:27:55 hp-114-tink-system kernel: Workqueue: events drm_fb_helper_damage_work
Jul 27 07:28:12 hp-114-tink-system rke2[2909320]: time="2026-07-27T07:28:12Z" level=warning msg="Proxy error: write failed: write tcp 127.0.0.1:9345->127.0.0.1:46562: write: connection reset by peer"

Root Cause​

The Longhorn V2 Data Engine relies on SPDK, whose reactor threads busy-poll on the CPUs designated by the engine's CPU mask. This continuous polling is expected behavior for high-performance storage processing.

The issue occurs when normal host kernel work continues running on those same CPUs, including the following:

• Network or storage IRQs target SPDK CPUs.
• Unbound kernel workqueues retain SPDK CPUs in /sys/devices/virtual/workqueue/cpumask.
• Per-workqueue CPU masks allow existing workers to run on SPDK CPUs.

When SPDK consumes 100% of these CPU cores, pending kernel tasks are starved, eventually triggering the kernel workqueue lockup detector. While isolated lockup warnings may not always result in an unrecoverable node, continuous task starvation cascades into network, RKE2, kubelet, and Longhorn control-plane failure.

To prevent this, all IRQs and kernel workqueues must be explicitly re-routed away from the CPUs defined in the Longhorn V2 CPU mask (for example, CPUs 0 and 1 when using the default mask 0x3).

Confirming the Issue​

Run the following checks on each affected Harvester node.

1. Check Kernel Logs​

journalctl -k --since "24 hours ago" | grep -E "BUG: workqueue lockup|soft lockup"

Look for the CPU reported in messages such as pool cpus=0, pool cpus=1, kworker/0, or kworker/1.

2. Check the Longhorn V2 CPU Mask​

kubectl -n longhorn-system get settings.longhorn.io data-engine-cpu-mask -o jsonpath='{.value}{"\n"}'

The value may be data-engine-specific. For example:

{"v2":"0x3"}

Some Longhorn versions support CPU-list input for this setting. If the value is already a CPU list, use that list directly in the checks below.

Convert the mask to CPU IDs:

python3 - <<'PY'
mask = int("0x3", 0)
print(",".join(str(cpu) for cpu in range(mask.bit_length()) if mask & (1 << cpu)))
PY

For 0x3, the output is:

0,1

3. Check SPDK Placement​

SPDK_PIDS=$(pgrep -f '[s]pdk_tgt' || true)

for pid in ${SPDK_PIDS}; do
  ps -T -p "${pid}" -o pid,tid,psr,stat,pcpu,comm,args
done

This command matches the full command line because the SPDK process or its threads may appear as reactor_<cpu> in the kernel process name. In that case, pgrep -x spdk_tgt can return empty even when spdk_tgt is running.

Example output for the default CPU mask 0x3:

    PID     TID PSR STAT %CPU COMMAND         COMMAND
2913161 2913161   0 RLl  99.9 reactor_0       spdk_tgt -L all --mem-size 2048 -m 0x3
2913161 2913166  21 SLl   0.0 dpdk-intr       spdk_tgt -L all --mem-size 2048 -m 0x3
2913161 2913220   1 RLl  99.9 reactor_1       spdk_tgt -L all --mem-size 2048 -m 0x3

The PSR column shows the CPU that each thread is running on. In the example above, reactor_0 is running on CPU 0 and reactor_1 is running on CPU 1, so the SPDK CPU list is 0,1.

To check whether running or blocked kernel workers are currently on the SPDK CPUs, use the following more targeted command. Replace SPDK_CPUS with the comma-separated CPU IDs from the reactor_* rows.

SPDK_CPUS="0,1"

ps -eLo pid,tid,psr,stat,pcpu,wchan:30,comm,args | \
  awk -v cpus="${SPDK_CPUS}" '
    BEGIN {
      printf "%7s %7s %3s %-5s %5s %-30s %-16s %s\n", "PID", "TID", "PSR", "STAT", "%CPU", "WCHAN", "COMMAND", "ARGS"
      split(cpus, cpu_list, ",")
      for (i in cpu_list) {
        spdk_cpu[cpu_list[i]] = 1
      }
    }
    NR > 1 && spdk_cpu[$3] && $4 ~ /[RD]/ && ($7 ~ /^kworker\// || $7 ~ /^ksoftirqd\//) {
      print
    }
  '

On nodes that are not affected, this command normally prints only the header line. If the header line is removed from the command, empty output is expected. This means there are no running or uninterruptible kworker/* or ksoftirqd/* threads on the SPDK CPUs at that moment.

Example output showing kernel work on an SPDK CPU:

    PID     TID PSR STAT   %CPU WCHAN                          COMMAND          ARGS
     17      17   0 R     0.0 -                              ksoftirqd/0     [ksoftirqd/0]
  32008   32008   0 R     0.0 -                              kworker/0:0+eve [kworker/0:0+events]
2935011 2935011   0 D     0.0 mgag200_ddc_algo_bit_data_pre_ kworker/0:2+eve [kworker/0:2+events]
4012518 4012518   0 R     0.0 -                              kworker/0:3+mm_ [kworker/0:3+mm_percpu_wq]

In this example, the PSR value is 0, and the process names are ksoftirqd/0 and kworker/0:*. This means kernel softirq and workqueue threads are running or blocked on CPU 0.

If the SPDK reactor threads are running on the CPUs reported in the workqueue lockup, or if the targeted command shows kworker/* or ksoftirqd/* activity on the SPDK CPUs, continue with the IRQ and workqueue checks.

4. Check IRQ Affinity​

The following example checks CPUs 0 and 1 because the reactor_* rows above are running on those CPUs. Replace SPDK_CPUS with the CPU IDs used in your environment.

SPDK_CPUS="0,1"

cpu_list_overlaps() {
  python3 - "$1" "$2" <<'PY'
import sys

target = {int(cpu) for cpu in sys.argv[1].split(",") if cpu}
seen = set()

for part in sys.argv[2].split(","):
    part = part.strip()
    if not part:
        continue
    if "-" in part:
        start, end = map(int, part.split("-", 1))
        seen.update(range(start, end + 1))
    else:
        seen.add(int(part))

sys.exit(0 if target & seen else 1)
PY
}

for irqdir in /proc/irq/[0-9]*; do
  irq=${irqdir##*/}
  eff=$(cat "${irqdir}/effective_affinity_list" 2>/dev/null || true)
  conf=$(cat "${irqdir}/smp_affinity_list" 2>/dev/null || true)

  if cpu_list_overlaps "${SPDK_CPUS}" "${eff}"; then
    echo "IRQ=${irq} configured=${conf} effective=${eff}"
    grep -w "^ *${irq}:" /proc/interrupts 2>/dev/null || true
  fi
done

Example output:

IRQ=108 configured=0-5,12-17 effective=0
 108:   25762242          0 ... IR-PCI-MSIX-0000:04:00.0    3-edge      netboot-TxRx-3
IRQ=109 configured=0-5,12-17 effective=1
 109:          0   15555565 ... IR-PCI-MSIX-0000:04:00.0    4-edge      netboot-TxRx-4
IRQ=120 configured=0-5,12-17 effective=0
 120:   10898692          0 ... IR-PCI-MSIX-0000:04:00.0   15-edge      netboot-TxRx-15
IRQ=121 configured=0-5,12-17 effective=1
 121:          0    8135324 ... IR-PCI-MSIX-0000:04:00.0   16-edge      netboot-TxRx-16
IRQ=92 configured=0 effective=0
  92:    2242087          0 ... IR-PCI-MSIX-0000:08:00.0   13-edge      nvme0q13
IRQ=93 configured=1 effective=1
  93:          0     448454 ... IR-PCI-MSIX-0000:08:00.0   14-edge      nvme0q14

The effective value shows where the IRQ is actually running. In this example, NIC queues and NVMe queues are effectively landing on CPUs 0 and 1, which are the default SPDK reactor CPUs for mask 0x3. If device IRQs, especially high-traffic NIC or storage IRQs, are effectively landing on SPDK CPUs, the node is at risk.

5. Check Kernel Workqueue Masks​

cat /sys/devices/virtual/workqueue/cpumask

for f in /sys/devices/virtual/workqueue/*/cpumask; do
  echo "${f}: $(cat "${f}")"
done

Example output:

/sys/devices/virtual/workqueue/blkcg_punt_bio/cpumask: ffffff
/sys/devices/virtual/workqueue/ib-comp-unb-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/iscsi_conn_cleanup/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-auth-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-delete-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-reset-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/scsi_tmf_0/cpumask: ffffff
/sys/devices/virtual/workqueue/writeback/cpumask: ffffff

On a 24-CPU node, ffffff means the workqueue can run on CPUs 0-23. If SPDK uses the default CPU mask 0x3, CPUs 0 and 1 are included in this workqueue mask. If the global or per-workqueue masks include the SPDK CPUs, unbound kernel work may still run on the SPDK CPUs.

Preferred Risk-Reduction Setting​

Longhorn added the data-engine-cpu-isolation-enabled setting to reduce the chance of this issue. When enabled for the V2 Data Engine, the Longhorn V2 instance-manager:

• Persists the SPDK CPU mask under /var/lib/longhorn/instance-manager/v2/spdk_cpu_mask on the host.
• Programs /proc/irq/*/smp_affinity to the inverse of the SPDK CPU mask.
• Writes the same inverse mask to /sys/devices/virtual/workqueue/cpumask.
• Updates per-workqueue CPU masks when possible.
• Reconciles stale affinity state on the next instance-manager restart if the setting is later disabled.

This setting steers IRQs and unbound workqueues away from SPDK CPUs. In some environments, this may stop the workqueue lockup messages entirely. In others, it may only reduce how often they occur because CPU-bound or per-CPU kernel workers such as kworker/0:* are tied to a specific CPU and cannot be moved by the unbound workqueue CPU mask.

Use this setting when the Longhorn version bundled with Harvester includes it. If the Longhorn setting exists but Harvester does not expose it in the UI, you can still configure it through the Longhorn setting resource. For Harvester versions earlier than v1.9.0, the bundled Longhorn version does not include the setting, so use the manual workaround in the next section.

If the setting exists in your Longhorn version, you can check it with:

kubectl -n longhorn-system get settings.longhorn.io data-engine-cpu-isolation-enabled

Enable it for the V2 Data Engine:

kubectl -n longhorn-system patch settings.longhorn.io data-engine-cpu-isolation-enabled \
  --type=merge \
  -p '{"value":"{\"v2\":\"true\"}"}'

Then wait for the V2 instance-manager pods to be recreated after all V2 volumes are detached.

Verify the instance-manager log:

kubectl -n longhorn-system logs <v2-instance-manager-pod> | \
  grep -E "Setting IRQ affinity|Setting workqueue cpumask|Applied IRQ affinity|Applied global workqueue cpumask"

Example expected messages:

Setting IRQ affinity to exclude SPDK CPUs
Applied IRQ affinity mask
Setting workqueue cpumask to exclude SPDK CPUs
Applied global workqueue cpumask
Applied per-workqueue cpumask

Manual Workaround for Versions Without the Longhorn Setting​

Use this workaround for Harvester versions earlier than v1.9.0, where the bundled Longhorn version does not include data-engine-cpu-isolation-enabled. For later versions, first check whether the Longhorn setting exists and prefer the setting-based risk-reduction path when possible.

The goal is to move IRQs and unbound workqueues away from the SPDK CPUs. For example, if SPDK uses the default CPUs 0 and 1, IRQs and unbound workqueues should use CPU 2 through the last online CPU.

This workaround is a risk-reduction step, not a guaranteed fix for every workqueue lockup. CPU-bound or per-CPU kernel workers can still run on the SPDK CPUs because they are tied to those CPUs by the kernel. The expected result is that the node remains stable and any remaining workqueue stalls recover quickly instead of hanging for a long time.

Repeat this workaround whenever the Longhorn V2 CPU mask changes. If CPU allocation is managed dynamically, the IRQ and workqueue masks must be recalculated after each placement change.

1. Identify the Non-SPDK CPUs and Mask​

The default Longhorn V2 CPU mask is 0x3, so SPDK uses CPUs 0 and 1.

On the example node, all online CPUs are 0-23:

cat /sys/devices/system/cpu/online

0-23

Use the following values for this default example:

The Linux affinity mask is a CPU bitmap written in hexadecimal. CPU 0 is bit 0, CPU 1 is bit 1, and so on. On a 24-CPU node, all CPUs enabled is ffffff. Excluding CPUs 0 and 1 clears the lowest two bits, so the mask becomes fffffc. Leading zeros are optional, so fffffc and 00fffffc are equivalent.

Do not copy these values blindly if your Longhorn V2 CPU mask or online CPU list is different. The non-SPDK CPU list must be all online CPUs except the CPUs used by the SPDK reactor threads.

2. Apply IRQ Affinity at Runtime​

Set NON_SPDK_AFFINITY_MASK to the Linux affinity mask from the previous step.

NON_SPDK_AFFINITY_MASK=fffffc

echo "${NON_SPDK_AFFINITY_MASK}" > /proc/irq/default_smp_affinity

for f in /proc/irq/[0-9]*/smp_affinity; do
  echo "${NON_SPDK_AFFINITY_MASK}" > "${f}" 2>/dev/null || true
done

Some IRQs may reject affinity updates because they are managed by the kernel. This is expected. Always verify effective_affinity_list after applying the change.

Changing IRQ affinity alone may not make existing workqueue lockup messages disappear. Kernel workqueues can still run on the SPDK CPUs until the workqueue CPU masks are updated, and workers that are already stuck may continue to be reported by the kernel. Apply the workqueue affinity change as well, and then verify whether new lockup messages stop appearing.

These runtime IRQ affinity changes are not persistent across reboot. Use the persistence step later in this section if the runtime change mitigates the issue.

3. Apply Workqueue Affinity at Runtime​

NON_SPDK_AFFINITY_MASK=fffffc

echo "${NON_SPDK_AFFINITY_MASK}" > /sys/devices/virtual/workqueue/cpumask

for f in /sys/devices/virtual/workqueue/*/cpumask; do
  echo "${NON_SPDK_AFFINITY_MASK}" > "${f}" 2>/dev/null || true
done

These runtime workqueue affinity changes are not persistent across reboot. Use the persistence step later in this section if the runtime change mitigates the issue.

4. Restart the V2 Instance-Manager Pod​

After changing both IRQ and workqueue affinity, restart the Longhorn V2 instance-manager pod on the affected node. This is required because the existing spdk_tgt process can keep the SPDK reactor threads on the busy CPUs, and already-stuck CPU-bound workers may continue to be reported by the kernel until spdk_tgt is recreated.

Check the Longhorn V2 volumes first:

kubectl -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,DATAENGINE:.spec.dataEngine,STATE:.status.state,NODE:.status.currentNodeID

Only continue after all v2 volumes whose NODE is the affected node are detached.

Find the V2 instance-manager pod on the affected node:

AFFECTED_NODE=hp-114-tink-system

kubectl -n longhorn-system get pods \
  -l longhorn.io/component=instance-manager,longhorn.io/data-engine=v2 \
  --field-selector spec.nodeName="${AFFECTED_NODE}" \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Restart the V2 instance-manager pod:

V2_IM_POD=<v2-instance-manager-pod-name>

kubectl -n longhorn-system delete pod "${V2_IM_POD}"

5. Make IRQ and Workqueue Affinity Persistent​

The previous IRQ and workqueue affinity commands only change runtime procfs and sysfs state, so they are lost after reboot. If those runtime changes reduce or stop new lockup messages, use a Harvester CloudInit resource to reapply both IRQ and workqueue affinity after reboot. This follows the same model as the Longhorn CPU isolation setting, which updates host IRQ and workqueue affinity at runtime.

The following manifest is a Harvester CloudInit resource, not a file that you manually place on each node. Save it on a machine with kubectl access to the Harvester cluster and apply it with kubectl apply -f <file-name>.yaml. The Harvester controller writes the content to /oem/99_longhorn_v2_cpu_affinity.yaml on each node matched by matchSelector.

You can add this resource after the cluster is already installed. The file is synchronized to the matched nodes after the resource is applied, but the cloud-init commands take effect only after those nodes are rebooted. After reboot, the commands are applied again on each boot.

Example CloudInit resource for the default SPDK CPUs 0 and 1 on a 24-CPU node:

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: longhorn-v2-cpu-affinity
spec:
  matchSelector:
    kubernetes.io/hostname: "hp-114-tink-system"
  filename: 99_longhorn_v2_cpu_affinity.yaml
  contents: |
    stages:
      network:
        - commands:
            - echo fffffc > /proc/irq/default_smp_affinity
            - for f in /proc/irq/[0-9]*/smp_affinity; do echo fffffc > "${f}" 2>/dev/null || true; done
            - echo fffffc > /sys/devices/virtual/workqueue/cpumask
            - for f in /sys/devices/virtual/workqueue/*/cpumask; do echo fffffc > "${f}" 2>/dev/null || true; done

Adjust the affinity mask and matchSelector for your environment. To target a different affected node, replace hp-114-tink-system with that node's kubernetes.io/hostname label value. Do not use matchSelector: {} unless every node should receive this workaround. This workaround is not dynamically reconciled. If the Longhorn V2 CPU mask changes, update the CloudInit resource and reboot the affected nodes.

After configuring the CloudInit resource, reboot the affected nodes for the commands to take effect. To apply the workaround immediately before reboot, run the runtime commands in the previous steps.

If Lockups Still Appear​

The IRQ and workqueue affinity changes reduce the chance of a workqueue lockup, but they may not eliminate it in every environment. The workqueue CPU mask mainly controls unbound workqueues. Bound or per-CPU workqueues can still run on their associated CPU, so kernel workers such as kworker/20:* may still appear on an SPDK CPU even after unbound workqueues are moved away.

If the node still reports occasional workqueue lockups after the IRQ and workqueue affinity mitigation is applied, use the following additional mitigations.

1. Change the Longhorn V2 Disk Driver to auto​

If the affected Longhorn V2 disk is using the aio disk driver, consider changing the requested disk driver to auto. For an NVMe disk, verify after reprovisioning that Longhorn reports the actual disk driver as nvme in the Longhorn Node custom resource. In recent validation, after applying IRQ affinity and confirming that the actual disk driver was nvme, no new workqueue lockup messages were observed during the test window.

Before changing the disk driver, remove the affected disk from Harvester. Follow the Remove Disks guide, and make sure the disk no longer contains active Longhorn replicas or backing images.

Find the BlockDevice resource for the disk:

kubectl -n longhorn-system get blockdevices.harvesterhci.io \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,PROVISION:.spec.provision,PHASE:.status.provisionPhase,STATE:.status.state,ENGINE:.spec.provisioner.longhorn.engineVersion,DRIVER:.spec.provisioner.longhorn.diskDriver

After the disk is removed and the BlockDevice is no longer provisioned, patch the disk driver:

BLOCKDEVICE=<blockdevice-name>

kubectl -n longhorn-system patch blockdevice.harvesterhci.io "${BLOCKDEVICE}" \
  --type=merge \
  -p '{"spec":{"provisioner":{"longhorn":{"engineVersion":"LonghornV2","diskDriver":"auto"}}}}'

Provision the disk again from the Harvester UI. Harvester will add the disk back to Longhorn V2. Then verify the requested driver in the BlockDevice resource and the actual driver in the Longhorn Node resource.

Verify the BlockDevice after provisioning:

kubectl -n longhorn-system get blockdevice.harvesterhci.io "${BLOCKDEVICE}" -o yaml

The matching BlockDevice should show:

spec:
  provision: true
  provisioner:
    longhorn:
      engineVersion: LonghornV2
      diskDriver: auto
status:
  provisionPhase: Provisioned
  state: Active

Also check the Longhorn Node custom resource to confirm the actual disk driver selected by Longhorn:

kubectl -n longhorn-system get nodes.longhorn.io \
  -o go-template='{{printf "NODE\tDISK\tTYPE\tPATH\tSPEC_DRIVER\tSTATUS_DRIVER\n"}}{{range .items}}{{ $node := . }}{{range $diskName, $disk := .spec.disks}}{{ $status := index $node.status.diskStatus $diskName }}{{printf "%s\t%s\t%s\t%s\t%s\t%s\n" $node.metadata.name $diskName $disk.diskType $disk.path $disk.diskDriver $status.diskDriver}}{{end}}{{end}}'

For the affected disk, SPEC_DRIVER should be auto, and STATUS_DRIVER should be nvme:

NODE                  DISK                                  TYPE    PATH          SPEC_DRIVER   STATUS_DRIVER
hp-161-tink-system    32f43222-1eb1-4ab6-9e65-c4f8ddad700d  block   0000:08:00.0  auto          nvme

The spec.disks.<blockdevice-name>.diskDriver value is the requested driver mode from Harvester. The status.diskStatus.<blockdevice-name>.diskDriver value is the driver Longhorn actually uses. For an NVMe disk, the status value should be nvme.

2. Enable Longhorn V2 Interrupt Mode​

If the lockup still occurs after changing the disk driver and applying IRQ affinity, consider enabling Longhorn V2 interrupt mode.

Check the setting:

kubectl -n longhorn-system get settings.longhorn.io data-engine-interrupt-mode-enabled

Enable interrupt mode for the V2 Data Engine:

kubectl -n longhorn-system patch settings.longhorn.io data-engine-interrupt-mode-enabled \
  --type=merge \
  -p '{"value":"{\"v2\":\"true\"}"}'

Wait for the V2 instance-manager pods to be recreated after all V2 volumes are detached.

Verification​

After applying the mitigation, verify the items that directly confirm the node recovered and the settings are still applied.

1. Kernel logs no longer report new workqueue lockups, or any remaining reports are short and do not keep increasing for a long time.

   journalctl -k --since "30 minutes ago" | grep -E "BUG: workqueue lockup|soft lockup" || true
   

   Copy

   If occasional messages still appear, compare the stuck for <seconds>s value over time. The mitigation is still useful if the stuck time stops growing, the node remains responsive, and RKE2, kubelet, and Longhorn keep making progress.

2. The Longhorn V2 instance-manager pod is running on the affected node.

   AFFECTED_NODE=hp-114-tink-system
   
   kubectl -n longhorn-system get pods \
     -l longhorn.io/component=instance-manager,longhorn.io/data-engine=v2 \
     --field-selector spec.nodeName="${AFFECTED_NODE}"
   

   Copy

3. If you configured the persistent workaround and rebooted the node, the IRQ and workqueue masks are still applied.

   cat /proc/irq/default_smp_affinity
   cat /sys/devices/virtual/workqueue/cpumask
   

   Copy

References​

• Harvester bug: harvester/harvester#11030
• Longhorn bug: longhorn/longhorn#13417
• Longhorn setting reference: Data Engine CPU Mask
• Longhorn setting reference: Data Engine Interrupt Mode Enabled
• Longhorn documentation: Interrupt Mode Support
• Linux kernel documentation: IRQs
• Linux kernel documentation: Workqueue

This article provides information and mitigation steps for the following vulnerabilities in Harvester:

• CVE-2026-53359

The mitigation steps involves disabling the nested virtualization feature of the KVM kernel module on your Harvester hosts.

On your Harvester hosts, use the following commands to confirm that the KVM kernel module is loaded with nested virtualization enabled:

lsmod | grep -iE "kvm_amd|kvm_intel"

sudo grep -H '' /sys/module/{kvm_amd|kvm_intel}/parameters/* 2>&1 |grep nested

If the above commands return module and parameter information about the KVM kernel module, then your host is affected by this vulnerability.

Deploy the following CloudInit configuration to disable the nested virtualization feature of the KVM kernel module on all your Harvester hosts:

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: disabled-nested-virtualization
spec:
  matchSelector:
    harvesterhci.io/managed: "true"
  filename: 99-disabled-nested-virtualization
  contents: |
    stages:
      initramfs:
      - name: "disable nested virtualization in kvm modules"
        files:
        - path: "/etc/modprobe.d/99-disabled-nested-virtualization.conf"
          content: |
            options kvm_amd nested=0
            options kvm_intel nested=0

Once the configuration is applied, reboot your Harvester hosts for the changes to take effect.

Once you have upgraded to a fixed version of Harvester, you can re-enable the nested virtualization feature by deleting the CloudInit configuration and rebooting your Harvester hosts:

kubectl delete cloudinit disabled-nested-virtualization

References​

• https://support.scc.suse.com/s/kb/Security-Vulnerability-x86-64-kvm-Guest-to-Host-escape-aka-Januscape-CVE-2026-53359
• https://www.suse.com/security/cve/CVE-2026-53359.html

When running production workloads on virtualized infrastructure like Harvester, memory management is critical. In Harvester versions prior to v1.4.0, certain workloads experienced sudden Virtual Machine (VM) terminations due to the host Linux operating system triggering Out-Of-Memory (OOM) kills.

What is KubeVirt? Harvester uses KubeVirt as its core virtualization engine. KubeVirt is an open-source technology that allows Kubernetes to run and manage traditional Virtual Machines inside standard containers, translating VM specifications directly into Pod configurations.

This article explores why these OOM events occur in a Kubernetes-native virtualization environment and how Harvester provides granular tools to eliminate them.

Anatomy of a VM OOM Event​

When a VM is terminated due to insufficient memory at the host level, the Linux kernel logs specific keywords that help pinpoint the fault. In Harvester, these logs generally fall into two distinct categories depending on which process triggered the exhaustion.

Example 1: virt-launcher invoked oom-killer

The virt-launcher process runs inside the dedicated Kubernetes Pod backing the VM. If this component or its direct sub-processes run out of the memory allocated to their cgroup, the kernel triggers a memory cgroup (memcg) OOM event.

Feb 03 19:57:08 ** kernel: virt-launcher invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=986
Feb 03 19:57:08 ** kernel: CPU: 40 PID: 40785 Comm: virt-launcher Tainted: G          I    X    5.14.21-150400.24.60-default #1 SLE15-SP4 9096397fa6646928cc6d185ba417f2af65b536f1
...

Feb 03 19:57:08 ** kernel: memory: usage 17024340kB, limit 17024340kB, failcnt 1243
Feb 03 19:57:08 ** kernel: memory+swap: usage 17024340kB, limit 9007199254740988kB, failcnt 0                                                                 
Feb 03 19:57:08 ** kernel: kmem: usage 143556kB, limit 9007199254740988kB, failcnt 0
Feb 03 19:57:08 ** kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice:
...
Feb 03 19:57:08 ** kernel: Tasks state (memory values in pages):
Feb 03 19:57:08 ** kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Feb 03 19:57:08 ** kernel: [  38886]     0 38886      243        1    28672        0          -998 pause
Feb 03 19:57:08 ** kernel: [  38917]     0 38917   310400     6921   192512        0           986 virt-launcher-m
Feb 03 19:57:08 ** kernel: [  38934]     0 38934  1200940    25126   954368        0           986 virt-launcher
Feb 03 19:57:08 ** kernel: [  38951]     0 38951   386525     8247   466944        0           986 libvirtd
Feb 03 19:57:08 ** kernel: [  38952]     0 38952    33619     3940   290816        0           986 virtlogd
Feb 03 19:57:08 ** kernel: [  39079]   107 39079  4457263  4201766 34439168        0           986 qemu-system-x86
Feb 03 19:57:08 ** kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice/cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,task=qemu-system-x86,pid=39079,uid=107

Example 2: CPU X/KVM invoked oom-killer

This occurs when a vCPU execution thread inside qemu-system-x86_64 attempts a memory operation that pushes the entire container beyond its Kubernetes memory limit.

[Thu May  9 14:52:38 2024] CPU 11/KVM invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=830
[Thu May  9 14:52:38 2024] CPU: 60 PID: 70888 Comm: CPU 11/KVM Not tainted 5.3.18-150300.59.101-default #1 SLE15-SP3
...
[Thu May  9 14:52:38 2024] memory: usage 67579904kB, limit 67579904kB, failcnt 67391
[Thu May  9 14:52:38 2024] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[Thu May  9 14:52:38 2024] kmem: usage 633636kB, limit 9007199254740988kB, failcnt 0
...
[Thu May  9 14:52:38 2024] Tasks state (memory values in pages):
[Thu May  9 14:52:38 2024] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Thu May  9 14:52:38 2024] [  70675]     0 70675      243        1    28672        0          -998 pause
[Thu May  9 14:52:38 2024] [  70728]     0 70728   310400     5467   188416        0           830 virt-launcher-m
[Thu May  9 14:52:38 2024] [  70746]     0 70746  1242373    25104  1073152        0           830 virt-launcher
[Thu May  9 14:52:38 2024] [  70762]     0 70762   455279    14110   770048        0           830 libvirtd
[Thu May  9 14:52:38 2024] [  70763]     0 70763    37704     3916   339968        0           830 virtlogd
[Thu May  9 14:52:38 2024] [  70870]   107 70870 18302464 16718510 135278592        0           830 qemu-system-x86
[Thu May  9 14:52:38 2024] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,mems_allowed=0-3,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice/cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,task=qemu-system-x86,pid=70870,uid=107
[Thu May  9 14:52:38 2024] Memory cgroup out of memory: Killed process 70870 (qemu-system-x86) total-vm:73209856kB, anon-rss:66852088kB, file-rss:21948kB, shmem-rss:4kB
[Thu May  9 14:52:38 2024] oom_reaper: reaped process 70870 (qemu-system-x86), now anon-rss:0kB, file-rss:132kB, shmem-rss:4kB

Root Cause: Native Hypervisors vs. Harvester Architecture​

Traditional Linux Host (e.g., Virtual Machine Manager)​

On a standard Linux host, a VM managed via QEMU/KVM runs inside a systemd machine.slice. The hypervisor process (qemu-system-x86_64) has access to the host's wider pool of resources, managed loosely unless strict cgroup limits are manually added.

$ systemd-cgls
Control group /:
-.slice
├─1173 bpfilter_umh
├─system.slice

└─machine.slice
  └─machine-qemu\x2d1\x2dharv41.scope 
    └─8632 /usr/bin/qemu-system-x86_64 -name guest=harv41,debug-threads=on -S -…

Harvester (Kubernetes/KubeVirt Engine)​

In Harvester, every VM is encapsulated inside a Kubernetes Pod. This introduces a strict cgroup boundary (kubepods.slice).

As shown below, multiple helper processes must live alongside the primary qemu-system-x86_64 process within the same tightly limited container memory budget:

-.slice
└─kubepods.slice
  │ ├─kubepods-burstable-pod99ee3a64_645b_4699_9384_5a3875d78b41.slice
  │ │ ├─cri-containerd-0c316eb8a4711bff1ce968b46ddb658e49378897454b4caf2a20704c808f33f1.scope …
  │ │ │ └─ 8505 /pause
  │ │ ├─cri-containerd-eb5deef29f064adfd2456d9f9c535674ac4d0c95c81b13afbdab5a89dc6a774b.scope …
  │ │ │ └─ 8590 /usr/bin/virt-tail --logfile /var/run/kubevirt-private/2ce151aa…
  │ │ └─cri-containerd-fd57a5cfc2b9b1f53eaf7b575c3273e6784f4c56a04a17d502ecfbd19e55b066.scope …
  │ │   ├─ 8542 /usr/bin/virt-launcher-monitor --qemu-timeout 301s --name vm2 -…
  │ │   ├─ 8558 /usr/bin/virt-launcher --qemu-timeout 301s --name vm2 --uid 2ce…
  │ │   ├─ 8591 /usr/sbin/virtqemud -f /var/run/libvirt/virtqemud.conf
  │ │   ├─ 8592 /usr/sbin/virtlogd -f /etc/libvirt/virtlogd.conf
  │ │   └─ 8823 /usr/bin/qemu-system-x86_64 -name guest=default_vm2,debug-threa…

The "Hidden" Memory Overhead​

Breaking Down the Memory Overhead Buffer​

When you define a virtual machine, for example, a VM configured with 4 vCPUs, 2 GiB of memory, and 1 Ethernet interface, KubeVirt does not just allocate exactly 2 GiB of memory to the container.

Instead, KubeVirt calculates an additional baseline memory overhead required to operate the virtualization stack. This overhead budget covers:

• CPU Simulators: Thread pools tracking guest state and handling context switches.
• Memory Management: Tracking structures such as QEMU page tables mapping guest RAM.
• Auxiliary Devices: Buffers for virtual network interfaces (NICs), storage queues, and video devices.

The High Stakes of VM OOM Kills​

Depending on the guest OS type, specific kernel workloads, and heavy storage/network I/O spikes, the memory consumed by these helper tasks can quickly exceed KubeVirt's default calculations. Because Kubernetes enforces a strict hard ceiling on the Pod container, the entire container triggers a CONSTRAINT_MEMCG OOM kill the moment this boundary is breached.

Unlike traditional, stateless Kubernetes workloads where a container crash is quickly mitigated by a rapid pod restart, an OOM kill on a VM pod carries severe operational consequences:

• Prolonged Downtime: A virtual machine is a stateful workload. It does not instantaneously serve traffic upon a container restart; it must undergo a full operating system boot cycle, run init scripts, and re-initialize services, drastically extending your Recovery Time Objective (RTO).

• Risk of Data Corruption: Sudden terminations during flight can abruptly cut off active storage queues. If the guest OS or database engine is in the middle of a critical write operation when the host terminates the qemu process, it can result in uncommitted journals, filesystem degradation, or severe data corruption on your persistent volumes.

The Solution: Tunable Memory Architectures​

To address this, Harvester introduced dual-layer configurations that give administrators full flexibility over how overhead buffers are calculated.

Global Adjustment: additional-guest-memory-overhead-ratio​

This cluster-wide setting functions as a multiplier for KubeVirt's automatically calculated memory overhead. For deep structural details, refer to the Harvester Advanced Documentation.

• Definition: Scales the calculated overhead buffer to accommodate heavy I/O or virtualization tasks.
• Default Value: 1.5 (Provides a 50% safety cushion above baseline calculations).
• Valid Range: 0 or 1.0 to 10.0.

💡 Important Operational Notes:

• Lifecycle Impact: Changes to this setting only apply to newly created virtual machines or existing VMs after they undergo a migration or a full power cycle.
• System Overhead: A higher ratio increases the host container's memory allocation, guaranteeing safety for heavy workloads but scaling up the overall system resource reservation footprint.
• Resource Allocation Trade-off: Setting this ratio excessively high can lock up unneeded host memory blocks, leading to predictable underutilization and significant memory waste across your compute nodes.

Per-VM Optimization: Reserved Memory​

For specific virtual machines running intensive or non-standard workloads, a global multiplier might not offer the precision required. Harvester allows administrators to define a dedicated Reserved Memory value directly on individual VMs. For complete configuration steps, see the Harvester VM Management Documentation.

⚠️ Under the Hood Memory Carving: When you configure this setting, Harvester explicitly scales down the available memory presented to the Guest OS inside the VM. For example, if a VM is configured with 2 GiB of memory and you set a Reserved Memory value of 256 MiB, the Guest OS will only see and utilize 1.75 GiB (2 GiB - 256 MiB).

Why Use Per-VM Reserved Memory?​

• Guaranteed Overhead Headroom: By restricting the Guest OS from consuming the top slice of its configured allocation, you guarantee an isolated, un-evictable memory runway for host helper tasks.

• Targeted Safety for Heavy Workloads: This mechanism is highly practical for mission-critical, high-performance, or special-purpose workloads (such as nested virtualization layers or intensive database engines). It effectively prevents the VM from running into host-level cgroup OOM termination by proactively limiting its internal usage boundaries, removing the risk of unexpected node-level kills.

• Optimized Cluster Usability: Using per-VM reservations eliminates the major disadvantage of cranking up the global additional-guest-memory-overhead-ratio for the whole cluster. Instead of forcing a massive, wasteful memory overhead reservation across every idle or lightweight VM on your hosts, you can maintain a lean global default and surgically protect only the heavy workloads—striking an ideal balance between system density and ironclad stability.

Outcome: Guaranteed Workload Stability​

By leveraging this dual-layer tunable memory architecture, Harvester fundamentally alters how host-level overhead is calculated, moving from rigid, generalized defaults to a precise, tiered enforcement model:

Total Memory Overhead = Auto-calculated Overhead * Ratio + Reserved Memory

Best Practices & Configuration Matrix​

The following matrix showcases how combinations of Reserved Memory and the Overhead Ratio change the actual layout of the Guest OS space versus what Kubernetes reserves as a hard boundary.

When optimizing your Harvester cluster to eliminate host-level container OOM events, use the following operational checklist to tailor your memory strategies:

• For General Workloads:

  • Stick to the default ratio of 1.5, or configure a slightly higher value of 2.0. This ensures that standard Guest operating systems receive exactly the memory requested while scaling out a stable, predictable background overhead buffer across the cluster.

• For High I/O and Storage-Heavy VMs:

  • If you observe periodic KVM OOM events during massive backup windows, large-scale data syncs, or intensive disk read/write cycles, increase the individual VM's allocation or implement a targeted Reserved Memory configuration to safely expand the helper overhead pool.

• For GPU Passthrough Workloads:

  • Virtual machines utilizing direct hardware acceleration or GPU passthrough are prime candidates for explicit Reserved Memory carving. The underlying host-side device drivers and memory-mapped I/O (MMIO) windows for high-performance graphics hardware require a significantly higher, specialized memory footprint outside the guest OS space. Allocating dedicated per-VM reserved memory prevents driver-instigated cgroup allocation breaches, keeping both the hardware pipeline and the hypervisor completely stable.

Quick Summary​

• The Problem: In Harvester's Kubernetes-native architecture, every virtual machine is bound by a strict Pod container limit. While this rigid cgroup boundary is essential for security, ensuring a single rogue or leaking VM can never starve neighboring workloads or crash the bare-metal host, it means heavy storage/network I/O, device drivers, or GPU passthrough can cause internal helper processes to breach this hard ceiling, triggering a sudden host-level OOM kill.

• The Solution: Harvester eliminates these crashes without losing secure resource control using a dual-layer memory tuning strategy:

  • Globally: The additional-guest-memory-overhead-ratio scales out a safety cushion cluster-wide for newly created or migrated VMs.
  • Per-VM: The Reserved Memory setting surgically carves out a chunk of the VM's configured RAM exclusively for background helper tasks—preventing wasteful memory reservations across the cluster while safely anchoring high-performance, mission-critical workloads.

Appendix: Lab Simulation — Manually Triggering the Host-Level OOM​

For engineers looking to validate this behavior safely in a staging environment, you can replicate this multi-process cgroup breach. A detailed script and case study can be found in the Harvester Development Summary: OOM Investigation.

The simulation process highlights a fundamental truth about modern virtualization boundaries:

• The Guest OS is Trustworthy: Testing shows that modern guest operating systems handle internal resource limits reliably. If a runaway application inside the guest OS eats up all available RAM, the guest kernel safely steps in and kills that specific process internally. The VM itself survives, and from the host's perspective, the virtual machine continues running normally.

• The Host Cgroup Boundary is the Weak Link: The true host-level crash only happens if processes inside the host cgroup expand unexpectedly. If an infrastructure task or helper process inside the Pod container balloons, it consumes the memory buffer that KubeVirt set aside, causing the entire cgroup, the VM's carrier, to slam into the hard Kubernetes ceiling and trigger a host-level OOM kill.

This article provides information and mitigation steps for the following vulnerabilities in Harvester:

• CVE-2026-42945
• CVE-2026-42946
• CVE-2026-40701
• CVE-2026-42934

You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:

kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'

If the command returns one of the affected versions, perform the following mitigation steps.

The primary resolution is to upgrade Harvester to one of these versions:

• 1.7.2 or newer
• 1.8.1 or newer

If upgrade is not possible, apply the following mitigation to protect your clusters.

All ingress resources with the nginx.ingress.kubernetes.io/rewrite-target annotation containing ? in the annotation value are at risk.

By default, Harvester does not include any ingress resources with this annotation. Run the following command on your clusters to identify affected custom ingress resources:

kubectl get ingress -A -o json | jq '.items[] | select(.metadata.annotations["nginx.ingress.kubernetes.io/rewrite-target"] // "" | contains("?")) | {namespace: .metadata.namespace, name: .metadata.name, rewrite: .metadata.annotations["nginx.ingress.kubernetes.io/rewrite-target"]}'

Any ingress resources reported by the above command are vulnerable. They should be updated to either remove the vulnerable annotation or change the annotation value to not contain a question mark ?.

The following validating admission policy can be applied to your cluster to reject ingress resources with the vulnerable configuration:

cat<<EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: ingress-nginx-annotation-validation-20260514
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["networking.k8s.io"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["ingresses"]
  validations:
    - expression: |
        !has(object.metadata.annotations) ||
        !object.metadata.annotations.exists(k, k == 'nginx.ingress.kubernetes.io/rewrite-target') ||
        !object.metadata.annotations['nginx.ingress.kubernetes.io/rewrite-target'].contains('?')
      message: "Ingress resources with 'nginx.ingress.kubernetes.io/rewrite-target' annotation containing '?' in the annotation value are not allowed, due to the following CVEs: CVE-2026-42945, CVE-2026-42946, CVE-2026-40701, CVE-2026-42934"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: ingress-nginx-annotation-validation-20260514
spec:
  policyName: ingress-nginx-annotation-validation-20260514
  validationActions: [Deny]
EOF

The policy can be removed once you upgrade to Harvester 1.7.2, 1.8.1 or newer:

kubectl delete validatingadmissionpolicy ingress-nginx-annotation-validation-20260514

kubectl delete validatingadmissionpolicybinding ingress-nginx-annotation-validation-20260514

References​

• https://depthfirst.com/research/nginx-rift-achieving-nginx-rce-via-an-18-year-old-vulnerability
• https://www.suse.com/security/cve/CVE-2026-42945.html
• https://www.suse.com/security/cve/CVE-2026-42946.html
• https://www.suse.com/security/cve/CVE-2026-40701.html
• https://www.suse.com/security/cve/CVE-2026-42934.html