This article provides information and mitigation steps for the following vulnerabilities in Harvester:
CVE-2026-53359
important
On July 6, 2026, researcher Hyunwoo Kim (@v4bel) publicly disclosed Januscape, a vulnerability in the Linux kernel’s KVM/x86 memory-management code, which allows a malicious virtual machine to break out of the guest and run code as root on the host it runs on. On hosts where the KVM device node /dev/kvm is world-accessible, an unprivileged local user can exploit the vulnerability to crash the host.
All supported versions of Harvester are affected, including 1.6.1 and earlier, 1.7.2 and earlier, and 1.8.1 and earlier.
Januscape is the latest in a series of Linux kernel privilege-escalation vulnerabilities that required a patch and a reboot of the affected hosts.
SUSE is working on fixing this issue. Meanwhile, apply the mitigation steps described in this article to protect your clusters.
The mitigation steps involves disabling the nested virtualization feature of the KVM kernel module on your Harvester hosts.
note
Nested virtualization is not supported on virtual machines running on Harvester. Disabling this feature will not affect the functionality of your Harvester cluster.
On your Harvester hosts, use the following commands to confirm that the KVM kernel module is loaded with nested virtualization enabled:
Once the configuration is applied, reboot your Harvester hosts for the changes to take effect.
warning
Do not disable the KVM kernel module on your Harvester hosts, as it is required for running virtual machines. Only disable the nested virtualization feature using the configuration provided above.
Once you have upgraded to a fixed version of Harvester, you can re-enable the nested virtualization feature by deleting the CloudInit configuration and rebooting your Harvester hosts:
When running production workloads on virtualized infrastructure like Harvester, memory management is critical. In Harvester versions prior to v1.4.0, certain workloads experienced sudden Virtual Machine (VM) terminations due to the host Linux operating system triggering Out-Of-Memory (OOM) kills.
What is KubeVirt? Harvester uses KubeVirt as its core virtualization engine. KubeVirt is an open-source technology that allows Kubernetes to run and manage traditional Virtual Machines inside standard containers, translating VM specifications directly into Pod configurations.
This article explores why these OOM events occur in a Kubernetes-native virtualization environment and how Harvester provides granular tools to eliminate them.
When a VM is terminated due to insufficient memory at the host level, the Linux kernel logs specific keywords that help pinpoint the fault. In Harvester, these logs generally fall into two distinct categories depending on which process triggered the exhaustion.
Example 1: virt-launcher invoked oom-killer
The virt-launcher process runs inside the dedicated Kubernetes Pod backing the VM. If this component or its direct sub-processes run out of the memory allocated to their cgroup, the kernel triggers a memory cgroup (memcg) OOM event.
Feb 03 19:57:08 ** kernel: virt-launcher invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=986 Feb 03 19:57:08 ** kernel: CPU: 40 PID: 40785 Comm: virt-launcher Tainted: G I X 5.14.21-150400.24.60-default #1 SLE15-SP4 9096397fa6646928cc6d185ba417f2af65b536f1 ... Feb 03 19:57:08 ** kernel: memory: usage 17024340kB, limit 17024340kB, failcnt 1243 Feb 03 19:57:08 ** kernel: memory+swap: usage 17024340kB, limit 9007199254740988kB, failcnt 0 Feb 03 19:57:08 ** kernel: kmem: usage 143556kB, limit 9007199254740988kB, failcnt 0 Feb 03 19:57:08 ** kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice: ... Feb 03 19:57:08 ** kernel: Tasks state (memory values in pages): Feb 03 19:57:08 ** kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Feb 03 19:57:08 ** kernel: [ 38886] 0 38886 243 1 28672 0 -998 pause Feb 03 19:57:08 ** kernel: [ 38917] 0 38917 310400 6921 192512 0 986 virt-launcher-m Feb 03 19:57:08 ** kernel: [ 38934] 0 38934 1200940 25126 954368 0 986 virt-launcher Feb 03 19:57:08 ** kernel: [ 38951] 0 38951 386525 8247 466944 0 986 libvirtd Feb 03 19:57:08 ** kernel: [ 38952] 0 38952 33619 3940 290816 0 986 virtlogd Feb 03 19:57:08 ** kernel: [ 39079] 107 39079 4457263 4201766 34439168 0 986 qemu-system-x86 Feb 03 19:57:08 ** kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice/cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,task=qemu-system-x86,pid=39079,uid=107
Example 2: CPU X/KVM invoked oom-killer
This occurs when a vCPU execution thread inside qemu-system-x86_64 attempts a memory operation that pushes the entire container beyond its Kubernetes memory limit.
[Thu May 9 14:52:38 2024] CPU 11/KVM invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=830 [Thu May 9 14:52:38 2024] CPU: 60 PID: 70888 Comm: CPU 11/KVM Not tainted 5.3.18-150300.59.101-default #1 SLE15-SP3 ... [Thu May 9 14:52:38 2024] memory: usage 67579904kB, limit 67579904kB, failcnt 67391 [Thu May 9 14:52:38 2024] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0 [Thu May 9 14:52:38 2024] kmem: usage 633636kB, limit 9007199254740988kB, failcnt 0 ... [Thu May 9 14:52:38 2024] Tasks state (memory values in pages): [Thu May 9 14:52:38 2024] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [Thu May 9 14:52:38 2024] [ 70675] 0 70675 243 1 28672 0 -998 pause [Thu May 9 14:52:38 2024] [ 70728] 0 70728 310400 5467 188416 0 830 virt-launcher-m [Thu May 9 14:52:38 2024] [ 70746] 0 70746 1242373 25104 1073152 0 830 virt-launcher [Thu May 9 14:52:38 2024] [ 70762] 0 70762 455279 14110 770048 0 830 libvirtd [Thu May 9 14:52:38 2024] [ 70763] 0 70763 37704 3916 339968 0 830 virtlogd [Thu May 9 14:52:38 2024] [ 70870] 107 70870 18302464 16718510 135278592 0 830 qemu-system-x86 [Thu May 9 14:52:38 2024] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,mems_allowed=0-3,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice/cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,task=qemu-system-x86,pid=70870,uid=107 [Thu May 9 14:52:38 2024] Memory cgroup out of memory: Killed process 70870 (qemu-system-x86) total-vm:73209856kB, anon-rss:66852088kB, file-rss:21948kB, shmem-rss:4kB [Thu May 9 14:52:38 2024] oom_reaper: reaped process 70870 (qemu-system-x86), now anon-rss:0kB, file-rss:132kB, shmem-rss:4kB
Root Cause: Native Hypervisors vs. Harvester Architecture
Traditional Linux Host (e.g., Virtual Machine Manager)
On a standard Linux host, a VM managed via QEMU/KVM runs inside a systemd machine.slice. The hypervisor process (qemu-system-x86_64) has access to the host's wider pool of resources, managed loosely unless strict cgroup limits are manually added.
$ systemd-cgls Control group /: -.slice ├─1173 bpfilter_umh ├─system.slice └─machine.slice └─machine-qemu\x2d1\x2dharv41.scope └─8632 /usr/bin/qemu-system-x86_64 -name guest=harv41,debug-threads=on -S -…
In Harvester, every VM is encapsulated inside a Kubernetes Pod. This introduces a strict cgroup boundary (kubepods.slice).
As shown below, multiple helper processes must live alongside the primary qemu-system-x86_64 process within the same tightly limited container memory budget:
When you define a virtual machine, for example, a VM configured with 4 vCPUs, 2 GiB of memory, and 1 Ethernet interface, KubeVirt does not just allocate exactly 2 GiB of memory to the container.
Instead, KubeVirt calculates an additional baseline memory overhead required to operate the virtualization stack. This overhead budget covers:
CPU Simulators: Thread pools tracking guest state and handling context switches.
Memory Management: Tracking structures such as QEMU page tables mapping guest RAM.
Auxiliary Devices: Buffers for virtual network interfaces (NICs), storage queues, and video devices.
Depending on the guest OS type, specific kernel workloads, and heavy storage/network I/O spikes, the memory consumed by these helper tasks can quickly exceed KubeVirt's default calculations. Because Kubernetes enforces a strict hard ceiling on the Pod container, the entire container triggers a CONSTRAINT_MEMCG OOM kill the moment this boundary is breached.
Unlike traditional, stateless Kubernetes workloads where a container crash is quickly mitigated by a rapid pod restart, an OOM kill on a VM pod carries severe operational consequences:
Prolonged Downtime: A virtual machine is a stateful workload. It does not instantaneously serve traffic upon a container restart; it must undergo a full operating system boot cycle, run init scripts, and re-initialize services, drastically extending your Recovery Time Objective (RTO).
Risk of Data Corruption: Sudden terminations during flight can abruptly cut off active storage queues. If the guest OS or database engine is in the middle of a critical write operation when the host terminates the qemu process, it can result in uncommitted journals, filesystem degradation, or severe data corruption on your persistent volumes.
To address this, Harvester introduced dual-layer configurations that give administrators full flexibility over how overhead buffers are calculated.
Global Adjustment: additional-guest-memory-overhead-ratio
This cluster-wide setting functions as a multiplier for KubeVirt's automatically calculated memory overhead. For deep structural details, refer to the Harvester Advanced Documentation.
Definition: Scales the calculated overhead buffer to accommodate heavy I/O or virtualization tasks.
Default Value:1.5 (Provides a 50% safety cushion above baseline calculations).
Valid Range:0 or 1.0 to 10.0.
💡 Important Operational Notes:
Lifecycle Impact: Changes to this setting only apply to newly created virtual machines or existing VMs after they undergo a migration or a full power cycle.
System Overhead: A higher ratio increases the host container's memory allocation, guaranteeing safety for heavy workloads but scaling up the overall system resource reservation footprint.
Resource Allocation Trade-off: Setting this ratio excessively high can lock up unneeded host memory blocks, leading to predictable underutilization and significant memory waste across your compute nodes.
For specific virtual machines running intensive or non-standard workloads, a global multiplier might not offer the precision required. Harvester allows administrators to define a dedicated Reserved Memory value directly on individual VMs. For complete configuration steps, see the Harvester VM Management Documentation.
⚠️ Under the Hood Memory Carving:
When you configure this setting, Harvester explicitly scales down the available memory presented to the Guest OS inside the VM. For example, if a VM is configured with 2 GiB of memory and you set a Reserved Memory value of 256 MiB, the Guest OS will only see and utilize 1.75 GiB (2 GiB - 256 MiB).
Guaranteed Overhead Headroom: By restricting the Guest OS from consuming the top slice of its configured allocation, you guarantee an isolated, un-evictable memory runway for host helper tasks.
Targeted Safety for Heavy Workloads: This mechanism is highly practical for mission-critical, high-performance, or special-purpose workloads (such as nested virtualization layers or intensive database engines). It effectively prevents the VM from running into host-level cgroup OOM termination by proactively limiting its internal usage boundaries, removing the risk of unexpected node-level kills.
Optimized Cluster Usability: Using per-VM reservations eliminates the major disadvantage of cranking up the global additional-guest-memory-overhead-ratio for the whole cluster. Instead of forcing a massive, wasteful memory overhead reservation across every idle or lightweight VM on your hosts, you can maintain a lean global default and surgically protect only the heavy workloads—striking an ideal balance between system density and ironclad stability.
By leveraging this dual-layer tunable memory architecture, Harvester fundamentally alters how host-level overhead is calculated, moving from rigid, generalized defaults to a precise, tiered enforcement model:
Total Memory Overhead = Auto-calculated Overhead * Ratio + Reserved Memory
The following matrix showcases how combinations of Reserved Memory and the Overhead Ratio change the actual layout of the Guest OS space versus what Kubernetes reserves as a hard boundary.
VM Configured Memory
Reserved Memory
additional-guest-memory-overhead-ratio
Guest OS Memory
POD Container Memory Limit
Total Memory Overhead
2 Gi
not configured
"0.0"
2 Gi - 100 Mi
2 Gi + 240 Mi
~340 Mi
2 Gi
256 Mi
"0.0"
2 Gi - 256 Mi
2 Gi + 240 Mi
~500 Mi
2 Gi
not configured
"1.0"
2 Gi
2 Gi + 240*1.0 Mi
~240 Mi
2 Gi
not configured
"3.0"
2 Gi
2 Gi + 240*3.0 Mi
~720 Mi
2 Gi
not configured
"1.5"
2 Gi
2 Gi + 240*1.5 Mi
~360 Mi
2 Gi
256 Mi
"1.5"
2 Gi - 256 Mi
2 Gi + 240*1.5 Mi
~620 Mi
When optimizing your Harvester cluster to eliminate host-level container OOM events, use the following operational checklist to tailor your memory strategies:
For General Workloads:
Stick to the default ratio of 1.5, or configure a slightly higher value of 2.0. This ensures that standard Guest operating systems receive exactly the memory requested while scaling out a stable, predictable background overhead buffer across the cluster.
For High I/O and Storage-Heavy VMs:
If you observe periodic KVM OOM events during massive backup windows, large-scale data syncs, or intensive disk read/write cycles, increase the individual VM's allocation or implement a targeted Reserved Memory configuration to safely expand the helper overhead pool.
For GPU Passthrough Workloads:
Virtual machines utilizing direct hardware acceleration or GPU passthrough are prime candidates for explicit Reserved Memory carving. The underlying host-side device drivers and memory-mapped I/O (MMIO) windows for high-performance graphics hardware require a significantly higher, specialized memory footprint outside the guest OS space. Allocating dedicated per-VM reserved memory prevents driver-instigated cgroup allocation breaches, keeping both the hardware pipeline and the hypervisor completely stable.
The Problem: In Harvester's Kubernetes-native architecture, every virtual machine is bound by a strict Pod container limit. While this rigid cgroup boundary is essential for security, ensuring a single rogue or leaking VM can never starve neighboring workloads or crash the bare-metal host, it means heavy storage/network I/O, device drivers, or GPU passthrough can cause internal helper processes to breach this hard ceiling, triggering a sudden host-level OOM kill.
The Solution: Harvester eliminates these crashes without losing secure resource control using a dual-layer memory tuning strategy:
Globally: The additional-guest-memory-overhead-ratio scales out a safety cushion cluster-wide for newly created or migrated VMs.
Per-VM: The Reserved Memory setting surgically carves out a chunk of the VM's configured RAM exclusively for background helper tasks—preventing wasteful memory reservations across the cluster while safely anchoring high-performance, mission-critical workloads.
Appendix: Lab Simulation — Manually Triggering the Host-Level OOM
For engineers looking to validate this behavior safely in a staging environment, you can replicate this multi-process cgroup breach. A detailed script and case study can be found in the Harvester Development Summary: OOM Investigation.
The simulation process highlights a fundamental truth about modern virtualization boundaries:
The Guest OS is Trustworthy: Testing shows that modern guest operating systems handle internal resource limits reliably. If a runaway application inside the guest OS eats up all available RAM, the guest kernel safely steps in and kills that specific process internally. The VM itself survives, and from the host's perspective, the virtual machine continues running normally.
The Host Cgroup Boundary is the Weak Link: The true host-level crash only happens if processes inside the host cgroup expand unexpectedly. If an infrastructure task or helper process inside the Pod container balloons, it consumes the memory buffer that KubeVirt set aside, causing the entire cgroup, the VM's carrier, to slam into the hard Kubernetes ceiling and trigger a host-level OOM kill.
These vulnerabilities affect RKE2 ingress-nginx controller v1.14.5 and earlier. All Harvester versions that use this controller (including 1.5.2 and earlier, 1.6.1 and earlier, 1.7.1 and earlier, and 1.8.0) are therefore affected.
2026-05-15: Until Harvester 1.7.2 and 1.8.1 are released with the fixes, apply the mitigation steps below to secure your clusters.
You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:
kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'
If the command returns one of the affected versions, perform the following mitigation steps.
The primary resolution is to upgrade Harvester to one of these versions:
1.7.2 or newer
1.8.1 or newer
If upgrade is not possible, apply the following mitigation to protect your clusters.
All ingress resources with the nginx.ingress.kubernetes.io/rewrite-target annotation containing ? in the annotation value are at risk.
By default, Harvester does not include any ingress resources with this annotation. Run the following command on your clusters to identify affected custom ingress resources:
Any ingress resources reported by the above command are vulnerable. They should be updated to either remove the vulnerable annotation or change the annotation value to not contain a question mark ?.
The following validating admission policy can be applied to your cluster to reject ingress resources with the vulnerable configuration:
cat<<EOF | kubectl apply -f - apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: ingress-nginx-annotation-validation-20260514 spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["networking.k8s.io"] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["ingresses"] validations: - expression: | !has(object.metadata.annotations) || !object.metadata.annotations.exists(k, k == 'nginx.ingress.kubernetes.io/rewrite-target') || !object.metadata.annotations['nginx.ingress.kubernetes.io/rewrite-target'].contains('?') message: "Ingress resources with 'nginx.ingress.kubernetes.io/rewrite-target' annotation containing '?' in the annotation value are not allowed, due to the following CVEs: CVE-2026-42945, CVE-2026-42946, CVE-2026-40701, CVE-2026-42934" --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: ingress-nginx-annotation-validation-20260514 spec: policyName: ingress-nginx-annotation-validation-20260514 validationActions: [Deny] EOF
info
This policy is a cluster-scoped resource that requires the proper administrator RBAC permissions to create.
important
This validating policy prevents the inclusion of the vulnerable annotation configuration in new and existing ingress resources. However, it cannot detect or block any vulnerable ingress resources that already exist in the cluster. Therefore, it is important to follow the instructions described above to also identify and update any existing vulnerable ingress resources.
The policy can be removed once you upgrade to Harvester 1.7.2, 1.8.1 or newer:
These vulnerabilities affect specific versions of the RKE2 ingress-nginx controller (v1.13.7 and earlier, v1.14.3 and earlier). All Harvester versions that use this controller (including 1.5.2 and earlier, 1.6.1 and earlier, and 1.7.0) are therefore affected.
These CVEs are fixed in Harvester 1.7.1 and newer.
important
Harvester does not utilize the ingress-nginx controller custom error backend. Therefore, it is not affected by CVE-2026-24513.
important
Currently, no mitigation is available for CVE-2026-24514. An upgrade to Harvester 1.7.1 is required.
You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:
kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'
If the command returns one of the affected versions, perform one of the following mitigation steps.
The primary resolution is to upgrade to Harvester 1.7.1 or newer, which includes the fixed RKE2 ingress-nginx controller.
If upgrade is not possible, deploy the following validating admission policy to your cluster to reject ingress resources with the vulnerable configuration:
cat<<EOF | kubectl apply -f - apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: ingress-nginx-annotation-validation spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["networking.k8s.io"] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["ingresses"] validations: - expression: | !('nginx.ingress.kubernetes.io/auth-proxy-set-headers' in object.metadata.annotations) && !('nginx.ingress.kubernetes.io/auth-method' in object.metadata.annotations) && (object.spec.rules.all(rule, rule.http.paths.all(path, path.pathType != 'ImplementationSpecific'))) message: "Ingress resources with the vulnerable annotations are not allowed. Please remove the 'nginx.ingress.kubernetes.io/auth-proxy-set-headers' and 'nginx.ingress.kubernetes.io/auth-method' annotations, and avoid using the 'ImplementationSpecific' path type." --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: ingress-nginx-annotation-validation spec: policyName: ingress-nginx-annotation-validation validationActions: [Deny] EOF
info
This policy is a cluster-scoped resource that requires the proper administrator RBAC permissions to create.
This validating policy rejects any ingress resources that contain the:
Node selector constraints can prevent the scheduler from live-migrating a virtual machine to a target node. This often indicates a mismatch between the virtual machine's requirements and the node's capabilities.
A node selector may require a specific CPU feature, but the target node lacks the corresponding label (for example, cpu-feature.node.kubevirt.io/fpu: "true"). This mismatch can occur when the host-model CPU models and features computed by KubeVirt change over time.
You can resolve this issue using four different approaches.
Reboot the virtual machine.
KubeVirt automatically adds node selectors (during a previous migration or initial start) that can restrict scheduling. You can clear these node selectors by rebooting the virtual machine.
Reboot the virtual machine and set up a common CPU model.
You can override KubeVirt's default host-model CPU configuration by setting up a common CPU model for virtual machine migration. The model is applied to the virtual machine as its domain CPU, and to the pod as its node selector configuration.
This is the recommended approach for environments that can tolerate restarting of virtual machines.
Modify the node labels.
If rebooting the virtual machine is not an option, you can manually manipulate the target node's labels to satisfy the scheduling requirements.
Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.
This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.
The annotation itself does not affect the pod's node selector. It only controls the presence of specific CPU-related labels on the node, which the node selector checks against. For more information, see the References section.
Identify labels that are missing from the virtual machine's node selector and add them to the target node.
You can add the missing labels using the following command:
kubectl label node<node-name><key>=<value>
This circumvents the standard scheduling restrictions, allowing the virtual machine to migrate to the target node.
If a new node that lacks the required features is added to the cluster, you must repeat these steps to allow the virtual machine to live-migrate to that node.
Remove the node labels.
If you want to ensure that the virtual machine does not acquire specific node selector constraints after live migration, you can remove the relevant CPU labels from the target node.
Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.
This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.
This method works only if the virtual machine's pod does not have an existing node selector that contains the labels listed in the References section. Otherwise, you must reboot the virtual machine to clear the constraints.
Check if the pod has a node selector.
kubectl get pod <pod-name> -o yaml |grep nodeSelector -A 5 -B 5
If no node selector exists, remove the relevant CPU labels from the node.
Performing this action prevents the pod from acquiring new node selector constraints, thus enabling its future migration to other nodes. However, the successful outcome of that migration is not guaranteed.
When guest virtual machines running on Harvester nodes experience very slow network throughput, disabling Generic Receive Offload (GRO) and Generic Segmentation Offload (GSO) on the host interfaces may resolve the issue.
In the testing environment, guest virtual machines experienced severely degraded download and transfer speeds, dropping as low as 100 bps. This extreme slowdown was particularly evident when apt-get update, curl, and scp were used to transfer data between virtual machines running on different nodes. In contrast, performance remained normal when the virtual machines were hosted on the same node.
The issue was observed in a Harvester cluster hosted on Dell servers using Broadcom NetXtreme-E Series BCM57508 NICs (100 Gbps). mgmt, the built-in cluster network, was used for both management and virtual machine traffic.
Harvester relies on Linux’s bridge-based virtual networking to connect guest virtual machines to physical networks.
The NetXtreme-E BCM57508 NICs were connected to leaf switches configured to transmit jumbo frames. When the default MTU of 1500 is used, these frames should ideally be segmented to approximately 1450 bytes before reaching the Harvester host kernel. However, the packets actually arriving at the kernel were fragmented into unexpectedly small sizes. This forced the kernel to process a significantly higher volume of packets, leading to increased CPU overhead and reduced download throughput.
Packets captures collected using the following command confirmed the unexpectedly small size of the incoming packets.
tcpdump -xx -i <interface-name> <interface-name> is the name of the physical interface on the host connected to the VMs.
Generic Receive Offload (GRO) and Generic Segmentation Offload (GSO) are kernel-level software offloading mechanisms designed to optimize network performance. GRO aggregates multiple small incoming packets into larger ones before passing them to the network stack. GSO performs the opposite on transmission, splitting large packets into smaller frames before sending them to the NIC.
While these features are typically used to enhance performance, in this specific scenario, they interfered with the normal TCP segmentation process. This interference led to inefficient packet segmentation and the creation of an excessive number of small fragments, which ultimately degraded overall network performance.
When GRO and GSO were disabled, the Linux network stack automatically reverted to using standard transport-layer segmentation methods, specifically TCP Segmentation Offload (TSO) and Large Receive Offload (LRO). These mechanisms maintained efficient packet aggregation and segmentation at the appropriate layers, ensuring properly sized packets were presented to the kernel, which successfully restored expected network performance.
The NetXtreme-E BCM57508 NICs may experience suboptimal interaction with GRO and GSO due to a Broadcom driver bug. Enabling these offload mechanisms led to inefficient packetization, producing many small packets instead of fewer large ones, which ultimately reduced network throughput.
When working with Longhorn, you may encounter two different VolumeAttachment resources with similar names: Kubernetes VolumeAttachment (storage.k8s.io/v1) and Longhorn VolumeAttachment (longhorn.io/v1beta2). This often causes confusion about why both exist, when each is created, whether they always appear together, and which one to check when troubleshooting. This document clarifies their distinct roles, shows how they work together (and when they don't), and provides real-world examples to help you identify attachment sources and effectively troubleshoot volume attachment issues.
The observations and analysis in this document are based on Longhorn latest 1.10.x branch.
Workflow: How K8s and Longhorn VolumeAttachments Work Together
When a Pod requires a Longhorn volume, two separate VolumeAttachment resources work together to complete the attachment process. The Kubernetes VolumeAttachment represents the CSI standard attachment request, while the Longhorn VolumeAttachment manages the actual attachment orchestration with ticket-based coordination.
The following diagram illustrates the complete flow from Pod scheduling to successful volume attachment:
Notice the csi-attacher ticket type - this confirms the attachment was triggered by Kubernetes VolumeAttachment through the CSI flow, not by Longhorn internal operations.
When multiple operations require volume attachment simultaneously, Longhorn uses a ticket-based priority system to coordinate access intelligently. This ensures critical operations take precedence while allowing background tasks to coexist when possible.
Each ticket type has an assigned priority level that determines selection order when the volume is detached:
Priority 2000 (Highest):
VolumeRestoreController
VolumeExpansionController
Priority 1000:
LonghornAPI
Priority 900:
CSIAttacher
ShareManagerController
SalvageController
Priority 800 (Lowest):
BackupController
SnapshotController
VolumeCloneController
VolumeEvictionController
When the volume is detached, the ticket with the highest priority is selected for attachment. If multiple tickets share the same priority, the first one (sorted by ID) is chosen.
note
For ReadWriteMany (RWX) Filesystem mode volumes, CSIAttacher tickets are ignored during ticket selection and detachment decisions. Only the ShareManagerController ticket is considered, as it manages the centralized sharing mechanism for RWX access. Individual CSI attacher tickets from Pods are summarized and handled by the Share Manager, not directly by the VolumeAttachment Controller.
Priority levels alone don't tell the complete story. Longhorn also implements an interruption mechanism to handle cases where request arrives while the volume is already attached to a different node.
Interruptible operations (can be interrupted):
BackupController
SnapshotController
VolumeCloneController - clone operations, but only when the volume is in VolumeCloneStateCopyCompletedAwaitingHealthy state
note
The VolumeCloneController is only interruptible in a specific state. During the data copy phase, clone operations cannot be interrupted. Interruption is only allowed after the copy completes and the volume is waiting to become healthy, preventing data corruption during active copy operations.
Workload operations (can trigger interruption):
CSIAttacher - Pod workloads requiring the volume on a different node
LonghornAPI - manual attachment requests via UI/API
The volume's currently attached node has only interruptible tickets
A different node has a workload ticket requesting the volume
note
Interruption is based on ticket type classification, not priority numbers. Priority numbers only affect the selection order during the attachment phase when the volume is detached.
This design ensures background operations never block workload rescheduling, while protecting active workloads from being interrupted by other background tasks.
The following examples demonstrate how VolumeAttachment resources behave in common scenarios. Each example shows the complete YAML resource state at different stages, helping you understand what to look for when troubleshooting or monitoring Longhorn operations.
Example 1: VolumeSnapshot Creation (Longhorn VolumeAttachment Only)
VolumeSnapshot operations use only Longhorn VolumeAttachment without involving Kubernetes VolumeAttachment. This demonstrates that Longhorn VolumeAttachment can operate independently for internal operations.
During VM migration, Harvester has two virt-launcher pods for the same VM: the original pod on the source node and a new pod on the target node. This multi-attach capability is enabled for RWX (ReadWriteMany) block mode volumes when the StorageClass has migratable: true parameter, which allows Longhorn to support live VM migration. In the following example, we migrate a VM from harvester-node-2 to harvester-node-0.
Two CSI attachment tickets coexist: One pointing to the source node (harvester-node-2) and another to the target node (harvester-node-0)
Both tickets are csi-attacher type: Indicating they were both triggered by Kubernetes VolumeAttachment through the CSI flow
Both tickets have satisfied: true status: This demonstrates Longhorn's support for attaching the same volume to multiple nodes simultaneously (RWX-like behavior for migration)
Target node ticket has special message: "The migrating attachment ticket is satisfied" explicitly identifies this as a migration scenario
Multi-attach is temporary: This dual-attachment state only exists during VM migration; the source node's ticket will be removed after migration completes
Longhorn uses two different VolumeAttachment resources for different purposes:
Kubernetes VolumeAttachment (storage.k8s.io/v1) follows the standard CSI specification and is created only when Pods are scheduled to nodes. It represents Kubernetes' official attachment intent and is managed by K8s Attach/Detach Controller and CSI External-Attacher.
Longhorn VolumeAttachment (longhorn.io/v1beta2) extends beyond CSI to support Longhorn's advanced features. It's created for multiple scenarios, including Pod workloads, snapshots, backups, clones, and manual operations. It uses a ticket-based system to coordinate concurrent attachment requests and is managed collaboratively by multiple Longhorn controllers.
Why both are needed: K8s VolumeAttachment ensures CSI compliance with the Kubernetes ecosystem, while Longhorn VolumeAttachment enables automation for background operations without manual intervention. Importantly, not all Longhorn operations trigger K8s VolumeAttachment—for example, creating a VolumeSnapshot only creates a Longhorn VolumeAttachment ticket (snapshot-controller), not a K8s VolumeAttachment.
When troubleshooting: Check both resources. K8s VolumeAttachment shows the CSI standard workflow status, while Longhorn VolumeAttachment shows the complete picture, including all internal operations via attachment tickets. Look at the ticket type to identify the operation source: csi-attacher means triggered by the K8s VolumeAttachment (Pod workload), while snapshot-controller, backup-controller, etc. indicate Longhorn internal operations.
However, in certain scenarios, users may require the VM filesystem to be quiesced during Velero backup creation to prevent data corruption, especially when the VM is experiencing heavy I/O operations.
This article describes how to customize Velero Backup Hooks to implement filesystem freeze during Velero backup processing, ensuring data consistency in the backup content.
KubeVirt's virt-freezer provides a mechanism to freeze and thaw guest filesystems. This capability can be leveraged to ensure filesystem consistency during VM backups. However, certain prerequisites must be met for filesystem freeze/thaw operations to function properly:
Based on Harvester project experience, some guest operating systems require additional configuration:
Linux distributions (e.g., RHEL, SLE Micro): May lack sufficient permissions for filesystem freeze operations by default, requiring custom policies
Windows guests: Require the VSS service to be enabled for filesystem freeze functionality
Important: Filesystem freeze/thaw functionality depends on guest VM configuration, which is outside Harvester's control. Users are responsible for ensuring compatibility before implementing Velero backup hooks with filesystem freeze.
If the guest VM is configured correctly, the Velero backup will complete successfully with HooksAttempted indicating successful hook execution.
Check the backup status using:
velero backup describe [Backup Name] --details
Example output showing successful hook execution:
Name: demo Namespace: velero Labels: velero.io/storage-location=default Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.33.3+rke2r1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=33 Phase: Completed Namespaces: Included: demo Excluded: <none> Resources: Included: * Excluded: <none> Cluster-scoped: auto Label selector: <none> Or label selector: <none> Storage Location: default Velero-Native Snapshot PVs: auto Snapshot Move Data: true Data Mover: velero .... Backup Volumes: Velero-Native Snapshots: <none included> CSI Snapshots: demo/vm-nfs-disk-0-au2ej: Data Movement: Operation ID: du-be5417aa-498e-4b93-b59f-e6498f95a6df.d7f97dab-3bb1-41e189381 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 5368709120 Result: succeeded Pod Volume Backups: <none included> HooksAttempted: 2 HooksFailed: 0
The output shows that Velero pre/post backup hooks completed successfully. In this case, the hooks are connected to guest VM filesystem freeze and thaw operations to ensure data consistency.
Implementing filesystem freeze hooks with Velero ensures data consistency during VM backups by quiescing the filesystem before snapshot creation. This approach is particularly valuable for VMs with high I/O activity or critical data that requires point-in-time consistency guarantees.
For Harvester to successfully migrate a virtual machine from one node to another, the source and target nodes must have compatible CPU models and features.
Harvester uses KubeVirt / QEMU / libvirt to manage and run virtual machines. When a VM starts, libvirt exposes a specific CPU feature set to the guest operating system. Live migration requires this CPU feature set to be identical on both source and target nodes.
Different CPU generations support different instruction sets and feature flags. If those differ, the live migration will be blocked to avoid instability or incorrect execution on the target node.
If the CPU model of a virtual machine isn't specified, KubeVirt assigns it the default host-model configuration so that the virtual machine has the CPU model closest to the one used on the host node.
KubeVirt automatically adjusts the node selectors of the associated virt-launcher Pod based on this configuration. If the CPU models and features of the source and target nodes do not match, the live migration may fail.
Let's examine an example.
When a virtual machine is first migrated to another node with the SierraForest CPU model, the following key-value pairs are added to the spec.nodeSelector field in the Pod spec.
The above nodeSelector configuration is retained for subsequent migrations, which may fail if the new target node doesn't have the corresponding features or model.
For example, compare the CPU model and feature labels added by KubeVirt to the following two nodes:
# Node A labels: cpu-model-migration.node.kubevirt.io/SierraForest:"true" cpu-feature.node.kubevirt.io/fpu:"true" cpu-feature.node.kubevirt.io/vme:"true" # Node B labels: cpu-model-migration.node.kubevirt.io/SierraForest:"true" cpu-feature.node.kubevirt.io/vme:"true"
This virtual machine will fail to migrate to Node B due to the missing fpu feature. However, if the virtual machine doesn't actually require this feature, this can be frustrating. Therefore, setting up a common CPU model can resolve this issue.
You can define a custom CPU model to ensure that the spec.nodeSelector configuration in the Pod spec is assigned a CPU model that is compatible and common to all nodes in the cluster.
Consider this example.
We have the following node information:
# Node A labels: cpu-model.node.kubevirt.io/IvyBridge:"true" cpu-feature.node.kubevirt.io/fpu:"true" cpu-feature.node.kubevirt.io/vme:"true" # Node B labels: cpu-model.node.kubevirt.io/IvyBridge:"true" cpu-feature.node.kubevirt.io/vme:"true"
If we set up IvyBridge as our CPU model in the virtual machine spec, KubeVirt only adds cpu-model.node.kubevirt.io/IvyBridge under spec.nodeSelector in the Pod spec.
Harvester allows the CPU model to be defined in the VM specification.
If host-passthrough is used, the VM exposes the exact host CPU to the guest. This provides maximum performance but completely prevents migration across different CPU generations.
If host-model is used, the VM derives a CPU model based on the host’s capabilities. This works only when all nodes expose identical CPU features. In mixed clusters, this setting commonly causes migration failures.
If an explicit CPU model is defined, the VM exposes a named CPU architecture defined by QEMU. This is the recommended approach for clusters that may contain different CPU generations.
For mixed environments, an explicit CPU model should always be used.
When selecting a CPU model, the goal is to choose the highest common denominator. This means the most modern CPU architecture that every node in the cluster supports. Pick a model supported by all nodes in your cluster, including the oldest CPU generation. Use a server-grade model that provides a reasonable set of capabilities (e.g., vector instructions, security features) without tying VMs to host-specific features. Use the same model on all VMs that need live migration.
Below are practical examples for common enterprise hardware combinations.
For a cluster that mixes Cascade Lake and Sapphire Rapids, the recommended CPU model is Cascadelake-Server. Sapphire Rapids processors are backward compatible with Cascade Lake instructions, making Cascadelake-Server the highest common denominator between these two generations.
For a cluster that mixes Skylake and Cascade Lake nodes, the appropriate model is Skylake-Server. This is the common ground between both generations. Cascade Lake includes additional AVX-512 optimizations, but using Skylake ensures compatibility across the entire cluster.
If your cluster contains Broadwell nodes along with anything newer, the safest baseline is Broadwell. Broadwell serves as a stable and widely supported baseline for older enterprise hardware. Any newer CPUs will support it.
If your virtual machines run only on a specific CPU model, you can set up a cluster-wide CPU model in the kubevirt resource.
You can edit it with kubectl edit kubevirt kubevirt -n harvester-system, then add the CPU model you want in the following spec:
spec: configuration: cpuModel: IvyBridge
Then, when a new virtual machine starts or an existing virtual machine restarts, the cluster-wide setting will be applied. The system follows these priorities when using CPU models if you configure them in both locations:
We have the default SSH user rancher, but other users may be required. Creating users with useradd will result in their deletion upon restarting the Harvester node; therefore, follow the steps below to create persistent SSH users.
