Harvester v1.7.0 introduced the kubevirt-migration setting, which exposes the cluster-wide live migration configuration that previously could only be changed by editing the kubevirt object directly.
If you customized spec.configuration.migrations on the kubevirt object before upgrading, that customization is at risk. This article explains why and provides a script that converts it into the setting so it is preserved.
You are affected if both of the following are true:
You are running a version earlier than v1.7.0 and you changed spec.configuration.migrations on the kubevirt object, for example to enable allowAutoConverge.
You plan to upgrade to v1.7.0 or later.
You are also affected if you have already upgraded to v1.7.x and have not yet changed the kubevirt-migration setting. In that case the configuration is still intact and the same script applies.
Check whether the kubevirt object carries a configuration:
kubectl get kubevirt kubevirt -n harvester-system -o jsonpath='{.spec.configuration.migrations}'| jq
If the command returns nothing, your cluster uses the KubeVirt defaults and no action is required.
The upgrade creates the kubevirt-migration setting without a value, so the Harvester UI displays the default values. The kubevirt object keeps your configuration, so live migration continues to behave as before.
The two are now inconsistent, and the inconsistency is not visible unless you compare them. Harvester writes the whole value of the setting to the kubevirt object, so the first time the setting is saved, all fields are written at once, including the fields that you did not change. Your configuration is replaced by the defaults, without warning.
In practice this means that changing an unrelated field such as parallelMigrationsPerCluster in the UI silently turns allowAutoConverge back off. Migrations of write-heavy virtual machines then start failing to converge, which is often noticed only during the node drains of a later upgrade.
Run the following script from a machine with kubectl access to the cluster. jq is required.
Run it before you upgrade to v1.7.0 or later. If you already upgraded to v1.7.x and have not yet changed the setting, run it now.
#!/bin/bash set -euo pipefail SETTING_NAME="kubevirt-migration" migrations=$(kubectl get kubevirt kubevirt -n harvester-system -o json | jq -c '.spec.configuration.migrations // empty') if[ -z "$migrations"];then echo"The kubevirt object has no migration configuration. Nothing to preserve." exit0 fi # nodeDrainTaintKey and network cannot be configured through the setting: # nodeDrainTaintKey is used by the upgrade process, and network is owned by # the vm-migration-network setting. The webhook rejects a value containing them. value=$(echo"$migrations"| jq -c 'del(.nodeDrainTaintKey, .network)') if["$value"="{}"];then echo"The migration configuration only contains fields that the setting does not manage. Nothing to preserve." exit0 fi ifsetting_json=$(kubectl get settings.harvesterhci.io "$SETTING_NAME" -o json 2>/dev/null);then current_value=$(echo"$setting_json"| jq -r '.value // empty') if[ -n "$current_value"];then echo"The $SETTING_NAME setting already has a value. Nothing to do." exit0 fi echo"Setting $SETTING_NAME to: $value" kubectl patch settings.harvesterhci.io "$SETTING_NAME" --type merge \ -p "$(jq -n --arg v"$value"'{"value": $v}')" else echo"Creating $SETTING_NAME with: $value" kubectl apply -f - <<EOF apiVersion: harvesterhci.io/v1beta1 kind: Setting metadata: name: $SETTING_NAME value: '$value' EOF fi echo"Result:" kubectl get settings.harvesterhci.io "$SETTING_NAME" -o yaml
The script does not write to the kubevirt object, so it cannot change how the cluster performs live migrations. It is idempotent: it does nothing if there is no configuration to preserve, or if the setting already has a value.
note
The Harvester webhook rejects changes to the kubevirt-migration setting while a virtual machine migration is in progress. If the script fails for that reason, wait until all migrations are completed and run it again.
The kubevirt-migration setting is the source of truth. Harvester writes it to the kubevirt object whenever the setting changes.
Do not edit spec.configuration.migrations on the kubevirt object directly. Any change is overwritten the next time the setting is reconciled. Use the setting instead, either in the Harvester UI under Advanced > Settings, or with kubectl.
For the list of supported fields and their defaults, see kubevirt-migration in the Harvester documentation.
A symlink-following vulnerability was found in KubeVirt's virt-handler migration proxy.
During live migration, virt-handler dials Unix sockets inside the target virt-launcher pod using /proc/<pid>/root/<path> via net.Dial(), without symlink protection. These socket paths reside in QEMU-owned directories that are writable by the virt-launcher user.
An attacker with namespace edit and pods/exec permissions can replace a migration proxy socket with a symlink pointing to the host CRI-O socket. Because virt-handler runs as root in the host mount namespace, absolute symlink targets resolve against the host filesystem. The bidirectional io.Copy proxy then relays attacker-controlled bytes to the container runtime, enabling full node compromise.
This policy blocks exec and attach requests to virt-launcher pods from non-admin and non-Harvester service accounts, to prevent an attacker from replacing the migration proxy sockets with symlinks.
note
Once you have upgraded to a fixed version of Harvester, remove this policy and binding:
Enabling the Longhorn V2 Data Engine can cause kernel workqueue lockups on some Harvester nodes, leading to system instability. In severe cases, rke2-server terminates, Longhorn stops processing operations, and affected nodes require recovery.
This issue typically occurs when SPDK is pinned to dedicated CPUs, but the Linux kernel continues running IRQ handlers or unbound workqueue workers on those same CPUs.
caution
The mitigation measure described in this article modifies node-wide host CPU affinity for IRQs and kernel workqueues, which can affect all workloads running on the host. Apply this workaround only to nodes running the Longhorn V2 Data Engine, and carefully validate your target CPU mask before making host-level changes.
This issue can affect Harvester clusters that meet all of the following conditions:
The Longhorn V2 Data Engine is enabled.
The Longhorn V2 instance-manager pod runs SPDK (spdk_tgt), which busy-polls the CPUs specified in the Longhorn V2 Data Engine CPU mask.
Host IRQ handlers or unbound kernel workqueues are permitted to run on those same SPDK-designated CPUs.
While the default Longhorn V2 CPU mask is 0x3 (maps to CPUs 0 and 1), this issue can occur with any CPU mask if kernel IRQs or unbound workqueues share the CPUs used by spdk_tgt.
This issue primarily affects bare-metal Harvester nodes running the Longhorn V2 Data Engine. Reproducibility varies depending on hardware and workload patterns.
The Longhorn V2 Data Engine relies on SPDK, whose reactor threads busy-poll on the CPUs designated by the engine's CPU mask. This continuous polling is expected behavior for high-performance storage processing.
The issue occurs when normal host kernel work continues running on those same CPUs, including the following:
Network or storage IRQs target SPDK CPUs.
Unbound kernel workqueues retain SPDK CPUs in /sys/devices/virtual/workqueue/cpumask.
Per-workqueue CPU masks allow existing workers to run on SPDK CPUs.
When SPDK consumes 100% of these CPU cores, pending kernel tasks are starved, eventually triggering the kernel workqueue lockup detector. While isolated lockup warnings may not always result in an unrecoverable node, continuous task starvation cascades into network, RKE2, kubelet, and Longhorn control-plane failure.
To prevent this, all IRQs and kernel workqueues must be explicitly re-routed away from the CPUs defined in the Longhorn V2 CPU mask (for example, CPUs 0 and 1 when using the default mask 0x3).
This command matches the full command line because the SPDK process or its threads may appear as reactor_<cpu> in the kernel process name. In that case, pgrep -x spdk_tgt can return empty even when spdk_tgt is running.
Example output for the default CPU mask 0x3:
PID TID PSR STAT %CPU COMMAND COMMAND 2913161 2913161 0 RLl 99.9 reactor_0 spdk_tgt -L all --mem-size 2048 -m 0x3 2913161 2913166 21 SLl 0.0 dpdk-intr spdk_tgt -L all --mem-size 2048 -m 0x3 2913161 2913220 1 RLl 99.9 reactor_1 spdk_tgt -L all --mem-size 2048 -m 0x3
The PSR column shows the CPU that each thread is running on. In the example above, reactor_0 is running on CPU 0 and reactor_1 is running on CPU 1, so the SPDK CPU list is 0,1.
To check whether running or blocked kernel workers are currently on the SPDK CPUs, use the following more targeted command. Replace SPDK_CPUS with the comma-separated CPU IDs from the reactor_* rows.
On nodes that are not affected, this command normally prints only the header line. If the header line is removed from the command, empty output is expected. This means there are no running or uninterruptible kworker/* or ksoftirqd/* threads on the SPDK CPUs at that moment.
Example output showing kernel work on an SPDK CPU:
PID TID PSR STAT %CPU WCHAN COMMAND ARGS 17 17 0 R 0.0 - ksoftirqd/0 [ksoftirqd/0] 32008 32008 0 R 0.0 - kworker/0:0+eve [kworker/0:0+events] 2935011 2935011 0 D 0.0 mgag200_ddc_algo_bit_data_pre_ kworker/0:2+eve [kworker/0:2+events] 4012518 4012518 0 R 0.0 - kworker/0:3+mm_ [kworker/0:3+mm_percpu_wq]
In this example, the PSR value is 0, and the process names are ksoftirqd/0 and kworker/0:*. This means kernel softirq and workqueue threads are running or blocked on CPU 0.
If the SPDK reactor threads are running on the CPUs reported in the workqueue lockup, or if the targeted command shows kworker/* or ksoftirqd/* activity on the SPDK CPUs, continue with the IRQ and workqueue checks.
The following example checks CPUs 0 and 1 because the reactor_* rows above are running on those CPUs. Replace SPDK_CPUS with the CPU IDs used in your environment.
SPDK_CPUS="0,1" cpu_list_overlaps(){ python3 - "$1""$2"<<'PY' import sys target = {int(cpu) for cpu in sys.argv[1].split(",") if cpu} seen = set() for part in sys.argv[2].split(","): part = part.strip() if not part: continue if "-" in part: start, end = map(int, part.split("-", 1)) seen.update(range(start, end + 1)) else: seen.add(int(part)) sys.exit(0 if target & seen else 1) PY } forirqdirin /proc/irq/[0-9]*;do irq=${irqdir##*/} eff=$(cat"${irqdir}/effective_affinity_list"2>/dev/null ||true) conf=$(cat"${irqdir}/smp_affinity_list"2>/dev/null ||true) if cpu_list_overlaps "${SPDK_CPUS}""${eff}";then echo"IRQ=${irq} configured=${conf} effective=${eff}" grep -w "^ *${irq}:" /proc/interrupts 2>/dev/null ||true fi done
The effective value shows where the IRQ is actually running. In this example, NIC queues and NVMe queues are effectively landing on CPUs 0 and 1, which are the default SPDK reactor CPUs for mask 0x3. If device IRQs, especially high-traffic NIC or storage IRQs, are effectively landing on SPDK CPUs, the node is at risk.
On a 24-CPU node, ffffff means the workqueue can run on CPUs 0-23. If SPDK uses the default CPU mask 0x3, CPUs 0 and 1 are included in this workqueue mask. If the global or per-workqueue masks include the SPDK CPUs, unbound kernel work may still run on the SPDK CPUs.
Longhorn added the data-engine-cpu-isolation-enabled setting to reduce the chance of this issue. When enabled for the V2 Data Engine, the Longhorn V2 instance-manager:
Persists the SPDK CPU mask under /var/lib/longhorn/instance-manager/v2/spdk_cpu_mask on the host.
Programs /proc/irq/*/smp_affinity to the inverse of the SPDK CPU mask.
Writes the same inverse mask to /sys/devices/virtual/workqueue/cpumask.
Updates per-workqueue CPU masks when possible.
Reconciles stale affinity state on the next instance-manager restart if the setting is later disabled.
This setting steers IRQs and unbound workqueues away from SPDK CPUs. In some environments, this may stop the workqueue lockup messages entirely. In others, it may only reduce how often they occur because CPU-bound or per-CPU kernel workers such as kworker/0:* are tied to a specific CPU and cannot be moved by the unbound workqueue CPU mask.
Use this setting when the Longhorn version bundled with Harvester includes it. If the Longhorn setting exists but Harvester does not expose it in the UI, you can still configure it through the Longhorn setting resource. For Harvester versions earlier than v1.9.0, the bundled Longhorn version does not include the setting, so use the manual workaround in the next section.
caution
The Longhorn setting is a danger-zone setting. It changes host-wide IRQ and workqueue affinity, takes effect only after the V2 instance-manager pod is recreated, and Longhorn refuses to apply the change while V2 volumes are attached. Stop workloads that use Longhorn V2 volumes and detach those volumes before changing the setting.
If the setting exists in your Longhorn version, you can check it with:
kubectl -n longhorn-system get settings.longhorn.io data-engine-cpu-isolation-enabled
Setting IRQ affinity to exclude SPDK CPUs Applied IRQ affinity mask Setting workqueue cpumask to exclude SPDK CPUs Applied global workqueue cpumask Applied per-workqueue cpumask
Manual Workaround for Versions Without the Longhorn Setting
Use this workaround for Harvester versions earlier than v1.9.0, where the bundled Longhorn version does not include data-engine-cpu-isolation-enabled. For later versions, first check whether the Longhorn setting exists and prefer the setting-based risk-reduction path when possible.
The goal is to move IRQs and unbound workqueues away from the SPDK CPUs. For example, if SPDK uses the default CPUs 0 and 1, IRQs and unbound workqueues should use CPU 2 through the last online CPU.
This workaround is a risk-reduction step, not a guaranteed fix for every workqueue lockup. CPU-bound or per-CPU kernel workers can still run on the SPDK CPUs because they are tied to those CPUs by the kernel. The expected result is that the node remains stable and any remaining workqueue stalls recover quickly instead of hanging for a long time.
Repeat this workaround whenever the Longhorn V2 CPU mask changes. If CPU allocation is managed dynamically, the IRQ and workqueue masks must be recalculated after each placement change.
The default Longhorn V2 CPU mask is 0x3, so SPDK uses CPUs 0 and 1.
On the example node, all online CPUs are 0-23:
cat /sys/devices/system/cpu/online
0-23
Use the following values for this default example:
Item
Value
SPDK CPUs
0,1
Non-SPDK CPU list
2-23
Linux affinity mask
fffffc
The Linux affinity mask is a CPU bitmap written in hexadecimal. CPU 0 is bit 0, CPU 1 is bit 1, and so on. On a 24-CPU node, all CPUs enabled is ffffff. Excluding CPUs 0 and 1 clears the lowest two bits, so the mask becomes fffffc. Leading zeros are optional, so fffffc and 00fffffc are equivalent.
Do not copy these values blindly if your Longhorn V2 CPU mask or online CPU list is different. The non-SPDK CPU list must be all online CPUs except the CPUs used by the SPDK reactor threads.
Some IRQs may reject affinity updates because they are managed by the kernel. This is expected. Always verify effective_affinity_list after applying the change.
Changing IRQ affinity alone may not make existing workqueue lockup messages disappear. Kernel workqueues can still run on the SPDK CPUs until the workqueue CPU masks are updated, and workers that are already stuck may continue to be reported by the kernel. Apply the workqueue affinity change as well, and then verify whether new lockup messages stop appearing.
These runtime IRQ affinity changes are not persistent across reboot. Use the persistence step later in this section if the runtime change mitigates the issue.
These runtime workqueue affinity changes are not persistent across reboot. Use the persistence step later in this section if the runtime change mitigates the issue.
After changing both IRQ and workqueue affinity, restart the Longhorn V2 instance-manager pod on the affected node. This is required because the existing spdk_tgt process can keep the SPDK reactor threads on the busy CPUs, and already-stuck CPU-bound workers may continue to be reported by the kernel until spdk_tgt is recreated.
caution
Detach all Longhorn V2 volumes attached to the affected node before restarting the V2 instance-manager pod. You do not need to detach V2 volumes attached to other nodes. Restarting an instance-manager while V2 volumes are still attached to the affected node can interrupt storage I/O and affect running VMs.
Check the Longhorn V2 volumes first:
kubectl -n longhorn-system get volumes.longhorn.io \ -o custom-columns=NAME:.metadata.name,DATAENGINE:.spec.dataEngine,STATE:.status.state,NODE:.status.currentNodeID
Only continue after all v2 volumes whose NODE is the affected node are detached.
Find the V2 instance-manager pod on the affected node:
The previous IRQ and workqueue affinity commands only change runtime procfs and sysfs state, so they are lost after reboot. If those runtime changes reduce or stop new lockup messages, use a Harvester CloudInit resource to reapply both IRQ and workqueue affinity after reboot. This follows the same model as the Longhorn CPU isolation setting, which updates host IRQ and workqueue affinity at runtime.
The following manifest is a Harvester CloudInit resource, not a file that you manually place on each node. Save it on a machine with kubectl access to the Harvester cluster and apply it with kubectl apply -f <file-name>.yaml. The Harvester controller writes the content to /oem/99_longhorn_v2_cpu_affinity.yaml on each node matched by matchSelector.
You can add this resource after the cluster is already installed. The file is synchronized to the matched nodes after the resource is applied, but the cloud-init commands take effect only after those nodes are rebooted. After reboot, the commands are applied again on each boot.
Example CloudInit resource for the default SPDK CPUs 0 and 1 on a 24-CPU node:
apiVersion: node.harvesterhci.io/v1beta1 kind: CloudInit metadata: name: longhorn-v2-cpu-affinity spec: matchSelector: kubernetes.io/hostname:"hp-114-tink-system" filename: 99_longhorn_v2_cpu_affinity.yaml contents:| stages: network: - commands: - echo fffffc > /proc/irq/default_smp_affinity - for f in /proc/irq/[0-9]*/smp_affinity; do echo fffffc > "${f}" 2>/dev/null || true; done - echo fffffc > /sys/devices/virtual/workqueue/cpumask - for f in /sys/devices/virtual/workqueue/*/cpumask; do echo fffffc > "${f}" 2>/dev/null || true; done
Adjust the affinity mask and matchSelector for your environment. To target a different affected node, replace hp-114-tink-system with that node's kubernetes.io/hostname label value. Do not use matchSelector: {} unless every node should receive this workaround. This workaround is not dynamically reconciled. If the Longhorn V2 CPU mask changes, update the CloudInit resource and reboot the affected nodes.
After configuring the CloudInit resource, reboot the affected nodes for the commands to take effect. To apply the workaround immediately before reboot, run the runtime commands in the previous steps.
The IRQ and workqueue affinity changes reduce the chance of a workqueue lockup, but they may not eliminate it in every environment. The workqueue CPU mask mainly controls unbound workqueues. Bound or per-CPU workqueues can still run on their associated CPU, so kernel workers such as kworker/20:* may still appear on an SPDK CPU even after unbound workqueues are moved away.
If the node still reports occasional workqueue lockups after the IRQ and workqueue affinity mitigation is applied, use the following additional mitigations.
If the affected Longhorn V2 disk is using the aio disk driver, consider changing the requested disk driver to auto. For an NVMe disk, verify after reprovisioning that Longhorn reports the actual disk driver as nvme in the Longhorn Node custom resource. In recent validation, after applying IRQ affinity and confirming that the actual disk driver was nvme, no new workqueue lockup messages were observed during the test window.
caution
Only use the SPDK NVMe disk driver when the NVMe device satisfies the Longhorn V2 IOMMU group isolation requirement. Longhorn uses vfio-pci for the SPDK NVMe path, and VFIO must claim the whole IOMMU group. If the NVMe device shares an IOMMU group with a PCIe bridge or another device that cannot be bound to VFIO, Longhorn cannot use the SPDK NVMe driver for that disk and the disk must stay on the aio driver. For details, see the Longhorn V2 Data Engine requirements.
Before changing the disk driver, remove the affected disk from Harvester. Follow the Remove Disks guide, and make sure the disk no longer contains active Longhorn replicas or backing images.
Find the BlockDevice resource for the disk:
kubectl -n longhorn-system get blockdevices.harvesterhci.io \ -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,PROVISION:.spec.provision,PHASE:.status.provisionPhase,STATE:.status.state,ENGINE:.spec.provisioner.longhorn.engineVersion,DRIVER:.spec.provisioner.longhorn.diskDriver
After the disk is removed and the BlockDevice is no longer provisioned, patch the disk driver:
Provision the disk again from the Harvester UI. Harvester will add the disk back to Longhorn V2. Then verify the requested driver in the BlockDevice resource and the actual driver in the Longhorn Node resource.
Verify the BlockDevice after provisioning:
kubectl -n longhorn-system get blockdevice.harvesterhci.io "${BLOCKDEVICE}" -o yaml
The matching BlockDevice should show:
spec: provision:true provisioner: longhorn: engineVersion: LonghornV2 diskDriver: auto status: provisionPhase: Provisioned state: Active
Also check the Longhorn Node custom resource to confirm the actual disk driver selected by Longhorn:
For the affected disk, SPEC_DRIVER should be auto, and STATUS_DRIVER should be nvme:
NODE DISK TYPE PATH SPEC_DRIVER STATUS_DRIVER hp-161-tink-system 32f43222-1eb1-4ab6-9e65-c4f8ddad700d block 0000:08:00.0 auto nvme
The spec.disks.<blockdevice-name>.diskDriver value is the requested driver mode from Harvester. The status.diskStatus.<blockdevice-name>.diskDriver value is the driver Longhorn actually uses. For an NVMe disk, the status value should be nvme.
If the lockup still occurs after changing the disk driver and applying IRQ affinity, consider enabling Longhorn V2 interrupt mode.
caution
Longhorn V2 interrupt mode should still be treated as experimental. It changes the SPDK execution model from continuous polling to interrupt-driven handling, which can reduce constant CPU pressure but may affect latency and performance. Longhorn also requires all V2 volumes to be detached before this setting can be changed.
Check the setting:
kubectl -n longhorn-system get settings.longhorn.io data-engine-interrupt-mode-enabled
If occasional messages still appear, compare the stuck for <seconds>s value over time. The mitigation is still useful if the stuck time stops growing, the node remains responsive, and RKE2, kubelet, and Longhorn keep making progress.
The Longhorn V2 instance-manager pod is running on the affected node.
This article provides information and mitigation steps for the following vulnerabilities in Harvester:
CVE-2026-53359
important
On July 6, 2026, researcher Hyunwoo Kim (@v4bel) publicly disclosed Januscape, a vulnerability in the Linux kernel’s KVM/x86 memory-management code, which allows a malicious virtual machine to break out of the guest and run code as root on the host it runs on. On hosts where the KVM device node /dev/kvm is world-accessible, an unprivileged local user can exploit the vulnerability to crash the host.
All supported versions of Harvester are affected, including 1.6.1 and earlier, 1.7.2 and earlier, and 1.8.1 and earlier.
Januscape is the latest in a series of Linux kernel privilege-escalation vulnerabilities that required a patch and a reboot of the affected hosts.
SUSE is working on fixing this issue. Meanwhile, apply the mitigation steps described in this article to protect your clusters.
The mitigation steps involves disabling the nested virtualization feature of the KVM kernel module on your Harvester hosts.
note
Nested virtualization is not supported on virtual machines running on Harvester. Disabling this feature will not affect the functionality of your Harvester cluster.
On your Harvester hosts, use the following commands to confirm that the KVM kernel module is loaded with nested virtualization enabled:
Once the configuration is applied, reboot your Harvester hosts for the changes to take effect.
warning
Do not disable the KVM kernel module on your Harvester hosts, as it is required for running virtual machines. Only disable the nested virtualization feature using the configuration provided above.
Once you have upgraded to a fixed version of Harvester, you can re-enable the nested virtualization feature by deleting the CloudInit configuration and rebooting your Harvester hosts:
When running production workloads on virtualized infrastructure like Harvester, memory management is critical. In Harvester versions prior to v1.4.0, certain workloads experienced sudden Virtual Machine (VM) terminations due to the host Linux operating system triggering Out-Of-Memory (OOM) kills.
What is KubeVirt? Harvester uses KubeVirt as its core virtualization engine. KubeVirt is an open-source technology that allows Kubernetes to run and manage traditional Virtual Machines inside standard containers, translating VM specifications directly into Pod configurations.
This article explores why these OOM events occur in a Kubernetes-native virtualization environment and how Harvester provides granular tools to eliminate them.
When a VM is terminated due to insufficient memory at the host level, the Linux kernel logs specific keywords that help pinpoint the fault. In Harvester, these logs generally fall into two distinct categories depending on which process triggered the exhaustion.
Example 1: virt-launcher invoked oom-killer
The virt-launcher process runs inside the dedicated Kubernetes Pod backing the VM. If this component or its direct sub-processes run out of the memory allocated to their cgroup, the kernel triggers a memory cgroup (memcg) OOM event.
Feb 03 19:57:08 ** kernel: virt-launcher invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=986 Feb 03 19:57:08 ** kernel: CPU: 40 PID: 40785 Comm: virt-launcher Tainted: G I X 5.14.21-150400.24.60-default #1 SLE15-SP4 9096397fa6646928cc6d185ba417f2af65b536f1 ... Feb 03 19:57:08 ** kernel: memory: usage 17024340kB, limit 17024340kB, failcnt 1243 Feb 03 19:57:08 ** kernel: memory+swap: usage 17024340kB, limit 9007199254740988kB, failcnt 0 Feb 03 19:57:08 ** kernel: kmem: usage 143556kB, limit 9007199254740988kB, failcnt 0 Feb 03 19:57:08 ** kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice: ... Feb 03 19:57:08 ** kernel: Tasks state (memory values in pages): Feb 03 19:57:08 ** kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Feb 03 19:57:08 ** kernel: [ 38886] 0 38886 243 1 28672 0 -998 pause Feb 03 19:57:08 ** kernel: [ 38917] 0 38917 310400 6921 192512 0 986 virt-launcher-m Feb 03 19:57:08 ** kernel: [ 38934] 0 38934 1200940 25126 954368 0 986 virt-launcher Feb 03 19:57:08 ** kernel: [ 38951] 0 38951 386525 8247 466944 0 986 libvirtd Feb 03 19:57:08 ** kernel: [ 38952] 0 38952 33619 3940 290816 0 986 virtlogd Feb 03 19:57:08 ** kernel: [ 39079] 107 39079 4457263 4201766 34439168 0 986 qemu-system-x86 Feb 03 19:57:08 ** kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice/cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,task=qemu-system-x86,pid=39079,uid=107
Example 2: CPU X/KVM invoked oom-killer
This occurs when a vCPU execution thread inside qemu-system-x86_64 attempts a memory operation that pushes the entire container beyond its Kubernetes memory limit.
[Thu May 9 14:52:38 2024] CPU 11/KVM invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=830 [Thu May 9 14:52:38 2024] CPU: 60 PID: 70888 Comm: CPU 11/KVM Not tainted 5.3.18-150300.59.101-default #1 SLE15-SP3 ... [Thu May 9 14:52:38 2024] memory: usage 67579904kB, limit 67579904kB, failcnt 67391 [Thu May 9 14:52:38 2024] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0 [Thu May 9 14:52:38 2024] kmem: usage 633636kB, limit 9007199254740988kB, failcnt 0 ... [Thu May 9 14:52:38 2024] Tasks state (memory values in pages): [Thu May 9 14:52:38 2024] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [Thu May 9 14:52:38 2024] [ 70675] 0 70675 243 1 28672 0 -998 pause [Thu May 9 14:52:38 2024] [ 70728] 0 70728 310400 5467 188416 0 830 virt-launcher-m [Thu May 9 14:52:38 2024] [ 70746] 0 70746 1242373 25104 1073152 0 830 virt-launcher [Thu May 9 14:52:38 2024] [ 70762] 0 70762 455279 14110 770048 0 830 libvirtd [Thu May 9 14:52:38 2024] [ 70763] 0 70763 37704 3916 339968 0 830 virtlogd [Thu May 9 14:52:38 2024] [ 70870] 107 70870 18302464 16718510 135278592 0 830 qemu-system-x86 [Thu May 9 14:52:38 2024] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,mems_allowed=0-3,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice/cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,task=qemu-system-x86,pid=70870,uid=107 [Thu May 9 14:52:38 2024] Memory cgroup out of memory: Killed process 70870 (qemu-system-x86) total-vm:73209856kB, anon-rss:66852088kB, file-rss:21948kB, shmem-rss:4kB [Thu May 9 14:52:38 2024] oom_reaper: reaped process 70870 (qemu-system-x86), now anon-rss:0kB, file-rss:132kB, shmem-rss:4kB
Root Cause: Native Hypervisors vs. Harvester Architecture
Traditional Linux Host (e.g., Virtual Machine Manager)
On a standard Linux host, a VM managed via QEMU/KVM runs inside a systemd machine.slice. The hypervisor process (qemu-system-x86_64) has access to the host's wider pool of resources, managed loosely unless strict cgroup limits are manually added.
$ systemd-cgls Control group /: -.slice ├─1173 bpfilter_umh ├─system.slice └─machine.slice └─machine-qemu\x2d1\x2dharv41.scope └─8632 /usr/bin/qemu-system-x86_64 -name guest=harv41,debug-threads=on -S -…
In Harvester, every VM is encapsulated inside a Kubernetes Pod. This introduces a strict cgroup boundary (kubepods.slice).
As shown below, multiple helper processes must live alongside the primary qemu-system-x86_64 process within the same tightly limited container memory budget:
When you define a virtual machine, for example, a VM configured with 4 vCPUs, 2 GiB of memory, and 1 Ethernet interface, KubeVirt does not just allocate exactly 2 GiB of memory to the container.
Instead, KubeVirt calculates an additional baseline memory overhead required to operate the virtualization stack. This overhead budget covers:
CPU Simulators: Thread pools tracking guest state and handling context switches.
Memory Management: Tracking structures such as QEMU page tables mapping guest RAM.
Auxiliary Devices: Buffers for virtual network interfaces (NICs), storage queues, and video devices.
Depending on the guest OS type, specific kernel workloads, and heavy storage/network I/O spikes, the memory consumed by these helper tasks can quickly exceed KubeVirt's default calculations. Because Kubernetes enforces a strict hard ceiling on the Pod container, the entire container triggers a CONSTRAINT_MEMCG OOM kill the moment this boundary is breached.
Unlike traditional, stateless Kubernetes workloads where a container crash is quickly mitigated by a rapid pod restart, an OOM kill on a VM pod carries severe operational consequences:
Prolonged Downtime: A virtual machine is a stateful workload. It does not instantaneously serve traffic upon a container restart; it must undergo a full operating system boot cycle, run init scripts, and re-initialize services, drastically extending your Recovery Time Objective (RTO).
Risk of Data Corruption: Sudden terminations during flight can abruptly cut off active storage queues. If the guest OS or database engine is in the middle of a critical write operation when the host terminates the qemu process, it can result in uncommitted journals, filesystem degradation, or severe data corruption on your persistent volumes.
To address this, Harvester introduced dual-layer configurations that give administrators full flexibility over how overhead buffers are calculated.
Global Adjustment: additional-guest-memory-overhead-ratio
This cluster-wide setting functions as a multiplier for KubeVirt's automatically calculated memory overhead. For deep structural details, refer to the Harvester Advanced Documentation.
Definition: Scales the calculated overhead buffer to accommodate heavy I/O or virtualization tasks.
Default Value:1.5 (Provides a 50% safety cushion above baseline calculations).
Valid Range:0 or 1.0 to 10.0.
💡 Important Operational Notes:
Lifecycle Impact: Changes to this setting only apply to newly created virtual machines or existing VMs after they undergo a migration or a full power cycle.
System Overhead: A higher ratio increases the host container's memory allocation, guaranteeing safety for heavy workloads but scaling up the overall system resource reservation footprint.
Resource Allocation Trade-off: Setting this ratio excessively high can lock up unneeded host memory blocks, leading to predictable underutilization and significant memory waste across your compute nodes.
For specific virtual machines running intensive or non-standard workloads, a global multiplier might not offer the precision required. Harvester allows administrators to define a dedicated Reserved Memory value directly on individual VMs. For complete configuration steps, see the Harvester VM Management Documentation.
⚠️ Under the Hood Memory Carving:
When you configure this setting, Harvester explicitly scales down the available memory presented to the Guest OS inside the VM. For example, if a VM is configured with 2 GiB of memory and you set a Reserved Memory value of 256 MiB, the Guest OS will only see and utilize 1.75 GiB (2 GiB - 256 MiB).
Guaranteed Overhead Headroom: By restricting the Guest OS from consuming the top slice of its configured allocation, you guarantee an isolated, un-evictable memory runway for host helper tasks.
Targeted Safety for Heavy Workloads: This mechanism is highly practical for mission-critical, high-performance, or special-purpose workloads (such as nested virtualization layers or intensive database engines). It effectively prevents the VM from running into host-level cgroup OOM termination by proactively limiting its internal usage boundaries, removing the risk of unexpected node-level kills.
Optimized Cluster Usability: Using per-VM reservations eliminates the major disadvantage of cranking up the global additional-guest-memory-overhead-ratio for the whole cluster. Instead of forcing a massive, wasteful memory overhead reservation across every idle or lightweight VM on your hosts, you can maintain a lean global default and surgically protect only the heavy workloads—striking an ideal balance between system density and ironclad stability.
By leveraging this dual-layer tunable memory architecture, Harvester fundamentally alters how host-level overhead is calculated, moving from rigid, generalized defaults to a precise, tiered enforcement model:
Total Memory Overhead = Auto-calculated Overhead * Ratio + Reserved Memory
The following matrix showcases how combinations of Reserved Memory and the Overhead Ratio change the actual layout of the Guest OS space versus what Kubernetes reserves as a hard boundary.
VM Configured Memory
Reserved Memory
additional-guest-memory-overhead-ratio
Guest OS Memory
POD Container Memory Limit
Total Memory Overhead
2 Gi
not configured
"0.0"
2 Gi - 100 Mi
2 Gi + 240 Mi
~340 Mi
2 Gi
256 Mi
"0.0"
2 Gi - 256 Mi
2 Gi + 240 Mi
~500 Mi
2 Gi
not configured
"1.0"
2 Gi
2 Gi + 240*1.0 Mi
~240 Mi
2 Gi
not configured
"3.0"
2 Gi
2 Gi + 240*3.0 Mi
~720 Mi
2 Gi
not configured
"1.5"
2 Gi
2 Gi + 240*1.5 Mi
~360 Mi
2 Gi
256 Mi
"1.5"
2 Gi - 256 Mi
2 Gi + 240*1.5 Mi
~620 Mi
When optimizing your Harvester cluster to eliminate host-level container OOM events, use the following operational checklist to tailor your memory strategies:
For General Workloads:
Stick to the default ratio of 1.5, or configure a slightly higher value of 2.0. This ensures that standard Guest operating systems receive exactly the memory requested while scaling out a stable, predictable background overhead buffer across the cluster.
For High I/O and Storage-Heavy VMs:
If you observe periodic KVM OOM events during massive backup windows, large-scale data syncs, or intensive disk read/write cycles, increase the individual VM's allocation or implement a targeted Reserved Memory configuration to safely expand the helper overhead pool.
For GPU Passthrough Workloads:
Virtual machines utilizing direct hardware acceleration or GPU passthrough are prime candidates for explicit Reserved Memory carving. The underlying host-side device drivers and memory-mapped I/O (MMIO) windows for high-performance graphics hardware require a significantly higher, specialized memory footprint outside the guest OS space. Allocating dedicated per-VM reserved memory prevents driver-instigated cgroup allocation breaches, keeping both the hardware pipeline and the hypervisor completely stable.
The Problem: In Harvester's Kubernetes-native architecture, every virtual machine is bound by a strict Pod container limit. While this rigid cgroup boundary is essential for security, ensuring a single rogue or leaking VM can never starve neighboring workloads or crash the bare-metal host, it means heavy storage/network I/O, device drivers, or GPU passthrough can cause internal helper processes to breach this hard ceiling, triggering a sudden host-level OOM kill.
The Solution: Harvester eliminates these crashes without losing secure resource control using a dual-layer memory tuning strategy:
Globally: The additional-guest-memory-overhead-ratio scales out a safety cushion cluster-wide for newly created or migrated VMs.
Per-VM: The Reserved Memory setting surgically carves out a chunk of the VM's configured RAM exclusively for background helper tasks—preventing wasteful memory reservations across the cluster while safely anchoring high-performance, mission-critical workloads.
Appendix: Lab Simulation — Manually Triggering the Host-Level OOM
For engineers looking to validate this behavior safely in a staging environment, you can replicate this multi-process cgroup breach. A detailed script and case study can be found in the Harvester Development Summary: OOM Investigation.
The simulation process highlights a fundamental truth about modern virtualization boundaries:
The Guest OS is Trustworthy: Testing shows that modern guest operating systems handle internal resource limits reliably. If a runaway application inside the guest OS eats up all available RAM, the guest kernel safely steps in and kills that specific process internally. The VM itself survives, and from the host's perspective, the virtual machine continues running normally.
The Host Cgroup Boundary is the Weak Link: The true host-level crash only happens if processes inside the host cgroup expand unexpectedly. If an infrastructure task or helper process inside the Pod container balloons, it consumes the memory buffer that KubeVirt set aside, causing the entire cgroup, the VM's carrier, to slam into the hard Kubernetes ceiling and trigger a host-level OOM kill.
These vulnerabilities affect RKE2 ingress-nginx controller v1.14.5 and earlier. All Harvester versions that use this controller (including 1.5.2 and earlier, 1.6.1 and earlier, 1.7.1 and earlier, and 1.8.0) are therefore affected.
2026-05-15: Until Harvester 1.7.2 and 1.8.1 are released with the fixes, apply the mitigation steps below to secure your clusters.
You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:
kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'
If the command returns one of the affected versions, perform the following mitigation steps.
The primary resolution is to upgrade Harvester to one of these versions:
1.7.2 or newer
1.8.1 or newer
If upgrade is not possible, apply the following mitigation to protect your clusters.
All ingress resources with the nginx.ingress.kubernetes.io/rewrite-target annotation containing ? in the annotation value are at risk.
By default, Harvester does not include any ingress resources with this annotation. Run the following command on your clusters to identify affected custom ingress resources:
Any ingress resources reported by the above command are vulnerable. They should be updated to either remove the vulnerable annotation or change the annotation value to not contain a question mark ?.
The following validating admission policy can be applied to your cluster to reject ingress resources with the vulnerable configuration:
cat<<EOF | kubectl apply -f - apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: ingress-nginx-annotation-validation-20260514 spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["networking.k8s.io"] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["ingresses"] validations: - expression: | !has(object.metadata.annotations) || !object.metadata.annotations.exists(k, k == 'nginx.ingress.kubernetes.io/rewrite-target') || !object.metadata.annotations['nginx.ingress.kubernetes.io/rewrite-target'].contains('?') message: "Ingress resources with 'nginx.ingress.kubernetes.io/rewrite-target' annotation containing '?' in the annotation value are not allowed, due to the following CVEs: CVE-2026-42945, CVE-2026-42946, CVE-2026-40701, CVE-2026-42934" --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: ingress-nginx-annotation-validation-20260514 spec: policyName: ingress-nginx-annotation-validation-20260514 validationActions: [Deny] EOF
info
This policy is a cluster-scoped resource that requires the proper administrator RBAC permissions to create.
important
This validating policy prevents the inclusion of the vulnerable annotation configuration in new and existing ingress resources. However, it cannot detect or block any vulnerable ingress resources that already exist in the cluster. Therefore, it is important to follow the instructions described above to also identify and update any existing vulnerable ingress resources.
The policy can be removed once you upgrade to Harvester 1.7.2, 1.8.1 or newer:
You have lost the admin password for the Harvester GUI
The admin password for the Harvester GUI can be reset if you can still login via ssh as the rancher user, or if you have the Harvester cluster's kubeconfig file saved locally. For details, see How can I reset the administrator password? in the documentation.
You have lost the rancher user's ssh/console login password
But you still have the Harvester cluster's kubeconfig
The rancher user's ssh/console login password can be reset by creating a CloudInit CRD to update the password. For details, see Password of user rancher in the documentation. Alternately you can create additional users with ssh access as described in How to create an SSH user for Harvester nodes.
You don't have a kubeconfig, but do have physical or remote console access
In this case, the rancher user's ssh/console login password can be changed by editing /oem/90_custom.yaml on each host.
Boot the Harvester installer, but don't proceed with the regular installation process. Instead, once the installer comes up, press CTRL-ALT-F2 to switch to VT2 and login as the rancher user with password rancher. Then proceed with the following steps:
Run sudo -i to become root
Mount the COS_OEM partition from the host:
# mkdir /tmp/oem # mount -L COS_OEM /tmp/oem
Run vim /tmp/oem/90_custom.yaml to edit 90_custom.yaml and change the password for the rancher user. You can specify either a plaintext password (not recommended) or a password hash generated with e.g. openssl passwd -6. Following is the section that you need to edit:
users: rancher: passwd: <PASSWORD_GOES_HERE>
Reboot the host. You should now be able to log in as the rancher user with the new password.
You can't boot the Harvester installer ISO, but can still reboot the host and access the boot menu
If you have no other option, then during system boot, edit the grub config and add rd.break at the end of the kernel command line (the one that starts with linux). This will drop you into the dracut emergency shell, with the root partition mounted under /sysroot. Unfortunately, this shell does not contain any text editor programs. Trying to edit /oem/90_custom.yaml under the circumstances would be unwise. Instead though, you can run this command:
# sed -i 's%rancher.*%rancher:$6$j0.h3TQv8RZPHJkB$3SbV978JLT2Qeq4KSCBZitErNlZZGfrDxnGW5HS0wHzWexGyPzeQBoQmQJetUhLFfquv/X5VWL6odxtlEec1u/:20468::::::%' /sysroot/etc/shadow
Then, hit CTRL-D to continue, and once the system finishes booting, the rancher user's password will be set back to rancherfor this boot only. You can then login on the console and use vim to update /oem/90_custom.yaml and permanently set the password to something more secure as described in the previous section.
These vulnerabilities affect specific versions of the RKE2 ingress-nginx controller (v1.13.7 and earlier, v1.14.3 and earlier). All Harvester versions that use this controller (including 1.5.2 and earlier, 1.6.1 and earlier, and 1.7.0) are therefore affected.
These CVEs are fixed in Harvester 1.7.1 and newer.
important
Harvester does not utilize the ingress-nginx controller custom error backend. Therefore, it is not affected by CVE-2026-24513.
important
Currently, no mitigation is available for CVE-2026-24514. An upgrade to Harvester 1.7.1 is required.
You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:
kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'
If the command returns one of the affected versions, perform one of the following mitigation steps.
The primary resolution is to upgrade to Harvester 1.7.1 or newer, which includes the fixed RKE2 ingress-nginx controller.
If upgrade is not possible, deploy the following validating admission policy to your cluster to reject ingress resources with the vulnerable configuration:
cat<<EOF | kubectl apply -f - apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: ingress-nginx-annotation-validation spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["networking.k8s.io"] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["ingresses"] validations: - expression: | !('nginx.ingress.kubernetes.io/auth-proxy-set-headers' in object.metadata.annotations) && !('nginx.ingress.kubernetes.io/auth-method' in object.metadata.annotations) && (object.spec.rules.all(rule, rule.http.paths.all(path, path.pathType != 'ImplementationSpecific'))) message: "Ingress resources with the vulnerable annotations are not allowed. Please remove the 'nginx.ingress.kubernetes.io/auth-proxy-set-headers' and 'nginx.ingress.kubernetes.io/auth-method' annotations, and avoid using the 'ImplementationSpecific' path type." --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: ingress-nginx-annotation-validation spec: policyName: ingress-nginx-annotation-validation validationActions: [Deny] EOF
info
This policy is a cluster-scoped resource that requires the proper administrator RBAC permissions to create.
This validating policy rejects any ingress resources that contain the:
Node selector constraints can prevent the scheduler from live-migrating a virtual machine to a target node. This often indicates a mismatch between the virtual machine's requirements and the node's capabilities.
A node selector may require a specific CPU feature, but the target node lacks the corresponding label (for example, cpu-feature.node.kubevirt.io/fpu: "true"). This mismatch can occur when the host-model CPU models and features computed by KubeVirt change over time.
You can resolve this issue using four different approaches.
Reboot the virtual machine.
KubeVirt automatically adds node selectors (during a previous migration or initial start) that can restrict scheduling. You can clear these node selectors by rebooting the virtual machine.
Reboot the virtual machine and set up a common CPU model.
You can override KubeVirt's default host-model CPU configuration by setting up a common CPU model for virtual machine migration. The model is applied to the virtual machine as its domain CPU, and to the pod as its node selector configuration.
This is the recommended approach for environments that can tolerate restarting of virtual machines.
Modify the node labels.
If rebooting the virtual machine is not an option, you can manually manipulate the target node's labels to satisfy the scheduling requirements.
Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.
This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.
The annotation itself does not affect the pod's node selector. It only controls the presence of specific CPU-related labels on the node, which the node selector checks against. For more information, see the References section.
Identify labels that are missing from the virtual machine's node selector and add them to the target node.
You can add the missing labels using the following command:
kubectl label node<node-name><key>=<value>
This circumvents the standard scheduling restrictions, allowing the virtual machine to migrate to the target node.
If a new node that lacks the required features is added to the cluster, you must repeat these steps to allow the virtual machine to live-migrate to that node.
Remove the node labels.
If you want to ensure that the virtual machine does not acquire specific node selector constraints after live migration, you can remove the relevant CPU labels from the target node.
Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.
This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.
This method works only if the virtual machine's pod does not have an existing node selector that contains the labels listed in the References section. Otherwise, you must reboot the virtual machine to clear the constraints.
Check if the pod has a node selector.
kubectl get pod <pod-name> -o yaml |grep nodeSelector -A 5 -B 5
If no node selector exists, remove the relevant CPU labels from the node.
Performing this action prevents the pod from acquiring new node selector constraints, thus enabling its future migration to other nodes. However, the successful outcome of that migration is not guaranteed.
When guest virtual machines running on Harvester nodes experience very slow network throughput, disabling Generic Receive Offload (GRO) and Generic Segmentation Offload (GSO) on the host interfaces may resolve the issue.
In the testing environment, guest virtual machines experienced severely degraded download and transfer speeds, dropping as low as 100 bps. This extreme slowdown was particularly evident when apt-get update, curl, and scp were used to transfer data between virtual machines running on different nodes. In contrast, performance remained normal when the virtual machines were hosted on the same node.
The issue was observed in a Harvester cluster hosted on Dell servers using Broadcom NetXtreme-E Series BCM57508 NICs (100 Gbps). mgmt, the built-in cluster network, was used for both management and virtual machine traffic.
Harvester relies on Linux’s bridge-based virtual networking to connect guest virtual machines to physical networks.
The NetXtreme-E BCM57508 NICs were connected to leaf switches configured to transmit jumbo frames. When the default MTU of 1500 is used, these frames should ideally be segmented to approximately 1450 bytes before reaching the Harvester host kernel. However, the packets actually arriving at the kernel were fragmented into unexpectedly small sizes. This forced the kernel to process a significantly higher volume of packets, leading to increased CPU overhead and reduced download throughput.
Packets captures collected using the following command confirmed the unexpectedly small size of the incoming packets.
tcpdump -xx -i <interface-name> <interface-name> is the name of the physical interface on the host connected to the VMs.
Generic Receive Offload (GRO) and Generic Segmentation Offload (GSO) are kernel-level software offloading mechanisms designed to optimize network performance. GRO aggregates multiple small incoming packets into larger ones before passing them to the network stack. GSO performs the opposite on transmission, splitting large packets into smaller frames before sending them to the NIC.
While these features are typically used to enhance performance, in this specific scenario, they interfered with the normal TCP segmentation process. This interference led to inefficient packet segmentation and the creation of an excessive number of small fragments, which ultimately degraded overall network performance.
When GRO and GSO were disabled, the Linux network stack automatically reverted to using standard transport-layer segmentation methods, specifically TCP Segmentation Offload (TSO) and Large Receive Offload (LRO). These mechanisms maintained efficient packet aggregation and segmentation at the appropriate layers, ensuring properly sized packets were presented to the kernel, which successfully restored expected network performance.
The NetXtreme-E BCM57508 NICs may experience suboptimal interaction with GRO and GSO due to a Broadcom driver bug. Enabling these offload mechanisms led to inefficient packetization, producing many small packets instead of fewer large ones, which ultimately reduced network throughput.
