19 posts tagged with "harvester"

Description​

On a cluster configured with a CPU model such as cpu-model.node.kubevirt.io/Cascadelake-Server=true, virtual machines may no longer start after an upgrade to Harvester v1.7.3.

The mitigation for CVE-2019-11135 disables TSX on affected CPUs. As a result, the hle and rtm CPU features are missing on those nodes, and virtual machines that require them cannot start.

For details about CVE-2019-11135, see the Intel Security Advisory. To check whether a processor is affected, refer to Intel's list of affected processors and search for CVE-2019-11135.

Resolution​

Following the check described in Red Hat's TAA advisory, check the TAA status on each node by running:

cat /sys/devices/system/cpu/vulnerabilities/tsx_async_abort

If the output reports Vulnerable, use the corresponding CPU model ending in -noTSX (for example, Skylake-Server-noTSX or Cascadelake-Server-noTSX) to disable TSX for the virtual machine. Interpret other output according to your operating system's security guidance. Make sure the selected model is supported by every node where the virtual machine may run.

Alternatively, configure Cascadelake-Server-v3 or a later version as the CPU model. After changing the CPU model, restart all affected virtual machines.

For more information about choosing and configuring a common CPU model, see Setting up a common CPU model for virtual machine migration.

References​

• harvester/harvester#11784
• CVE-2019-11135 - TSX Asynchronous Abort (Red Hat)
• Setting up a common CPU model for virtual machine migration

This article explains how to install the NetApp Trident CSI driver on a Harvester cluster. Once it's installed, VMs running in Harvester can store their volumes on a NetApp ONTAP storage system.

NetApp storage is added alongside Longhorn; it does not replace it. You can use any combination of these protocols:

• NFS (ontap-nas)
• iSCSI (ontap-san, sanType: iscsi)
• NVMe/TCP (ontap-san, sanType: nvme)
• Fibre Channel (ontap-san, sanType: fcp)

This procedure was tested with Harvester v1.9.0 and Trident 26.06.1 (Helm chart 100.2606.1), using ONTAP 9.16.1. With NFS, iSCSI and NVMe/TCP, the following worked on NetApp-backed volumes: booting VMs, live migration, online volume expansion, and Harvester VM snapshots. Trident also supports Fibre Channel, and the multipath configuration below covers FC LUNs. However, FC was not tested for this article.

Prerequisites​

• A Harvester cluster and its KubeConfig. To download it, open the Harvester UI, click Support in the lower left, then click Download KubeConfig.
• A workstation with kubectl and helm that can reach the Harvester cluster.
• An ONTAP SVM with:
  • A management LIF that the Harvester nodes can reach.
  • Data LIFs for each protocol you plan to use: NFS, iSCSI, NVMe/TCP and/or FC. NVMe/TCP requires the NVMe_oF license on the ONTAP system.
  • An ONTAP login for Trident with the http and ontapi applications enabled, using the SVM-scoped vsadmin role. Point managementLIF at the SVM's own management LIF; Trident then infers the SVM from that connection, so the backends below don't need an explicit svm field either. See User permissions in the Trident documentation.
• For FC only: FC HBAs in every Harvester node that mounts NetApp volumes, and fabric zoning between those HBAs and the SVM's FC LIFs.

Network design​

Trident's node plugin runs in the host network namespace, so the host's routing table decides which NIC carries storage traffic.

How Trident discovers data LIFs differs by protocol:

• iSCSI / NVMe/TCP: Trident ignores dataLIF on SAN backends and logs Specifying data LIF is no longer supported for SAN backends. It asks ONTAP for every SAN LIF in the SVM, and each node logs in to all of them, so multipath works automatically. You only need managementLIF.
• NFS: dataLIF takes a single address. If you leave it out, Trident uses the first NFS LIF it finds in the SVM. Set it explicitly to choose the network that carries NFS traffic.
• FC: traffic runs over the FC fabric, not the IP network, so the rest of this section does not apply. Only the management LIF must be reachable over IP.

Optional: a dedicated storage interface with HostNetworkConfig​

This takes two steps: create a cluster network on the storage NICs, then give each host an IP address on it with a HostNetworkConfig.

Step 1: create the cluster network. A cluster network groups the storage NICs into a bridge on each node. See Create a New Cluster Network for details.

1. In the Harvester UI, go to Networks > ClusterNetworks/Configs and click Create. Name it, for example cn-storage.
2. Click Create Network Config on the new cluster network.
3. On the Node Selector tab, select the nodes that mount NetApp volumes. Select all nodes does not work if the cluster has a witness node. Select nodes by name, or by a label such as netapp-trident=enabled (see the note further down).
4. On the Uplink tab, select the storage NICs, the bond mode, and the MTU. The MTU must match the switch ports and the ONTAP LIFs, for example 9000 for jumbo frames.
5. Click Save.

The switch ports for these NICs must be trunk ports that carry the storage VLAN tagged.

Step 2: give the hosts an IP address on it. On its own, a cluster network carries only VM traffic, because the hosts have no IP address on it. A HostNetworkConfig (Networks > Host Networks) creates a VLAN sub-interface on the cluster network's bridge on each node and gives it an IP address. Traffic to the ONTAP data LIFs on that subnet then leaves through that interface instead of the management network. The ONTAP data LIFs must be on the same VLAN and subnet.

Example with static addresses on VLAN 150:

apiVersion: network.harvesterhci.io/v1beta1
kind: HostNetworkConfig
metadata:
  name: netapp-storage
spec:
  clusterNetwork: cn-storage
  vlanID: 150
  mode: static
  nodeSelector:
    matchLabels:
      netapp-trident: enabled
  ips:
    <node-1>: 192.168.150.11/24
    <node-2>: 192.168.150.12/24

Check the result on a node with ip addr show cn-storage-br.150. Each node's state is reported in the resource's status.nodeStatus.

For iSCSI, you can add a second cluster network (or a second VLAN) with its own HostNetworkConfig and LIFs, so multipath has two independent paths.

1. Configure multipathd (iSCSI and FC)​

iSCSI and FC LUNs need multipathd, which is not enabled on Harvester by default. Both protocols present NetApp LUNs with vendor NETAPP and product LUN, so the same configuration covers them. The configuration below lets multipathd claim only NetApp LUNs. Every other device is blacklisted, so Longhorn and the local disks are never touched. Harvester's own /etc/multipath/conf.d/99-longhorn.conf is left as is.

The following CloudInit resource writes /etc/multipath/conf.d/98-netapp.conf and enables multipathd on every node. It is re-applied on every boot. Save it as 01-cloudinit-netapp-multipathd.yaml:

# Persists across reboots: writes /etc/multipath/conf.d/98-netapp.conf (multipath ONLY claims NETAPP LUNs)
apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: netapp-multipathd
spec:
  matchSelector: {}
  filename: 98_netapp_multipathd.yaml
  contents: |
    name: "enable multipathd for NetApp Trident iSCSI/FC (NETAPP LUNs only)"
    stages:
      initramfs:
        - files:
            - path: /etc/multipath/conf.d/98-netapp.conf
              permissions: 0644
              owner: 0
              group: 0
              content: |
                defaults {
                    user_friendly_names yes
                    find_multipaths no
                    # lets libmpathpersist (qemu-pr-helper) track SCSI-3 PR keys across paths, for shared disks (e.g. WSFC)
                    reservation_key file
                    # don't create kpartx maps for partitions that guests create on the LUNs
                    skip_kpartx yes
                }
                blacklist {
                    device {
                        vendor ".*"
                        product ".*"
                    }
                }
                blacklist_exceptions {
                    device {
                        vendor "NETAPP"
                        product "LUN.*"
                    }
                }
      network:
        - name: "enable and start multipathd"
          systemctl:
            enable:
              - multipathd
            start:
              - multipathd

kubectl apply -f 01-cloudinit-netapp-multipathd.yaml

Despite the initramfs stage name, Harvester's node CloudInit controller writes the file and runs the network stage as soon as the resource is created — no reboot needed. Confirm the file actually merged into the running daemon's config (don't rely on multipath -ll alone, since it's empty if no LUN happens to be attached yet):

multipathd -k"show config" | grep -A6 '^blacklist {'   # must show vendor ".*" / product ".*"

If you'd rather apply it by hand instead of waiting on the controller, run the following on each node as root. It writes the same file and starts multipathd, and stops at the first failure:

set -e
mkdir -p /etc/multipath/conf.d
cat > /etc/multipath/conf.d/98-netapp.conf <<'EOF'
defaults {
    user_friendly_names yes
    find_multipaths no
    reservation_key file
    skip_kpartx yes
}
blacklist {
    device {
        vendor ".*"
        product ".*"
    }
}
blacklist_exceptions {
    device {
        vendor "NETAPP"
        product "LUN.*"
    }
}
EOF
systemctl enable --now multipathd
multipath -ll   # must not list any local disk

Check that every node has a unique iSCSI initiator name. Nodes cloned from the same image can end up with the same name in /etc/iscsi/initiatorname.iscsi. ONTAP then treats those nodes as a single host. If you change a node's IQN, restart that node's trident-node-linux pod afterwards. Trident caches the IQN, and with a stale one volume staging fails with no devices present yet.

2. Install the Trident operator​

helm repo add netapp-trident https://netapp.github.io/trident-helm-chart
helm repo update
helm search repo netapp-trident --versions | head

helm upgrade --install trident netapp-trident/trident-operator --version <chart version> \
  --namespace trident --create-namespace --wait

Pick a version from the helm search output above rather than pinning one in this article; this procedure was tested with chart 100.2606.1 (Trident 26.06.1).

Wait until the trident-controller pod and one trident-node-linux pod per node are Running:

kubectl -n trident get pods -o wide

kubectl label node <node-1> <node-2> netapp-trident=enabled --overwrite

helm upgrade --install trident netapp-trident/trident-operator --version <chart version> \
  --namespace trident --create-namespace --wait \
  --set tridentNodePluginNodeSelector.netapp-trident=enabled

3. Create the ONTAP credentials Secret​

kubectl -n trident create secret generic netapp-creds \
  --from-literal=username=<ontap user> \
  --from-literal=password='<ontap password>'

4. Create the backends​

Create one TridentBackendConfig per protocol you use; you can drop any you don't need.

A few things the examples below deliberately leave out:

• spaceReserve: none and snapshotReserve: "0": these are already Trident's defaults, so setting them changes nothing. (Don't confuse snapshotReserve with thin provisioning, either — it reserves extra space on the ONTAP volume so a PVC's filesystem doesn't run out of room as Snapshot copies grow, it isn't what makes the volume thin.)
• limitAggregateUsage: this stops Trident from creating new volumes once an aggregate passes a given percent used, but aggregates are cluster-level ONTAP objects, so it only works with a cluster-scoped admin login — strictly more access than Trident needs for anything else in this article. On all-flash aggregates there's also no performance cliff near 80% full the way there was with spinning disks, so there's less reason to want it. Watch aggregate usage with your normal ONTAP monitoring instead.
• svm and dataLIF: not needed if managementLIF already points at an SVM-scoped management LIF (Trident infers the SVM from that LIF) and the SVM has only one data LIF per protocol, which is the recommended design. If your SVM has several LIFs for the same protocol, set dataLIF on the NFS backend to pick one explicitly (see Network design).
• nfsMountOptions: this is a Trident-side fallback, not the Kubernetes-native way to set mount options — it doesn't show up in the resulting PV's mountOptions field. Set mount options on the StorageClass instead (step 5).

If several Kubernetes or Harvester clusters share one SVM, give each cluster a unique backendName and set storagePrefix (for example storagePrefix: hv1_). That keeps volume names from colliding and makes each cluster's volumes easy to identify in ONTAP.

For Fibre Channel, copy the iSCSI backend below, then change name and backendName (for example to netapp-fc) and set sanType: fcp. Then add a matching StorageClass in step 5 with sanType: fcp. FC was not tested for this article.

Save as 03-backends.yaml, replacing the <...> values:

# NFS
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-nas
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-nas
  backendName: netapp-nas
  managementLIF: <SVM management LIF IP>
  autoExportPolicy: true
  autoExportCIDRs: ["<storage network CIDR>"]
  credentials:
    name: netapp-creds
---
# iSCSI (all iSCSI LIFs in the SVM are discovered; no dataLIF)
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-iscsi
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-san
  sanType: iscsi
  backendName: netapp-iscsi
  managementLIF: <SVM management LIF IP>
  credentials:
    name: netapp-creds
---
# NVMe/TCP (all NVMe LIFs in the SVM are discovered; no dataLIF)
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-nvme
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-san
  sanType: nvme
  backendName: netapp-nvme
  managementLIF: <SVM management LIF IP>
  credentials:
    name: netapp-creds

kubectl apply -f 03-backends.yaml
kubectl -n trident get tridentbackendconfigs

Every backend must report PHASE Bound and STATUS Success. If one doesn't, kubectl -n trident describe tridentbackendconfig <name> shows the error.

5. Create the StorageClasses and VolumeSnapshotClass​

A StorageClass with only backendType: ontap-san can place a volume on any SAN backend: iSCSI, NVMe/TCP or FC. Pin each class to a protocol with the sanType parameter (iscsi, nvme or fcp) instead of matching backends by name with storagePools — it's the more direct mechanism and keeps working if you ever rename a backend. storagePools is still the right tool if you run multiple SVMs and need to steer volumes to one of them.

For the NFS class, add fsType: "nfs". It's not needed to mount the volume, but without it Kubernetes skips applying a pod's fsGroup to the volume (no setgid bit, no group ownership change) — worth the one extra line. Mount options for NFS also belong here, as the StorageClass's top-level mountOptions, not on the backend.

Save as 04-storageclasses.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-nfs
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-nas"
  fsType: "nfs"
mountOptions:
  - nfsvers=4.1
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-iscsi
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-san"
  sanType: "iscsi"
  fsType: "ext4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-nvme
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-san"
  sanType: "nvme"
  fsType: "ext4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: trident-snapshotclass
driver: csi.trident.netapp.io
deletionPolicy: Delete

kubectl apply -f 04-storageclasses.yaml

Live migration requires ReadWriteMany volumes. CDI already knows Trident's defaults:

• ontap-san (iSCSI, NVMe/TCP and FC): ReadWriteMany with volumeMode: Block.
• ontap-nas: ReadWriteMany with volumeMode: Filesystem.

So these StorageClasses don't need the cdi.harvesterhci.io/storageProfileVolumeModeAccessModes annotation. None of the other cdi.harvesterhci.io/storageProfile* annotations from older third-party storage write-ups are needed either — check kubectl get storageprofile <storageclass> -o yaml after creating the class, and its status is already filled in (access modes, cloneStrategy: snapshot, snapshotClass) because CDI recognizes csi.trident.netapp.io directly. The separate storageclass.kubevirt.io/is-default-virt-class annotation is unrelated to any of this — it only matters if you want a NetApp class, instead of Longhorn, to be the default for new VM disks.

6. Register Trident for Harvester VM snapshots​

Harvester needs to know which VolumeSnapshotClass to use for Trident volumes.

1. In the Harvester UI, go to Advanced > Settings, find csi-driver-config, and select ⋮ > Edit Setting.
2. Click Add, then fill in the fields:
   • Provisioner: csi.trident.netapp.io
   • Volume Snapshot Class Name: trident-snapshotclass
3. Save.

Leave Backup Volume Snapshot Class Name empty. In Harvester v1.9, VM backups only work with Longhorn volumes. VM snapshots of Trident volumes work. See Third-Party Storage Support.

The resulting entry in the setting's JSON value is:

"csi.trident.netapp.io": {
  "volumeSnapshotClassName": "trident-snapshotclass",
  "backupVolumeSnapshotClassName": ""
}

7. Use it​

The new StorageClasses now appear when you create a volume, a VM image or a VM in the Harvester UI. A quick check:

1. Create a VM whose root disk uses netapp-iscsi, netapp-nvme or netapp-nfs.
2. Start the VM, live-migrate it to another node, and take a VM snapshot.
3. On a node running the VM, multipath -ll (iSCSI) or ls /sys/class/nvme-subsystem (NVMe) shows the NetApp devices.

Troubleshooting​

• Backends go offline after the ONTAP system was unreachable. If they don't recover on their own, restart the controller: kubectl -n trident rollout restart deploy/trident-controller.
• Deleted volumes still use space on ONTAP. ONTAP keeps deleted volumes in the volume recovery queue for 12 hours. While a deleted FlexClone (such as a VM disk cloned from an image) sits in the queue, its parent snapshot stays busy, so the parent volume and its VolumeSnapshot can't be deleted. To free them immediately, run on ONTAP: set -privilege advanced; volume recovery-queue purge-all -vserver <svm name>.
• multipath -ll lists local disks. multipathd was started without the NetApp-only configuration. Put 98-netapp.conf in place, flush the unused maps with multipath -F, and restart multipathd.
• iSCSI volumes fail to stage with no devices present yet. Check for duplicate initiator names across nodes, and for a stale IQN in Trident (see step 1).

References​

• NetApp Trident documentation
• Deploy the Trident operator using Helm
• ONTAP SAN driver and ONTAP NAS driver
• Harvester third-party storage support

Harvester v1.7.0 introduced the kubevirt-migration setting, which exposes the cluster-wide live migration configuration that previously could only be changed by editing the kubevirt object directly.

If you customized spec.configuration.migrations on the kubevirt object before upgrading, that customization is at risk. This article explains why and provides a script that converts it into the setting so it is preserved.

Who is affected​

You are affected if both of the following are true:

• You are running a version earlier than v1.7.0 and you changed spec.configuration.migrations on the kubevirt object, for example to enable allowAutoConverge.
• You plan to upgrade to v1.7.0 or later.

You are also affected if you have already upgraded to v1.7.x and have not yet changed the kubevirt-migration setting. In that case the configuration is still intact and the same script applies.

Check whether the kubevirt object carries a configuration:

kubectl get kubevirt kubevirt -n harvester-system -o jsonpath='{.spec.configuration.migrations}' | jq

If the command returns nothing, your cluster uses the KubeVirt defaults and no action is required.

What happens without this procedure​

The upgrade creates the kubevirt-migration setting without a value, so the Harvester UI displays the default values. The kubevirt object keeps your configuration, so live migration continues to behave as before.

The two are now inconsistent, and the inconsistency is not visible unless you compare them. Harvester writes the whole value of the setting to the kubevirt object, so the first time the setting is saved, all fields are written at once, including the fields that you did not change. Your configuration is replaced by the defaults, without warning.

In practice this means that changing an unrelated field such as parallelMigrationsPerCluster in the UI silently turns allowAutoConverge back off. Migrations of write-heavy virtual machines then start failing to converge, which is often noticed only during the node drains of a later upgrade.

Procedure​

Run the following script from a machine with kubectl access to the cluster. jq is required.

Run it before you upgrade to v1.7.0 or later. If you already upgraded to v1.7.x and have not yet changed the setting, run it now.

#!/bin/bash
set -euo pipefail

SETTING_NAME="kubevirt-migration"

migrations=$(kubectl get kubevirt kubevirt -n harvester-system -o json | jq -c '.spec.configuration.migrations // empty')
if [ -z "$migrations" ]; then
  echo "The kubevirt object has no migration configuration. Nothing to preserve."
  exit 0
fi

# nodeDrainTaintKey and network cannot be configured through the setting:
# nodeDrainTaintKey is used by the upgrade process, and network is owned by
# the vm-migration-network setting. The webhook rejects a value containing them.
value=$(echo "$migrations" | jq -c 'del(.nodeDrainTaintKey, .network)')
if [ "$value" = "{}" ]; then
  echo "The migration configuration only contains fields that the setting does not manage. Nothing to preserve."
  exit 0
fi

if setting_json=$(kubectl get settings.harvesterhci.io "$SETTING_NAME" -o json 2>/dev/null); then
  current_value=$(echo "$setting_json" | jq -r '.value // empty')
  if [ -n "$current_value" ]; then
    echo "The $SETTING_NAME setting already has a value. Nothing to do."
    exit 0
  fi

  echo "Setting $SETTING_NAME to: $value"
  kubectl patch settings.harvesterhci.io "$SETTING_NAME" --type merge \
    -p "$(jq -n --arg v "$value" '{"value": $v}')"
else
  echo "Creating $SETTING_NAME with: $value"
  kubectl apply -f - <<EOF
apiVersion: harvesterhci.io/v1beta1
kind: Setting
metadata:
  name: $SETTING_NAME
value: '$value'
EOF
fi

echo "Result:"
kubectl get settings.harvesterhci.io "$SETTING_NAME" -o yaml

The script does not write to the kubevirt object, so it cannot change how the cluster performs live migrations. It is idempotent: it does nothing if there is no configuration to preserve, or if the setting already has a value.

Verification​

Confirm that the setting now holds your configuration:

kubectl get settings.harvesterhci.io kubevirt-migration -o jsonpath='{.value}' | jq

Confirm that the kubevirt object is unchanged:

kubectl get kubevirt kubevirt -n harvester-system -o jsonpath='{.spec.configuration.migrations}' | jq

After you upgrade, the two remain consistent, and the Harvester UI shows the configuration that the cluster is actually using.

After the upgrade​

The kubevirt-migration setting is the source of truth. Harvester writes it to the kubevirt object whenever the setting changes.

Do not edit spec.configuration.migrations on the kubevirt object directly. Any change is overwritten the next time the setting is reconciled. Use the setting instead, either in the Harvester UI under Advanced > Settings, or with kubectl.

For the list of supported fields and their defaults, see kubevirt-migration in the Harvester documentation.

For background on the individual options and on per-virtual-machine migration policies, see VM Live Migration Policy and Configuration.

When running production workloads on virtualized infrastructure like Harvester, memory management is critical. In Harvester versions prior to v1.4.0, certain workloads experienced sudden Virtual Machine (VM) terminations due to the host Linux operating system triggering Out-Of-Memory (OOM) kills.

What is KubeVirt? Harvester uses KubeVirt as its core virtualization engine. KubeVirt is an open-source technology that allows Kubernetes to run and manage traditional Virtual Machines inside standard containers, translating VM specifications directly into Pod configurations.

This article explores why these OOM events occur in a Kubernetes-native virtualization environment and how Harvester provides granular tools to eliminate them.

Anatomy of a VM OOM Event​

When a VM is terminated due to insufficient memory at the host level, the Linux kernel logs specific keywords that help pinpoint the fault. In Harvester, these logs generally fall into two distinct categories depending on which process triggered the exhaustion.

Example 1: virt-launcher invoked oom-killer

The virt-launcher process runs inside the dedicated Kubernetes Pod backing the VM. If this component or its direct sub-processes run out of the memory allocated to their cgroup, the kernel triggers a memory cgroup (memcg) OOM event.

Feb 03 19:57:08 ** kernel: virt-launcher invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=986
Feb 03 19:57:08 ** kernel: CPU: 40 PID: 40785 Comm: virt-launcher Tainted: G          I    X    5.14.21-150400.24.60-default #1 SLE15-SP4 9096397fa6646928cc6d185ba417f2af65b536f1
...

Feb 03 19:57:08 ** kernel: memory: usage 17024340kB, limit 17024340kB, failcnt 1243
Feb 03 19:57:08 ** kernel: memory+swap: usage 17024340kB, limit 9007199254740988kB, failcnt 0                                                                 
Feb 03 19:57:08 ** kernel: kmem: usage 143556kB, limit 9007199254740988kB, failcnt 0
Feb 03 19:57:08 ** kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice:
...
Feb 03 19:57:08 ** kernel: Tasks state (memory values in pages):
Feb 03 19:57:08 ** kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Feb 03 19:57:08 ** kernel: [  38886]     0 38886      243        1    28672        0          -998 pause
Feb 03 19:57:08 ** kernel: [  38917]     0 38917   310400     6921   192512        0           986 virt-launcher-m
Feb 03 19:57:08 ** kernel: [  38934]     0 38934  1200940    25126   954368        0           986 virt-launcher
Feb 03 19:57:08 ** kernel: [  38951]     0 38951   386525     8247   466944        0           986 libvirtd
Feb 03 19:57:08 ** kernel: [  38952]     0 38952    33619     3940   290816        0           986 virtlogd
Feb 03 19:57:08 ** kernel: [  39079]   107 39079  4457263  4201766 34439168        0           986 qemu-system-x86
Feb 03 19:57:08 ** kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice/cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,task=qemu-system-x86,pid=39079,uid=107

Example 2: CPU X/KVM invoked oom-killer

This occurs when a vCPU execution thread inside qemu-system-x86_64 attempts a memory operation that pushes the entire container beyond its Kubernetes memory limit.

[Thu May  9 14:52:38 2024] CPU 11/KVM invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=830
[Thu May  9 14:52:38 2024] CPU: 60 PID: 70888 Comm: CPU 11/KVM Not tainted 5.3.18-150300.59.101-default #1 SLE15-SP3
...
[Thu May  9 14:52:38 2024] memory: usage 67579904kB, limit 67579904kB, failcnt 67391
[Thu May  9 14:52:38 2024] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[Thu May  9 14:52:38 2024] kmem: usage 633636kB, limit 9007199254740988kB, failcnt 0
...
[Thu May  9 14:52:38 2024] Tasks state (memory values in pages):
[Thu May  9 14:52:38 2024] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Thu May  9 14:52:38 2024] [  70675]     0 70675      243        1    28672        0          -998 pause
[Thu May  9 14:52:38 2024] [  70728]     0 70728   310400     5467   188416        0           830 virt-launcher-m
[Thu May  9 14:52:38 2024] [  70746]     0 70746  1242373    25104  1073152        0           830 virt-launcher
[Thu May  9 14:52:38 2024] [  70762]     0 70762   455279    14110   770048        0           830 libvirtd
[Thu May  9 14:52:38 2024] [  70763]     0 70763    37704     3916   339968        0           830 virtlogd
[Thu May  9 14:52:38 2024] [  70870]   107 70870 18302464 16718510 135278592        0           830 qemu-system-x86
[Thu May  9 14:52:38 2024] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,mems_allowed=0-3,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice/cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,task=qemu-system-x86,pid=70870,uid=107
[Thu May  9 14:52:38 2024] Memory cgroup out of memory: Killed process 70870 (qemu-system-x86) total-vm:73209856kB, anon-rss:66852088kB, file-rss:21948kB, shmem-rss:4kB
[Thu May  9 14:52:38 2024] oom_reaper: reaped process 70870 (qemu-system-x86), now anon-rss:0kB, file-rss:132kB, shmem-rss:4kB

Root Cause: Native Hypervisors vs. Harvester Architecture​

Traditional Linux Host (e.g., Virtual Machine Manager)​

On a standard Linux host, a VM managed via QEMU/KVM runs inside a systemd machine.slice. The hypervisor process (qemu-system-x86_64) has access to the host's wider pool of resources, managed loosely unless strict cgroup limits are manually added.

$ systemd-cgls
Control group /:
-.slice
├─1173 bpfilter_umh
├─system.slice

└─machine.slice
  └─machine-qemu\x2d1\x2dharv41.scope 
    └─8632 /usr/bin/qemu-system-x86_64 -name guest=harv41,debug-threads=on -S -…

Harvester (Kubernetes/KubeVirt Engine)​

In Harvester, every VM is encapsulated inside a Kubernetes Pod. This introduces a strict cgroup boundary (kubepods.slice).

As shown below, multiple helper processes must live alongside the primary qemu-system-x86_64 process within the same tightly limited container memory budget:

-.slice
└─kubepods.slice
  │ ├─kubepods-burstable-pod99ee3a64_645b_4699_9384_5a3875d78b41.slice
  │ │ ├─cri-containerd-0c316eb8a4711bff1ce968b46ddb658e49378897454b4caf2a20704c808f33f1.scope …
  │ │ │ └─ 8505 /pause
  │ │ ├─cri-containerd-eb5deef29f064adfd2456d9f9c535674ac4d0c95c81b13afbdab5a89dc6a774b.scope …
  │ │ │ └─ 8590 /usr/bin/virt-tail --logfile /var/run/kubevirt-private/2ce151aa…
  │ │ └─cri-containerd-fd57a5cfc2b9b1f53eaf7b575c3273e6784f4c56a04a17d502ecfbd19e55b066.scope …
  │ │   ├─ 8542 /usr/bin/virt-launcher-monitor --qemu-timeout 301s --name vm2 -…
  │ │   ├─ 8558 /usr/bin/virt-launcher --qemu-timeout 301s --name vm2 --uid 2ce…
  │ │   ├─ 8591 /usr/sbin/virtqemud -f /var/run/libvirt/virtqemud.conf
  │ │   ├─ 8592 /usr/sbin/virtlogd -f /etc/libvirt/virtlogd.conf
  │ │   └─ 8823 /usr/bin/qemu-system-x86_64 -name guest=default_vm2,debug-threa…

The "Hidden" Memory Overhead​

Breaking Down the Memory Overhead Buffer​

When you define a virtual machine, for example, a VM configured with 4 vCPUs, 2 GiB of memory, and 1 Ethernet interface, KubeVirt does not just allocate exactly 2 GiB of memory to the container.

Instead, KubeVirt calculates an additional baseline memory overhead required to operate the virtualization stack. This overhead budget covers:

• CPU Simulators: Thread pools tracking guest state and handling context switches.
• Memory Management: Tracking structures such as QEMU page tables mapping guest RAM.
• Auxiliary Devices: Buffers for virtual network interfaces (NICs), storage queues, and video devices.

The High Stakes of VM OOM Kills​

Depending on the guest OS type, specific kernel workloads, and heavy storage/network I/O spikes, the memory consumed by these helper tasks can quickly exceed KubeVirt's default calculations. Because Kubernetes enforces a strict hard ceiling on the Pod container, the entire container triggers a CONSTRAINT_MEMCG OOM kill the moment this boundary is breached.

Unlike traditional, stateless Kubernetes workloads where a container crash is quickly mitigated by a rapid pod restart, an OOM kill on a VM pod carries severe operational consequences:

• Prolonged Downtime: A virtual machine is a stateful workload. It does not instantaneously serve traffic upon a container restart; it must undergo a full operating system boot cycle, run init scripts, and re-initialize services, drastically extending your Recovery Time Objective (RTO).

• Risk of Data Corruption: Sudden terminations during flight can abruptly cut off active storage queues. If the guest OS or database engine is in the middle of a critical write operation when the host terminates the qemu process, it can result in uncommitted journals, filesystem degradation, or severe data corruption on your persistent volumes.

The Solution: Tunable Memory Architectures​

To address this, Harvester introduced dual-layer configurations that give administrators full flexibility over how overhead buffers are calculated.

Global Adjustment: additional-guest-memory-overhead-ratio​

This cluster-wide setting functions as a multiplier for KubeVirt's automatically calculated memory overhead. For deep structural details, refer to the Harvester Advanced Documentation.

• Definition: Scales the calculated overhead buffer to accommodate heavy I/O or virtualization tasks.
• Default Value: 1.5 (Provides a 50% safety cushion above baseline calculations).
• Valid Range: 0 or 1.0 to 10.0.

💡 Important Operational Notes:

• Lifecycle Impact: Changes to this setting only apply to newly created virtual machines or existing VMs after they undergo a migration or a full power cycle.
• System Overhead: A higher ratio increases the host container's memory allocation, guaranteeing safety for heavy workloads but scaling up the overall system resource reservation footprint.
• Resource Allocation Trade-off: Setting this ratio excessively high can lock up unneeded host memory blocks, leading to predictable underutilization and significant memory waste across your compute nodes.

Per-VM Optimization: Reserved Memory​

For specific virtual machines running intensive or non-standard workloads, a global multiplier might not offer the precision required. Harvester allows administrators to define a dedicated Reserved Memory value directly on individual VMs. For complete configuration steps, see the Harvester VM Management Documentation.

⚠️ Under the Hood Memory Carving: When you configure this setting, Harvester explicitly scales down the available memory presented to the Guest OS inside the VM. For example, if a VM is configured with 2 GiB of memory and you set a Reserved Memory value of 256 MiB, the Guest OS will only see and utilize 1.75 GiB (2 GiB - 256 MiB).

Why Use Per-VM Reserved Memory?​

• Guaranteed Overhead Headroom: By restricting the Guest OS from consuming the top slice of its configured allocation, you guarantee an isolated, un-evictable memory runway for host helper tasks.

• Targeted Safety for Heavy Workloads: This mechanism is highly practical for mission-critical, high-performance, or special-purpose workloads (such as nested virtualization layers or intensive database engines). It effectively prevents the VM from running into host-level cgroup OOM termination by proactively limiting its internal usage boundaries, removing the risk of unexpected node-level kills.

• Optimized Cluster Usability: Using per-VM reservations eliminates the major disadvantage of cranking up the global additional-guest-memory-overhead-ratio for the whole cluster. Instead of forcing a massive, wasteful memory overhead reservation across every idle or lightweight VM on your hosts, you can maintain a lean global default and surgically protect only the heavy workloads—striking an ideal balance between system density and ironclad stability.

Outcome: Guaranteed Workload Stability​

By leveraging this dual-layer tunable memory architecture, Harvester fundamentally alters how host-level overhead is calculated, moving from rigid, generalized defaults to a precise, tiered enforcement model:

Total Memory Overhead = Auto-calculated Overhead * Ratio + Reserved Memory

Best Practices & Configuration Matrix​

The following matrix showcases how combinations of Reserved Memory and the Overhead Ratio change the actual layout of the Guest OS space versus what Kubernetes reserves as a hard boundary.

When optimizing your Harvester cluster to eliminate host-level container OOM events, use the following operational checklist to tailor your memory strategies:

• For General Workloads:

  • Stick to the default ratio of 1.5, or configure a slightly higher value of 2.0. This ensures that standard Guest operating systems receive exactly the memory requested while scaling out a stable, predictable background overhead buffer across the cluster.

• For High I/O and Storage-Heavy VMs:

  • If you observe periodic KVM OOM events during massive backup windows, large-scale data syncs, or intensive disk read/write cycles, increase the individual VM's allocation or implement a targeted Reserved Memory configuration to safely expand the helper overhead pool.

• For GPU Passthrough Workloads:

  • Virtual machines utilizing direct hardware acceleration or GPU passthrough are prime candidates for explicit Reserved Memory carving. The underlying host-side device drivers and memory-mapped I/O (MMIO) windows for high-performance graphics hardware require a significantly higher, specialized memory footprint outside the guest OS space. Allocating dedicated per-VM reserved memory prevents driver-instigated cgroup allocation breaches, keeping both the hardware pipeline and the hypervisor completely stable.

Quick Summary​

• The Problem: In Harvester's Kubernetes-native architecture, every virtual machine is bound by a strict Pod container limit. While this rigid cgroup boundary is essential for security, ensuring a single rogue or leaking VM can never starve neighboring workloads or crash the bare-metal host, it means heavy storage/network I/O, device drivers, or GPU passthrough can cause internal helper processes to breach this hard ceiling, triggering a sudden host-level OOM kill.

• The Solution: Harvester eliminates these crashes without losing secure resource control using a dual-layer memory tuning strategy:

  • Globally: The additional-guest-memory-overhead-ratio scales out a safety cushion cluster-wide for newly created or migrated VMs.
  • Per-VM: The Reserved Memory setting surgically carves out a chunk of the VM's configured RAM exclusively for background helper tasks—preventing wasteful memory reservations across the cluster while safely anchoring high-performance, mission-critical workloads.

Appendix: Lab Simulation — Manually Triggering the Host-Level OOM​

For engineers looking to validate this behavior safely in a staging environment, you can replicate this multi-process cgroup breach. A detailed script and case study can be found in the Harvester Development Summary: OOM Investigation.

The simulation process highlights a fundamental truth about modern virtualization boundaries:

• The Guest OS is Trustworthy: Testing shows that modern guest operating systems handle internal resource limits reliably. If a runaway application inside the guest OS eats up all available RAM, the guest kernel safely steps in and kills that specific process internally. The VM itself survives, and from the host's perspective, the virtual machine continues running normally.

• The Host Cgroup Boundary is the Weak Link: The true host-level crash only happens if processes inside the host cgroup expand unexpectedly. If an infrastructure task or helper process inside the Pod container balloons, it consumes the memory buffer that KubeVirt set aside, causing the entire cgroup, the VM's carrier, to slam into the hard Kubernetes ceiling and trigger a host-level OOM kill.

Issue​

Node selector constraints can prevent the scheduler from live-migrating a virtual machine to a target node. This often indicates a mismatch between the virtual machine's requirements and the node's capabilities.

A node selector may require a specific CPU feature, but the target node lacks the corresponding label (for example, cpu-feature.node.kubevirt.io/fpu: "true"). This mismatch can occur when the host-model CPU models and features computed by KubeVirt change over time.

Solutions​

You can resolve this issue using four different approaches.

• Reboot the virtual machine.

  KubeVirt automatically adds node selectors (during a previous migration or initial start) that can restrict scheduling. You can clear these node selectors by rebooting the virtual machine.

• Reboot the virtual machine and set up a common CPU model.

  You can override KubeVirt's default host-model CPU configuration by setting up a common CPU model for virtual machine migration. The model is applied to the virtual machine as its domain CPU, and to the pod as its node selector configuration.

  This is the recommended approach for environments that can tolerate restarting of virtual machines.

• Modify the node labels.

  If rebooting the virtual machine is not an option, you can manually manipulate the target node's labels to satisfy the scheduling requirements.

  1. Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.

     This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.

     kubectl annotate node <node-name> node-labeller.kubevirt.io/skip-node="true"
     

     Copy

     important

     The annotation itself does not affect the pod's node selector. It only controls the presence of specific CPU-related labels on the node, which the node selector checks against. For more information, see the References section.

  2. Identify labels that are missing from the virtual machine's node selector and add them to the target node.

     You can add the missing labels using the following command:

     kubectl label node <node-name> <key>=<value>
     

     Copy

     This circumvents the standard scheduling restrictions, allowing the virtual machine to migrate to the target node.

  If a new node that lacks the required features is added to the cluster, you must repeat these steps to allow the virtual machine to live-migrate to that node.

• Remove the node labels.

  If you want to ensure that the virtual machine does not acquire specific node selector constraints after live migration, you can remove the relevant CPU labels from the target node.

  1. Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.

     This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.

     kubectl annotate node <node-name> node-labeller.kubevirt.io/skip-node="true"
     

     Copy

     important

     This method works only if the virtual machine's pod does not have an existing node selector that contains the labels listed in the References section. Otherwise, you must reboot the virtual machine to clear the constraints.

  2. Check if the pod has a node selector.

     kubectl get pod <pod-name> -o yaml | grep nodeSelector -A 5 -B 5
     

     Copy

  3. If no node selector exists, remove the relevant CPU labels from the node.

     Performing this action prevents the pod from acquiring new node selector constraints, thus enabling its future migration to other nodes. However, the successful outcome of that migration is not guaranteed.

References​

KubeVirt Node Labels​

The KubeVirt CPU node-labeller manages the following labels:

• cpu-feature.node.kubevirt.io/*
• cpu-model-migration.node.kubevirt.io/*
• cpu-model.node.kubevirt.io/*
• host-model-cpu.node.kubevirt.io
• host-model-required-features.node.kubevirt.io

External Links​

• Labeling nodes with CPU models, CPU features and machine types

Problem Description​

For Harvester to successfully migrate a virtual machine from one node to another, the source and target nodes must have compatible CPU models and features.

Harvester uses KubeVirt / QEMU / libvirt to manage and run virtual machines. When a VM starts, libvirt exposes a specific CPU feature set to the guest operating system. Live migration requires this CPU feature set to be identical on both source and target nodes.

Different CPU generations support different instruction sets and feature flags. If those differ, the live migration will be blocked to avoid instability or incorrect execution on the target node.

If the CPU model of a virtual machine isn't specified, KubeVirt assigns it the default host-model configuration so that the virtual machine has the CPU model closest to the one used on the host node.

KubeVirt automatically adjusts the node selectors of the associated virt-launcher Pod based on this configuration. If the CPU models and features of the source and target nodes do not match, the live migration may fail.

Let's examine an example.

When a virtual machine is first migrated to another node with the SierraForest CPU model, the following key-value pairs are added to the spec.nodeSelector field in the Pod spec.

spec:
  nodeSelector:
    cpu-model-migration.node.kubevirt.io/SierraForest: "true"
    cpu-feature.node.kubevirt.io/fpu: "true"
    cpu-feature.node.kubevirt.io/vme: "true"

The above nodeSelector configuration is retained for subsequent migrations, which may fail if the new target node doesn't have the corresponding features or model.

For example, compare the CPU model and feature labels added by KubeVirt to the following two nodes:

# Node A
labels:
  cpu-model-migration.node.kubevirt.io/SierraForest: "true"
  cpu-feature.node.kubevirt.io/fpu: "true"
  cpu-feature.node.kubevirt.io/vme: "true"

# Node B
labels:
  cpu-model-migration.node.kubevirt.io/SierraForest: "true"
  cpu-feature.node.kubevirt.io/vme: "true"

This virtual machine will fail to migrate to Node B due to the missing fpu feature. However, if the virtual machine doesn't actually require this feature, this can be frustrating. Therefore, setting up a common CPU model can resolve this issue.

How to Set Up a Common CPU Model​

You can define a custom CPU model to ensure that the spec.nodeSelector configuration in the Pod spec is assigned a CPU model that is compatible and common to all nodes in the cluster.

Consider this example.

We have the following node information:

# Node A
labels:
  cpu-model.node.kubevirt.io/IvyBridge: "true"
  cpu-feature.node.kubevirt.io/fpu: "true"
  cpu-feature.node.kubevirt.io/vme: "true"

# Node B
labels:
  cpu-model.node.kubevirt.io/IvyBridge: "true"
  cpu-feature.node.kubevirt.io/vme: "true"

If we set up IvyBridge as our CPU model in the virtual machine spec, KubeVirt only adds cpu-model.node.kubevirt.io/IvyBridge under spec.nodeSelector in the Pod spec.

# Virtual Machine Spec
spec:
  template:
    spec:
      domain:
        cpu:
          model: IvyBridge

# Pod spec
spec:
  nodeSelector:
    cpu-model.node.kubevirt.io/IvyBridge: "true"

With this configuration, your virtual machine can be migrated to any node that has the label cpu-model.node.kubevirt.io/IvyBridge.

Understanding CPU Model Options in Harvester​

Harvester allows the CPU model to be defined in the VM specification.

If host-passthrough is used, the VM exposes the exact host CPU to the guest. This provides maximum performance but completely prevents migration across different CPU generations.

If host-model is used, the VM derives a CPU model based on the host’s capabilities. This works only when all nodes expose identical CPU features. In mixed clusters, this setting commonly causes migration failures.

If an explicit CPU model is defined, the VM exposes a named CPU architecture defined by QEMU. This is the recommended approach for clusters that may contain different CPU generations. For mixed environments, an explicit CPU model should always be used.

Choosing a CPU Model for Live Migration​

When selecting a CPU model, the goal is to choose the highest common denominator. This means the most modern CPU architecture that every node in the cluster supports. Pick a model supported by all nodes in your cluster, including the oldest CPU generation. Use a server-grade model that provides a reasonable set of capabilities (e.g., vector instructions, security features) without tying VMs to host-specific features. Use the same model on all VMs that need live migration.

Below are practical examples for common enterprise hardware combinations.

For a cluster that mixes Cascade Lake and Sapphire Rapids, the recommended CPU model is Cascadelake-Server. Sapphire Rapids processors are backward compatible with Cascade Lake instructions, making Cascadelake-Server the highest common denominator between these two generations.

For a cluster that mixes Skylake and Cascade Lake nodes, the appropriate model is Skylake-Server. This is the common ground between both generations. Cascade Lake includes additional AVX-512 optimizations, but using Skylake ensures compatibility across the entire cluster.

If your cluster contains Broadwell nodes along with anything newer, the safest baseline is Broadwell. Broadwell serves as a stable and widely supported baseline for older enterprise hardware. Any newer CPUs will support it.

CVE-2019-11135: TSX Asynchronous Abort​

Some CPU models (such as Cascadelake-Server) may stop working after an upgrade because of the mitigation for CVE-2019-11135. For details and the resolution, see CVE-2019-11135: TSX Asynchronous Abort.

Set Up Cluster-Wide Configuration​

If your virtual machines run only on a specific CPU model, you can set up a cluster-wide CPU model in the kubevirt resource.

You can edit it with kubectl edit kubevirt kubevirt -n harvester-system, then add the CPU model you want in the following spec:

spec:
  configuration:
    cpuModel: IvyBridge

Then, when a new virtual machine starts or an existing virtual machine restarts, the cluster-wide setting will be applied. The system follows these priorities when using CPU models if you configure them in both locations:

1. CPU model in the virtual machine spec.
2. CPU model in the KubeVirt spec.

References​

• CPU Model Matching
• QEMU / KVM CPU model configuration
• CVE-2019-11135 - TSX Asynchronous Abort (Red Hat)

Harvester's embedded Rancher UI may display warnings about expiring KubeVirt certificates. You can safely ignore these warnings because automatic certificate rotation is handled by KubeVirt and is enabled by default.

KubeVirt Certificate Rotation Strategy​

KubeVirt provides a self-signed certificate mechanism that rotates both CA and certifcates on a defined recurring interval. You can check the setting certificateRotateStrategy by running the following command:

kubectl get kubevirt -n harvester-system -o yaml

By default, the value of certificateRotateStrategy is empty, which means that KubeVirt uses its default rotation settings and no manual configuration is required.

certificateRotateStrategy: {}

Configuration Fields​

You can use the following fields to configure certificateRotateStrategy.

• .ca.duration: Validity period of the CA certificate. The default value is "168h".
• .ca.renewBefore: Amount of time before a CA certificate expires during which a new certificate is issued. The default value is "33.6h".
• .server.duration: Validity period of server component certificates (for example, virt-api, virt-handler, and virt-operator). The default value is "24h".
• .server.renewBefore: Amount of time before a server certificate expires during which a new certificate is issued. The default value is "4.8h".

Example of a complete configuration:

certificateRotateStrategy:
  selfSigned:
    ca:
      duration: 168h
      renewBefore: 33.6h
    server:
      duration: 24h
      renewBefore: 4.8h

Certificate Rotation Triggers​

Certificate rotation can be triggered by several conditions. The following list only outlines key triggers and is not exhaustive.

• Missing certificate: A required certificate does not exist.
• Invalid CA signature: A certificate was not signed by the specified CA.
• Proactive renewal: The renewBefore value takes effect. A new certificate must be issued before the current one expires.
• CA expiration: The CA certificate has expired, so the certificate signed by the CA is also rotated.

When certificate rotation is triggered, you should see virt-operator log records similar to the following:

{"component":"virt-operator","level":"info","msg":"secret kubevirt-virt-api-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.045809Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-controller-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.056759Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-exportproxy-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.063530Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-virt-handler-server-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.068608Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-virt-handler-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.074555Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-operator-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.078719Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-export-ca updated","pos":"core.go:278","timestamp":"2024-12-06T08:03:36.063496Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-ca updated","pos":"core.go:278","timestamp":"2024-12-06T08:04:06.052750Z"}

References​

• Harvester: Issue 5798
• https://kubevirt.io/2020/KubeVirt-Security-Fundamentals.html
• https://github.com/kubevirt/kubevirt/blob/v1.1.1/pkg/virt-operator/resource/generate/components/secrets.go#L326
• https://github.com/kubevirt/kubevirt/blob/v1.1.1/pkg/virt-operator/resource/apply/certificates.go

Scenarios:

1. The Harvester cluster is installed with 3+ nodes.

2. The Rancher Manager/Server is deployed independently. (Hereafter it is mentioned as Rancher Manager)

3. The Harvester cluster is imported to this Rancher Manager and works as a node driver.

4. The Rancher Manager deploys a couple of downstream K8s clusters, the machines/nodes of those clusters are backed by Harvester VMs.

5. There are also some traditional VMs deployed on the Harvester cluster, which have no direct connection with the Rancher Manager.

You plan to move those Harvester nodes geographically, or to power off the whole cluster for some time, it is essential to shutdown the Harvester cluster and restart later.

General Principle​

To safely shutdown a Harvester cluster, you need to follow the roughly reverse order of the cluster installation and the workload deployments.

Those facts need to be taken into account particularly:

1. The common methodology of Kubernetes operator/controller is to try things continuously until they meet expectations. When the cluster is shutting down node by node, if you don't stop those workloads in advance, they will try hard until the last node is off. It causes the last few nodes to have heavy CPU/memory/network/storage usage and increases the chance of data corruption.

2. Each Harvester node has limited capacity of CPU/memory/network/storage and the max-pod-number, when all workloads are crowded on the last few nodes, the unexpected pod eviction, scheduling failure and other phenomena may happen.

3. Harvester has embedded Longhorn as the default CSI driver, each PV can have 3 or more replicas, when replicas are rescheduled to other nodes, Longhorn will copy data from source node and rebuild the replica. Undoubtedly, stop the PVs as much as possible before the cluster shutdown to avoid the data moving.

4. Unlike normal Kubernetes deployments which have no PVs and are more flexible & agile to deploy anywhere on the cluster, the VMs are backed by massive sized PVs, slowly to move/migrate or even pinned on certain nodes to take the advantage of PCI-passthrough/vGPU/... and are much more sensitive to data consistency.

Needless to say, it is a bad practice to brutally power off the nodes on production environments.

1. Precondition​

1.1 Generate a Support-bundle File​

For trouble-shooting purpose, it is essential to follow this instruction to generate a support-bundle file before taking any actions. And make sure the workload namespaces are added.

1.2 Keep Network Stability​

A good practice is to have detailed documents about the infrastructure related settings.

• The bare metal server NIC slot/port connections with the remote (ToR) Switches.

• The VLAN for the management network.

• (Optional) The DHCP Server, ip-pools and ip-mac bindings for the Harvester cluster if DHCP server is used. If there is no fixed IP binding, when the server restarts after some days it may get a different IP from the DHCP server.

• The VLANs for the VM networks, the CIDRs, default gateways and optional DHCP servers.

• NTP servers.

• DNS servers.

• (Optional) The http proxy.

• (Optional) The private containerd-registry.

• (Optional) The firewall configurations.

See the Harvester ISO Installation to review the infrastructure related settings for the Harvester cluster.

Before the Harvester cluster is restarted later, check and test those settings again to make sure the infrastructure is ready.

2. Backup​

(Optional) Backup VMs if Possible​

It is always a good practice to backup things before a whole cluster shutdown.

(Optional) Backup Downstream K8s Clusters if Possible​

Harvester doesn't touch the (Rancher Manager managed) downstream K8s clusters' workload, when they are not able to be migrated to other node drivers, suggests to backup those clusters.

(Optional) Stop or Migrate Downstream K8s Clusters if Possible​

Harvester doesn't touch the downstream K8s clusters' workload, but suggests to stop or migrate the downstream clusters to avoid your service interruption.

3. Shutdown Workloads​

3.1 Shutdown Traditional VMs​

1. Shutdown VM from the VM shell (e.g. Linux shutdown command), the OS itself will save data to disks.

2. Check the VM status from Harvester UI - VM page, when it is not Off, then click the Stop command.

3.2 Shutdown Rancher Downstream Cluster Machines(VMs)​

Suppose your Harvester cluster was imported to Rancher as a node driver before.

When Rancher deploys a downstream cluster on node driver Harvester, it creates a couple of VMs on Harvester automatically. Directly stopping those VMs on Harvester is not a good practice when Rancher is still managing the downstream cluster. For example, Rancher may create new VMs if you stop them from Harvester.

If you have got a solution to shutdown those downstream clusters, and check those VMs are Off; or there is no downstream clusters, then jump to the step disable some addons.

Unless you have already deleted all the downstream clusters which are deploy on this Harvester, DO NOT remove this imported Harvester from the Rancher Manager. Harvester will get a different driver-id when it is imported later, but those aforementioned downstream clusters are connected to driver-id.

To safely shutdown those VMs but still keep the Rancher Manager managed downstream cluster alive, please follow the steps below:

Disconnect Harvester from the Rancher Manager​

The cattle-cluster-agent-*** pod is the direct connection between Rancher Manager and Harvester cluster, and this pod is monitored and managed by the embedded Rancher in Harvester, scaling down this pod does not work. The embedded Rancher will scale it up automatically.

Run steps below to suspend the connection.

All following CLI commands are executed upon Harvester cluster.

1. Set the management.cattle.io/scale-available of deployment rancher to be "" instead of "3" or other values.

This change will stop the auto-scaling.

harvester$ kubectl edit deployment -n cattle-system rancher
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
...
    management.cattle.io/scale-available: "3"  // record this value, and change it to ""
...
  generation: 16
  labels:
    app: rancher
    app.kubernetes.io/managed-by: Helm
...
  name: rancher
  namespace: cattle-system

2. Scale down the rancher deployment.

harvester$ kubectl scale deployment -n cattle-system rancher --replicas=0
deployment.apps/rancher scaled


harvester$ get deployment -n cattle-system rancher
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
rancher   0/0     0            0           33d

3. Make sure the rancher-* pods are gone.

Check the rancher-* pods on cattle-system are gone, if any of them is stucking at Terminating, use kubectl delete pod -n cattle-system rancher-pod-name --force to delete it.

harvester$ kubectl get pods -n cattle-system
NAME                                         READY   STATUS        RESTARTS       AGE
..
rancher-856f674f7d-5dqb6                     0/1     Terminating   0              3d22h
rancher-856f674f7d-h4vsw                     1/1     Running       23 (68m ago)   33d
rancher-856f674f7d-m6s4r                     0/1     Pending       0              3d19h
...

4. Scale down the cattle-cluster-agent deployment.

harvester$ kubectl scale deployment -n cattle-system cattle-cluster-agent --replicas=0
deployment.apps/cattle-cluster-agent scaled


harvester$ kubectl get deployment -n cattle-system
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
cattle-cluster-agent        0/0     0            0           23d

Please note:

1. From now on, this Harvester is Unavailable on the Rancher Manager.

2. The Harvester WebUI returns 503 Service Temporarily Unavailable, all operations below can be done via kubectl.

Shutdown Rancher Downstream Cluster Machines(VMs)​

1. Shutdown VM from the VM shell (e.g. Linux shutdown command).

2. Check the vmi instances, if any is still Running, stop it.

harvester$ kubectl get vmi
NAMESPACE   NAME   AGE    PHASE     IP            NODENAME   READY
default     vm1    5m6s   Running   10.52.0.214   harv41     True


harvester$ virtctl stop vm1 --namespace default
VM vm1 was scheduled to stop

harvester$ kubectl get vmi -A
NAMESPACE   NAME   AGE    PHASE     IP            NODENAME   READY
default     vm1    5m6s   Running   10.52.0.214   harv41     False


harvester$ kubectl get vmi -A
No resources found

harvester$ kubectl get vm -A
NAMESPACE   NAME   AGE   STATUS    READY
default     vm1    7d    Stopped   False

3.3 Disable Some Addons​

Harvester has some addons which are backed by PVCs, it is necessary to disable them.

The rancher-monitoring addon should be disabled.

The experimental Rancher Manager addon should be disabled.

For other addons, please follow the Harvester document to keep or disable them.

From Harvester UI addon page, write down those none-Disabled addons, click Disable menu to disable them, wait until the state becomes Disabled

From CLI:

$ kubectl get addons.harvesterhci.io -A

NAMESPACE                  NAME                    HELMREPO                                                 CHARTNAME                         ENABLED
cattle-logging-system      rancher-logging         http://harvester-cluster-repo.cattle-system.svc/charts   rancher-logging                   false
cattle-monitoring-system   rancher-monitoring      http://harvester-cluster-repo.cattle-system.svc/charts   rancher-monitoring                true
harvester-system           harvester-seeder        http://harvester-cluster-repo.cattle-system.svc/charts   harvester-seeder                  false
harvester-system           nvidia-driver-toolkit   http://harvester-cluster-repo.cattle-system.svc/charts   nvidia-driver-runtime             false
harvester-system           pcidevices-controller   http://harvester-cluster-repo.cattle-system.svc/charts   harvester-pcidevices-controller   false
harvester-system           vm-import-controller    http://harvester-cluster-repo.cattle-system.svc/charts   harvester-vm-import-controller    false

Example: disable rancher-monitoring

$ kubectl edit addons.harvesterhci.io -n cattle-monitoring-system rancher-monitoring

...
spec:
  chart: rancher-monitoring
  enabled: false               // set this field to be false
...

3.4 (Optional) Disable other Workloads​

If you have deployed some customized workloads on the Harvester cluster directly, it is better to disable/remove them.

3.5 Check Longhorn Volumes​

The volumes should be in state detached, check the related workload if some volumes are still in state attached.

harvester$ kubectl get volume -A
NAMESPACE         NAME                                       DATA ENGINE   STATE      ROBUSTNESS   SCHEDULED   SIZE          NODE     AGE
longhorn-system   pvc-3323944c-00d9-4b35-ae38-a00b1e8a8841   v1            detached   unknown                  5368709120             13d
longhorn-system   pvc-394713a4-d08c-4a45-bf7a-d44343f29dea   v1            attached   healthy                  6442450944    harv41   8d    // still attached and in use
longhorn-system   pvc-5cf00ae2-e85e-413e-a4f1-8bc4242d4584   v1            detached   unknown                  2147483648             13d
longhorn-system   pvc-620358ca-94b3-4bd4-b008-5c144fd815c9   v1            attached   healthy                  2147483648    harv41   8d    // still attached and in use
longhorn-system   pvc-8174f05c-919b-4a8b-b1ad-4fc110c5e2bf   v1            detached   unknown                  10737418240            13d

4. Shutdown Nodes​

Get all nodes from Harvester WebUI Host Management.

From CLI:

harvester$ kubectl get nodes -A
NAME     STATUS   ROLES                       AGE   VERSION
harv2    Ready    <none>                      24d   v1.27.10+rke2r1  // worker node
harv41   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1  // control-plane node
harv42   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1  // control-plane node
harv43   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1  // control-plane node

4.1 Shutdown the Worker Nodes​

1. SSH to the Harvester worker nodes.

2. Run command sudo -i shutdown.

$ sudo -i shutdown

Shutdown scheduled for Mon 2024-07-22 06:58:56 UTC, use 'shutdown -c' to cancel.

3. Wait until all those nodes are down.

4.2 Shutdown Control-plane Nodes and Witness Node​

So far, there are generally three control-plane nodes left, and three etcd-* pods are running in kube-system namespaces.

The first step is to find which one of the etcd-* pod is running as the leader.

Run below command upon any of the etcd-* pod, note the IS LEADER column.

$ kubectl exec -n kube-system etcd-harv41 -- env ETCDCTL_API=3 etcdctl endpoint status --cluster -w table --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key

+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.122.141:2379 | c70780b7862269c9 |   3.5.9 |   34 MB |      true |      false |        45 |    6538756 |            6538756 |        |
| https://192.168.122.142:2379 | db04095b49eb5352 |   3.5.9 |   34 MB |     false |       true |        45 |    6538756 |            6538756 |        |
| https://192.168.122.143:2379 | c27585769b2ce977 |   3.5.9 |   34 MB |     false |       true |        45 |    6538756 |            6538756 |        |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Witness Node​

If your cluster has one Witness Node and the etcd leader happens to be on this node.

harvester$ kubectl get nodes -A
NAME     STATUS     ROLES                       AGE    VERSION
harv2    Ready      <none>                      25d    v1.27.10+rke2r1  // worker node
harv41   Ready      control-plane,etcd,master   55d    v1.27.10+rke2r1  // control-plane node
harv42   Ready      control-plane,etcd,master   55d    v1.27.10+rke2r1  // control-plane node
harv43   Ready      etcd                         1d    v1.27.10+rke2r1  // witness node

+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.122.141:2379 | c70780b7862269c9 |   3.5.9 |   34 MB |     false |       true |        46 |    6538829 |            6538829 |        |
| https://192.168.122.142:2379 | db04095b49eb5352 |   3.5.9 |   34 MB |     false |       true |        46 |    6538829 |            6538829 |        |
| https://192.168.122.143:2379 | a21534d02463b347 |   3.5.9 |   34 MB |      true |      false |        46 |    6538829 |            6538829 |        |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Run kubectl delete pod -n kube-system etcd-name command to delete the etcd pod on the witness node to trigger the pod replacement and leader re-election so that the etcd leader will be located on one of the control-plane nodes. Check the etcd leader again to make sure.

+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.122.141:2379 | c70780b7862269c9 |   3.5.9 |   34 MB |      true |      false |        47 |    6538833 |            6538833 |        |
| https://192.168.122.142:2379 | db04095b49eb5352 |   3.5.9 |   34 MB |     false |       true |        47 |    6538833 |            6538833 |        |
| https://192.168.122.143:2379 | a21534d02463b347 |   3.5.9 |   34 MB |     false |       true |        47 |    6538833 |            6538833 |        |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

To now, the etcd has three running instances and the leader is located on the control-plane node.

Shutdown the two IS LEADER == false nodes one by one.

harvester-node-shell$ sudo -i shutdown

4.3 Shutdown the Last Control-plane Node​

Shutdown the last IS LEADER == true node. Remember its physical information for restarting it first in the steps below.

harvester-last-node-shell$ sudo -i shutdown

5. Restart​

If the Harvester cluster has been moved to a new location, or has been off for days, or your infrastructure has changes, check and test the network stability.

5.1 Restart the Control-plane Nodes and the Witness Node​

The first step is to start those etcd located nodes one after another.

Restart the Leader Control-plane Node​

Power on the last shutdown node first. After about three minutes, continue the next step.

When you check the etcd pod log on this node, the following message may be observed.

sent MsgPreVote request to db04095b49eb5352 at term 5

"msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"db04095b49eb5352","rtt":"0s","error":"dial tcp 192.168.122.142:2380: connect: no route to host"

The etcd is wating for the other two members to be online and then vote a leader.

Restart the Rest of Control-plane Nodes and the Witness Node​

Power on the rest nodes which also hosted the etcd pod before.

Wait until all the three control-plane nodes or possibly two control-plane and one witness nodes are Ready.

From CLI:

harvester$ kubectl get nodes -A
NAME     STATUS   ROLES                       AGE   VERSION
harv41   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1
harv42   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1
harv43   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1

The etcd forms a quorum and can tolerant the failure of one node.

Check the VIP​

The following EXTERNAL-IP should be the same as the VIP of the Harvester cluster.

harvester$ kubectl get service -n kube-system ingress-expose
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)                      AGE
ingress-expose   LoadBalancer   10.53.50.107   192.168.122.144   443:32701/TCP,80:31480/TCP   34d

5.2 Restart the Worker Nodes​

Wait until all nodes are Ready.

From CLI:

harvester$ kubectl get nodes -A
NAME     STATUS   ROLES                       AGE   VERSION
harv2    Ready    <none>                      24d   v1.27.10+rke2r1  // worker node
harv41   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1
harv42   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1
harv43   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1

Healthy Check​

Basic Components​

Harvester deploys some basic components on the following namespaces. When a bare-metal server is powered on, it may take upto around 15 minutes for the Harvester OS to be running and all the deployments on this node to be ready.

If any of them continues to show the status like Failed/CrashLoopBackOff, a troubleshooting is needed to confirm the root cause.

NAMESPACE                         NAME                                                     READY   STATUS      RESTARTS       AGE
cattle-fleet-local-system         fleet-agent-645766877f-bt424                             1/1     Running     0              11m

cattle-fleet-system               fleet-controller-57f78dcd48-5tkkj                        1/1     Running     4 (14m ago)    42h
cattle-fleet-system               gitjob-d5bb7b548-jscgk                                   1/1     Running     2 (14m ago)    42h

cattle-system                     harvester-cluster-repo-6c6458bd46-7jcrl                  1/1     Running     2 (14m ago)    42h
cattle-system                     system-upgrade-controller-6f86d6d4df-f8jg7               1/1     Running     2 (14m ago)    42h
cattle-system                     rancher-7bc9d94b87-g4k4v                                 1/1     Running     3 (14m ago)    42h  // note: if embedded Rancher was stopped in the above steps, it is not Running now
cattle-system                     rancher-webhook-6c5c6fbb65-2cbbs                         1/1     Running     2 (14m ago)    42h

harvester-system                  harvester-787b467f4-qlfwt                                1/1     Running     2 (14m ago)    39h
harvester-system                  harvester-load-balancer-56d9c8758c-cvcmk                 1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-load-balancer-webhook-6b4d4d9d6b-4tsgl         1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-network-controller-9pzxh                       1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-network-controller-manager-69bcf67c7f-44zqj    1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-network-webhook-6c5d48bdf5-8kn9r               1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-node-disk-manager-c4c5k                        1/1     Running     3 (14m ago)    42h
harvester-system                  harvester-node-manager-qbvbr                             1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-node-manager-webhook-6d8b48f559-m5shk          1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-webhook-87dc4cdd8-jg2q6                        1/1     Running     2 (14m ago)    39h
harvester-system                  kube-vip-n4s8l                                           1/1     Running     3 (14m ago)    42h
harvester-system                  virt-api-799b99fb65-g8wgq                                1/1     Running     2 (14m ago)    42h
harvester-system                  virt-controller-86b84c8f8f-4hhlg                         1/1     Running     2 (14m ago)    42h
harvester-system                  virt-controller-86b84c8f8f-krq4f                         1/1     Running     3 (14m ago)    42h
harvester-system                  virt-handler-j9gwn                                       1/1     Running     2 (14m ago)    42h
harvester-system                  virt-operator-7585847fbc-hvs26                           1/1     Running     2 (14m ago)    42h

kube-system                       cloud-controller-manager-harv41                          1/1     Running     5 (14m ago)    42h
kube-system                       etcd-harv41                                              1/1     Running     2              42h
kube-system                       harvester-snapshot-validation-webhook-8594c5f8f8-8mk57   1/1     Running     2 (14m ago)    42h
kube-system                       harvester-snapshot-validation-webhook-8594c5f8f8-dkjmf   1/1     Running     2 (14m ago)    42h
kube-system                       harvester-whereabouts-cpqvl                              1/1     Running     2 (14m ago)    42h

kube-system                       kube-apiserver-harv41                                    1/1     Running     2              42h
kube-system                       kube-controller-manager-harv41                           1/1     Running     4 (14m ago)    42h
kube-system                       kube-proxy-harv41                                        1/1     Running     2 (14m ago)    42h
kube-system                       kube-scheduler-harv41                                    1/1     Running     2 (14m ago)    42h
kube-system                       rke2-canal-d5kmc                                         2/2     Running     4 (14m ago)    42h
kube-system                       rke2-coredns-rke2-coredns-84b9cb946c-qbwnb               1/1     Running     2 (14m ago)    42h
kube-system                       rke2-coredns-rke2-coredns-autoscaler-b49765765-6bjsk     1/1     Running     2 (14m ago)    42h
kube-system                       rke2-ingress-nginx-controller-cphgw                      1/1     Running     2 (14m ago)    42h
kube-system                       rke2-metrics-server-655477f655-gsnsc                     1/1     Running     2 (14m ago)    42h
kube-system                       rke2-multus-8nqg4                                        1/1     Running     2 (14m ago)    42h
kube-system                       snapshot-controller-5fb6d65787-nmjdh                     1/1     Running     2 (14m ago)    42h
kube-system                       snapshot-controller-5fb6d65787-phvq7                     1/1     Running     3 (14m ago)    42h

longhorn-system                   backing-image-manager-5c32-ea70                          1/1     Running     0              13m
longhorn-system                   csi-attacher-749459cf65-2x792                            1/1     Running     6 (13m ago)    42h
longhorn-system                   csi-attacher-749459cf65-98tj4                            1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-attacher-749459cf65-nwglq                            1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-provisioner-775b4f76f4-h9mwd                         1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-provisioner-775b4f76f4-nvjzt                         1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-provisioner-775b4f76f4-zvd6w                         1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-resizer-68867d54f5-4hf5j                             1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-resizer-68867d54f5-fs9ht                             1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-resizer-68867d54f5-ht5hj                             1/1     Running     6 (13m ago)    42h
longhorn-system                   csi-snapshotter-8469656cc7-6c47f                         1/1     Running     6 (13m ago)    42h
longhorn-system                   csi-snapshotter-8469656cc7-9kk2v                         1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-snapshotter-8469656cc7-vf9z4                         1/1     Running     5 (13m ago)    42h
longhorn-system                   engine-image-ei-94d5ee6c-pqx9h                           1/1     Running     2 (14m ago)    42h
longhorn-system                   instance-manager-beb75434e263a2aa9eedc0609862fed2        1/1     Running     0              13m
longhorn-system                   longhorn-csi-plugin-85qm7                                3/3     Running     14 (13m ago)   42h
longhorn-system                   longhorn-driver-deployer-6448498bc6-sv857                1/1     Running     2 (14m ago)    42h
longhorn-system                   longhorn-loop-device-cleaner-bqg9v                       1/1     Running     2 (14m ago)    42h
longhorn-system                   longhorn-manager-nhxbl                                   2/2     Running     6 (14m ago)    42h
longhorn-system                   longhorn-ui-7f56fcf5ff-clc8b                             1/1     Running     6 (13m ago)    42h
longhorn-system                   longhorn-ui-7f56fcf5ff-m95sh                             1/1     Running     7 (13m ago)    42h

Storage Network​

When the Storage Network has been enabled on the cluster, follow those steps to check if the Longhorn PODs have the correct second IP assigned to them.

5.3 Enable Addons​

Enable those previously disabled addons, wait until they are DepoloySuccessful.

5.4 Restore the Connection to the Rancher Manager​

Run following 1, 2 commands on the Harvester cluster.

1. Set the management.cattle.io/scale-available of rancher deployment to be the value recorded on the above steps.

This change will enable the auto-scaling.

harvester$ kubectl edit deployment -n cattle-system rancher
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
...
    management.cattle.io/scale-available: "3"  // recorded on the above steps
...
  generation: 16
  labels:
    app: rancher
    app.kubernetes.io/managed-by: Helm
...
  name: rancher
  namespace: cattle-system

2. Scale up the rancher deployment on Harvester cluster.

harvester$ kubectl scale deployment -n cattle-system rancher --replicas=3
deployment.apps/rancher scaled

harvester$ get deployment -n cattle-system rancher
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
rancher   0/0     0            0           33d

...

harvester$ kubectl get deployment -n cattle-system
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
cattle-cluster-agent        2/2     2            2           23d
rancher                     1/2     2            1           33d

3. Check the virtualization management on the Rancher Manager.

The Harvester cluster continues to be active on the Rancher Virtualization Management .

4. Check the Harvester cluster WebUI.

You should be able to access the Harvester WebUI again.

5.5 Start VMs​

5.1 Start Traditional VMs​

When there are many VMs deployed on the cluster, don't start all the VMs at the same time, suggest starting them group by group.

Wait until they are Running.

5.2 Rancher Downstream Cluster Machines(VMs)​

After the Harvester cluster is re-connected to the Rancher Manager successfully, the Rancher Manager will handle the downstream K8s clusters' machines(vms) automatically. Wait until all the downstream clusters are ready.

See Rancher Manager Access Downstream Clusters to monitor and operator the downstream clusters.

If Rancher Manager does not restart the machines(vms) automatically, you can start those VMs from the Vitrual Machines page on Harvester UI.

5.6 Generate a new Support-bundle File​

Generate a new support-bundle file on the Harvester cluster.

Together with the previously generated support-bundle file, the two files record the cluster settings, configurations and status before shutting down and after rebooting. It is helpful for troubleshooting.

User-Provided Credentials on Harvester​

When installing a Harvester cluster, you are asked to provide the following credential related information:

• Cluster token of the first node that is added to the cluster. Other nodes must use this token to join the cluster.

• Password for the default Linux user rancher on each node.

• SSH keys on each node (optional).

• HTTP proxy on each node (optional).

You may plan to change them from time to time, the following paragraphs describe the detailed steps.

Cluster Token​

Cluster Token on Nodes Joining an Existing Cluster​

When a node is unable to join a cluster because of a cluster token error, perform the recommended troubleshooting steps.

Cluster Token (RKE2 Token Rotation)​

Harvester does not allow you to change the cluster token even if RKE2 is a core component of Harvester.

The RKE2 documentation states that the November 2023 releases of RKE2 (v1.28.3+rke2r2, v1.27.7+rke2r2, v1.26.10+rke2r2, and v1.25.15+rke2r2) allow you to rotate the cluster token using the command rke2 token rotate --token original --new-token new.

During testing, the command was run on the first node of a cluster running Harvester v1.3.0 with RKE2 v1.27.10+rke2r1.

1. Rotate the token on initial node.

/opt/rke2/bin $ ./rke2 token rotate --token rancher --new-token rancher1

WARNING: Recommended to keep a record of the old token. If restoring from a snapshot, you must use the token associated with that snapshot.
WARN[0000] Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation. 
Token rotated, restart rke2 nodes with new token

2. When the first cluster node was rebooted, RKE2 service was unable to start.

RKE2 log:

...
May 29 15:45:11 harv41 rke2[3293]: time="2024-05-29T15:45:11Z" level=info msg="etcd temporary data store connection OK"
May 29 15:45:11 harv41 rke2[3293]: time="2024-05-29T15:45:11Z" level=info msg="Reconciling bootstrap data between datastore and disk"
May 29 15:45:11 harv41 rke2[3293]: time="2024-05-29T15:45:11Z" level=fatal msg="Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token"
May 29 15:45:11 harv41 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
...

This known issue was logged on Github issue rke2 token rotate does not work as expected (v1.27.10+rke2r1).

:::Warning

Do not attempt to rotate the RKE2 token on your cluster before Harvester announces official support for this feature (even if the embedded RKE2 binary has the token rotate option).

:::

Password of the Default User rancher​

This process is node-specific. You must change the password of the default user on each node even if the same password is used on all Harvester nodes.

SSH keys​

You must log into a Harvester node using the default user account rancher to change the SSH keys.

HTTP Proxy​

After a Harvester cluster is installed, you can use the Harvester UI to change the HTTP proxy.

Alternatively, you can use kubectl or the rest API against the URI /harvesterhci.io.setting/http-proxy.

$ kubectl get settings.harvesterhci.io http-proxy -oyaml

apiVersion: harvesterhci.io/v1beta1
default: '{}'
kind: Setting
metadata:
  creationTimestamp: "2024-05-13T20:44:20Z"
  generation: 1
  name: http-proxy
  resourceVersion: "5914"
  uid: 282506bb-f1dd-4247-bf0e-93640698c1f5
status: {}

Harvester has a webhook that checks this setting to ensure it meets all conditions, e.g. the internal IPs and CIDRs are specified in the noProxy field.

Other Credentials and Settings​

auto-rotate-rke2-certs​

Harvester is built on top of Kubernetes, RKE2, and Rancher. RKE2 generates a list of *.crt and *.key files that allow Kubernetes components to function. The *.crt file expires after one year by default.

$ ls /var/lib/rancher/rke2/server/tls/ -alth

...
-rw-r--r-- 1 root root  570 May 27 08:45 server-ca.nochain.crt
-rw------- 1 root root 1.7K May 27 08:45 service.current.key
-rw-r--r-- 1 root root  574 May 27 08:45 client-ca.nochain.crt
drwxr-xr-x 2 root root 4.0K May 13 20:45 kube-controller-manager
drwxr-xr-x 2 root root 4.0K May 13 20:45 kube-scheduler
drwx------ 6 root root 4.0K May 13 20:45 .
drwx------ 8 root root 4.0K May 13 20:45 ..
-rw-r--r-- 1 root root 3.9K May 13 20:40 dynamic-cert.json
drwx------ 2 root root 4.0K May 13 20:39 temporary-certs
-rw------- 1 root root 1.7K May 13 20:39 service.key
-rw-r--r-- 1 root root 1.2K May 13 20:39 client-auth-proxy.crt
-rw------- 1 root root  227 May 13 20:39 client-auth-proxy.key
-rw-r--r-- 1 root root 1.2K May 13 20:39 client-rke2-cloud-controller.crt
...
-rw-r--r-- 1 root root 1.2K May 13 20:39 client-admin.crt
-rw------- 1 root root  227 May 13 20:39 client-admin.key
...


$ openssl x509 -enddate -noout -in /var/lib/rancher/rke2/server/tls/client-admin.crt

notAfter=May 13 20:39:42 2025 GMT

When a cluster has been running for over one year, Kubernetes components may fail to start after upgrades or node rebooting. The workaround is to delete the related files and restart the pod.

Harvester v1.3.0 added the setting auto-rotate-rke2-certs, which allows you to set the Harvester cluster to automatically rotate certificates for RKE2 services. When you enable the setting and specify a certificate validity period, Harvester automatically replaces the certificate before the specified period ends.

Harvester Cloud Credentials​

See the article Renew Harvester Cloud Credentials.

additional-ca​

See the documentation for this setting.

ssl-certificates​

See the documentation for this setting.

ssl-parameters​

See the documentation for this setting.

containerd-registry​

See the documentation for this setting.

Expiration of kubeconfig Tokens in Rancher 2.8.x​

In Rancher 2.8.x, the default value of the kubeconfig-default-token-ttl-minutes setting is 30 days.

A side effect of using this default value is the expiration of authentication tokens embedded in kubeconfigs that Rancher uses to provision guest Kubernetes clusters on Harvester. When such tokens expire, Rancher loses the ability to perform management operations for the corresponding Rancher-managed guest Kubernetes clusters. Issue #44912 tracks the issue described in this article.

Workaround​

You can patch the expired Harvester cloud credentials to use a new authentication token.

1. Identify the expired cloud credentials and which Harvester cluster is affected by them.

   

2. Download a new kubeconfig file for the affected Harvester cluster.

   

3. Patch the cloud credentials. The cloud credential is stored as a secret in cattle-global-data namespace, and can be replaced with the new kubeconfig file. Ensure that the environment variable KUBECONFIG_FILE contains the path to the new kubeconfig file.

   #!/bin/sh
   CLOUD_CREDENTIAL_ID=$1  # .metadata.name of the cloud credential
   KUBECONFIG_FILE=$2      # path to the downloaded kubeconfig file
   
   kubeconfig="$(base64 -w 0 "${KUBECONFIG_FILE}")"
   
   patch_file=$(mktemp)
   
   cat > ${patch_file} <<EOF
   data:
     harvestercredentialConfig-kubeconfigContent: $kubeconfig
   EOF
   
   kubectl patch secret ${CLOUD_CREDENTIAL_ID} -n cattle-global-data --patch-file ${patch_file} --type merge
   rm ${patch_file}
   

   Copy

   important

   macOS users must use gbase64 to ensure that the -w flag is supported.

Expiration of kubeconfig Tokens in Rancher 2.9.3​

In Rancher 2.9.3 the Rancher UI displays a warning when a Harvester cloud credential or a related cluster contains an expired token. You can renew the token on the Cloud Credentials screen by selecting ⋮ > Renew, or the Clusters screen by selecting ⋮ > Renew Cloud Credential

Expiration of kubeconfig Tokens in Rancher 2.9.4 and later versions​

As previous versions, however now all expired Harvester cloud credentials will display a warning regardless of Rancher upgrade history.