Harvester allows you to add disks as data volumes. However, only disks that have a World Wide Name (WWN) are displayed on the UI. This occurs because the Harvester node-disk-manager uses the ID_WWN value from udev to uniquely identify disks. The value may not exist in certain situations, particularly when the disks are connected to certain hardware RAID controllers. In these situations, you can view the disks only if you access the host using SSH and run a command such as cat /proc/partitions.
To allow extra disks without WWNs to be visible to Harvester, perform either of the following workarounds:
Use this method only if the provisioner of the extra disk is Longhorn V1, which is filesystem-based. This method will not work correctly with LVM and Longhorn V2, which are both block device-based.
When you create a filesystem on a disk (for example, using the command mkfs.ext4 /dev/sda), a filesystem UUID is assigned to the disk. Harvester uses this value to identify disks without a WWN.
In Harvester versions earlier than v1.6.0, you can use this workaround for only one extra disk because of a bug in duplicate device checking.
Workaround 2: Add a udev rule for generating fake WWNs
note
This method works with all of the supported provisioners.
You can add a udev rule that generates a fake WWN for each extra disk based on the device serial number. Harvester accepts the generated WWNs because the only requirement is a unique ID_WWN value as presented by udev.
A YAML file containing the necessary udev rule must be created in the /oem directory on each host. This process can be automated across the Harvester cluster using a CloudInit Resource.
Create a YAML file named fake-scsi-wwn-generator.yaml with the following contents:
apiVersion: node.harvesterhci.io/v1beta1 kind: CloudInit metadata: name: fake-scsi-wwn-generator spec: matchSelector:{} filename: 90_fake_scsi_wwn_generator.yaml contents:| name: "Add udev rules to generate missing SCSI disk WWNs" stages: initramfs: - files: - path: /etc/udev/rules.d/59-fake-scsi-wwn-generator.rules permissions: 420 owner: 0 group: 0 content: | # For anything that looks like a SCSI disk (/dev/sd*), # if it has a serial number, but does _not_ have a WWN, # create a fake WWN based on the serial number. We need # to set both ID_WWN so Harvester's node-disk-manager # can see the WWN, and ID_WWN_WITH_EXTENSION which is # what 60-persistent-storage.rules uses to generate a # /dev/disk/by-id/wwn-* symlink for the device. ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="sd*[!0-9]", \ ENV{ID_SERIAL}=="?*", \ ENV{ID_WWN}!="?*", ENV{ID_WWN_WITH_EXTENSION}!="?*", \ ENV{ID_WWN}="fake.$env{ID_SERIAL}", \ ENV{ID_WWN_WITH_EXTENSION}="fake.$env{ID_SERIAL}"
Apply the file's contents to the cluster by running the command kubectl apply -f fake-scsi-wwn-generator.yaml.
The file /oem/90_fake_scsi_wwn_generator.yaml is automatically created on all cluster nodes.
Reboot all nodes to apply the new udev rule.
Once the rule is applied, you should be able to view and add extra disks that were previously not visible on the Harvester UI.
Harvester 1.5 introduces support for the provisioning of virtual machine root volumes and data volumes using external Container Storage Interface (CSI) drivers.
This article demonstrates how to use Velero 1.16.0 to perform backup and restore of virtual machines in Harvester.
It goes through commands and manifests to:
Back up virtual machines in a namespace, their NFS CSI volumes, and associated namespace-scoped configuration
Export the backup artifacts to an AWS S3 bucket
Restore to a different namespace on the same cluster
Restore to a different cluster
Velero is a Kubernetes-native backup and restore tool that enables users to perform scheduled and on-demand backups of virtual machines to external object storage providers such as S3, Azure Blob, or GCS, aligning with enterprise backup and disaster recovery practices.
note
The commands and manifests used in this article are tested with Harvester 1.5.1.
The CSI NFS driver and Velero configuration and versions used are for demonstration purposes only. Adjust them according to your environment and requirements.
important
The examples provided are intended to backup and restore Linux virtual machine workloads. It is not suitable for backing up guest clusters provisioned via the Harvester Rancher integration.
To backup and restore guest clusters like RKE2, please refer to the distro official documentation.
Confirm the virtual machine image is successfully uploaded to Harvester:
Follow the instructions in the third-party storage documentation to create a virtual machine with NFS root and data volumes, using the image uploaded in the previous step.
For NFS CSI snapshot to work, the NFS data volume must have the volumeMode set to Filesystem:
optional
For testing purposes, once the virtual machine is ready, access it via SSH and add some files to both the root and data volumes.
The data volume needs to be partitioned, with a file system created and mounted before files can be written to it.
This creates a backup of the demo-src namespace containing resources like the virtual machine created earlier, its volumes, secrets and other associated configuration.
Depending on the size of the virtual machine and its volumes, the backup may take a while to complete.
The DataUpload custom resources provide insights into the backup progress:
kubectl -n velero get datauploads -l velero.io/backup-name="${BACKUP_NAME}"
Confirm that the backup completed successfully:
velero backup get "${BACKUP_NAME}"
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR backup-demo-src-1747954979 Completed 0 0 2025-05-22 16:04:46 -0700 PDT 29d default <none>
After the backup completes, Velero removes the CSI snapshots from the storage side to free up the snapshot data space.
tips
The velero backup describe and velero backup logs commands can be used to assess details of the backup including resources included, skipped, and any warnings or errors encountered during the backup process.
This restore modifier removes the harvesterhci.io/volumeForVirtualMachine annotation from the virtual machine data volumes to ensure that the restoration do not conflict with the CDI volume import populator.
Create the restore modifier:
kubectl -n velero create cm modifier-data-volumes --from-file=modifier-data-volumes.yaml
The virtual machine MAC address and firmware UUID are reset to avoid potential conflicts with existing virtual machines.
the virtual machine image manifest is excluded because Velero restores the entire state of the virtual machine from the backup.
the modifier-data-volumes restore modifier is invoked to modify the virtual machine data volumes metadata to prevent conflicts with the CDI volume import populator.
While the restore operation is still in-progress, the DataDownload custom resources can be used to examine the progress of the operation:
RESTORE_NAME=backup-demo-src-1747954979-20250522164015 kubectl -n velero get datadownload -l velero.io/restore-name="${RESTORE_NAME}"
Confirm that the restore completed successfully:
velero restore get
NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR backup-demo-src-1747954979-20250522164015 backup-demo-src-1747954979 Completed 2025-05-22 16:40:15 -0700 PDT 2025-05-22 16:40:49 -0700 PDT 0 6 2025-05-22 16:40:15 -0700 PDT <none>
Verify that the virtual machine and its configuration are restored to the new demo-dst namespace:
note
Velero uses Kopia as its default data mover. This issue describes some of its limitations on advanced file system features such as setuid/gid, hard links, mount points, sockets, xattr, ACLs, etc.
Velero provides the --data-mover option to configure custom data movers to satisfy different use cases. For more information, see the Velero's documentation.
tips
The velero restore describe and velero restore logs commands provide more insights into the restore operation including the resources restored, skipped, and any warnings or errors encountered during the restore process.
This section extends the above scenario to demonstrate the steps to restore the backup to a different Harvester cluster.
On the target cluster, install Velero, and set up the NFS CSI and NFS server following the instructions from the Deploy the NFS CSI and Example Server section.
Once Velero is configured to use the same backup location as the source cluster, it automatically discovers the available backups:
velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR backup-demo-src-1747954979 Completed 0 0 2025-05-22 16:04:46 -0700 PDT 29d default <none>
By default, Velero only supports resource filtering by resource groups and labels. In order to backup/restore a single instance of virtual machine, custom labels must be applied to the virtual machine, and its virtual machine instance, pod, data volumes, persistent volumes claim, persistent volumes and cloudinit secret resources. It's recommended to backup the entire namespace and perform resource filtering during restore to ensure that backup contains all the dependency resources required by the virtual machine.
The restoration of virtual machine image is not fully supported yet.
In earlier versions of Harvester (v1.0.3 and prior), Longhorn volumes may get corrupted during the replica rebuilding process (reference: Analysis: Potential Data/Filesystem Corruption). In Harvester v1.1.0 and later versions, the Longhorn team has fixed this issue. This article covers manual steps you can take to scan the VM's filesystem and repair it if needed.
Before you scan the filesystem, it is recommend you back up the volume first. For an example, refer to the following steps to stop the VM and backup the volume.
Find the target VM.
Stop the target VM.
The target VM is stopped and the related volumes are detached. Now go to the Longhorn UI to backup this volume.
This section will introduce how to scan the filesystem (e.g., XFS, EXT4) using related tools.
Before scanning, you need to know the filesystem's device/partition.
Identify the filesystem's device by checking the major and minor numbers of that device.
Obtain the major and minor numbers from the listed volume information.
In the following example, the volume name is pvc-ea7536c0-301f-479e-b2a2-e40ddc864b58.
harvester-node-0:~ # ls /dev/longhorn/pvc-ea7536c0-301f-479e-b2a2-e40ddc864b58 -al brw-rw---- 1 root root 8, 0 Oct 23 14:43 /dev/longhorn/pvc-ea7536c0-301f-479e-b2a2-e40ddc864b58
The output indicates that the major and minor numbers are 8:0.
Obtain the device name from the output of the lsblk command.
harvester-node-0:~ # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 3G 1 loop / sda 8:0 0 40G 0 disk ├─sda1 8:1 0 2M 0 part ├─sda2 8:2 0 20M 0 part └─sda3 8:3 0 40G 0 part
The output indicates that 8:0 are the major and minor numbers of the device named sda. Therefore, /dev/sda is related to the volume named pvc-ea7536c0-301f-479e-b2a2-e40ddc864b58.
You should now know the filesystem's partition. In the example below, sda3 is the filesystem's partition.
Use the Filesystem toolbox image to scan and repair.
# docker run -it --rm --privileged registry.opensuse.org/isv/rancher/harvester/toolbox/main/fs-toolbox:latest -- bash
When scanning a EXT4 filesystem, use the e2fsck command as follows, where the /dev/sde1 is the problematic partition of the device.
# e2fsck -f /dev/sde1
To repair the corrupted partition, run the following command.
# e2fsck -fp /dev/sde1
After using the 'e2fsck' command, you should also see logs related to scanning and repairing the partition. Scanning and repairing the corrupted partition is successful if there are no errors in these logs.
Harvester replicates volumes data across disks in a cluster. Before removing a disk, the user needs to evict replicas on the disk to other disks to preserve the volumes' configured availability. For more information about eviction in Longhorn, please check Evicting Replicas on Disabled Disks or Nodes.
This document describes how to evict Longhorn disks using the kubectl command. Before that, users must ensure the environment is set up correctly.
There are two recommended ways to do this:
Log in to any management node and switch to root (sudo -i).
Download Kubeconfig file and use it locally
Install kubectl and yq program manually.
Open Harvester GUI, click support at the bottom left of the page and click Download KubeConfig to download the Kubeconfig file.
Set the Kubeconfig file's path to KUBECONFIG environment variable. For example, export KUBECONFIG=/path/to/kubeconfig.