6 posts tagged with "storage"

When working with Longhorn, you may encounter two different VolumeAttachment resources with similar names: Kubernetes VolumeAttachment (storage.k8s.io/v1) and Longhorn VolumeAttachment (longhorn.io/v1beta2). This often causes confusion about why both exist, when each is created, whether they always appear together, and which one to check when troubleshooting. This document clarifies their distinct roles, shows how they work together (and when they don't), and provides real-world examples to help you identify attachment sources and effectively troubleshoot volume attachment issues.

For additional context, see the official documentation at https://longhorn.io/docs/latest/advanced-resources/volumeattachment/

Workflow: How K8s and Longhorn VolumeAttachments Work Together​

When a Pod requires a Longhorn volume, two separate VolumeAttachment resources work together to complete the attachment process. The Kubernetes VolumeAttachment represents the CSI standard attachment request, while the Longhorn VolumeAttachment manages the actual attachment orchestration with ticket-based coordination.

The following diagram illustrates the complete flow from Pod scheduling to successful volume attachment:

┌─────────────────────────────────────────────────────────────┐
│                  Pod Scheduled to Node                      │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Kubernetes Attach/Detach Controller                         │
│ Creates K8s VolumeAttachment                                │
│   APIVersion: storage.k8s.io/v1                             │
│   Spec:                                                     │
│     Attacher: driver.longhorn.io                            │
│     NodeName: worker-node-1                                 │
│     Source.PersistentVolumeName: pvc-xxx                    │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ CSI External-Attacher (Longhorn)                            │
│ Watches K8s VolumeAttachment                                │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn CSI Plugin                                         │
│ Calls ControllerPublishVolume()                             │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Manager API                                        │
│ Creates/Updates Longhorn VolumeAttachment                   │
│   APIVersion: longhorn.io/v1beta2                           │
│   Spec:                                                     │
│     Volume: my-volume                                       │
│     AttachmentTickets:                                      │
│       csi-attacher-<hash>:                                  │
│         ID: <pod-id>                                        │
│         Type: csi-attacher                                  │
│         NodeID: worker-node-1                               │
│         Parameters: {...}                                   │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn VolumeAttachment Controller                        │
│ 1. Evaluates all attachment tickets                         │
│ 2. Selects appropriate ticket to satisfy                    │
│ 3. Updates Volume.Spec.NodeID = worker-node-1               │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Volume Controller                                  │
│ Performs actual volume attachment operation                 │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn VolumeAttachment Controller                        │
│ Updates ticket status: Satisfied = true                     │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn CSI Plugin                                         │
│ Returns attach success to external-attacher                 │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ CSI External-Attacher                                       │
│ Updates K8s VolumeAttachment.Status.Attached = true         │
└─────────────────────────────────────────────────────────────┘

The resulting Longhorn VolumeAttachment YAML:

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  name: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  namespace: longhorn-system
  labels:
    longhornvolume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
spec:
  volume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  attachmentTickets:
    # This CSI ticket was triggered by K8s VolumeAttachment (Pod binding)
    csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec:
      id: csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec
      type: csi-attacher
      nodeID: harvester-node-1
      parameters:
        disableFrontend: "false"
status:
  attachmentTicketStatuses:
    csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec:
      id: csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec
      satisfied: true  # Volume successfully attached
      conditions:
      - type: Satisfied
        status: "True"

Notice the csi-attacher ticket type - this confirms the attachment was triggered by Kubernetes VolumeAttachment through the CSI flow, not by Longhorn internal operations.

Trigger Points​

Understanding when each VolumeAttachment is created or modified is crucial for troubleshooting attachment issues:

1. K8s VolumeAttachment Creation: Triggered when Pod is scheduled to a node requiring a PVC

   • Managed by Kubernetes Attach/Detach (AD) Controller
   • One VolumeAttachment per PV-node combination
   • Represents Kubernetes' intent to attach the volume

2. Longhorn VolumeAttachment Ticket Addition: Triggered by various Longhorn components based on operation needs:

   • CSIAttacher - when CSI ControllerPublishVolume is called
   • SnapshotController - when creating snapshots of volumes
   • BackupController - when backing up volumes
   • LonghornAPI - when users manually attach volumes via Longhorn UI
   • VolumeCloneController - when managing source volume during clone
   • VolumeRestoreController - when restoring data from backups
   • VolumeExpansionController - when expanding volume size
   • ShareManagerController - for RWX volume sharing
   • SalvageController - for volume salvage operations

Attachment Ticket Priority and Coordination​

When multiple operations require volume attachment simultaneously, Longhorn uses a ticket-based priority system to coordinate access intelligently. This ensures critical operations take precedence while allowing background tasks to coexist when possible.

How Priority Works​

Each ticket type has an assigned priority level that determines selection order when the volume is detached:

• Priority 2000 (Highest):
  • VolumeRestoreController
  • VolumeExpansionController
• Priority 1000:
  • LonghornAPI
• Priority 900:
  • CSIAttacher
  • ShareManagerController
  • SalvageController
• Priority 800 (Lowest):
  • BackupController
  • SnapshotController
  • VolumeCloneController
  • VolumeEvictionController

When the volume is detached, the ticket with the highest priority is selected for attachment. If multiple tickets share the same priority, the first one (sorted by ID) is chosen.

Interruption Mechanism​

Priority levels alone don't tell the complete story. Longhorn also implements an interruption mechanism to handle cases where request arrives while the volume is already attached to a different node.

Interruptible operations (can be interrupted):

• BackupController
• SnapshotController
• VolumeCloneController - clone operations, but only when the volume is in VolumeCloneStateCopyCompletedAwaitingHealthy state

Workload operations (can trigger interruption):

• CSIAttacher - Pod workloads requiring the volume on a different node
• LonghornAPI - manual attachment requests via UI/API
• ShareManagerController - RWX volume sharing operations

The interruption only occurs when:

1. The volume's currently attached node has only interruptible tickets
2. A different node has a workload ticket requesting the volume

This design ensures background operations never block workload rescheduling, while protecting active workloads from being interrupted by other background tasks.

Real-World Scenarios​

Scenario 1: Backup During Active Pod Usage

• Pod is running on node-A with a CSIAttacher ticket
• BackupController creates a ticket for node-A (same node)
• Both tickets coexist peacefully - backup runs alongside the Pod
• CSI attachment and backup execution use the engine on the same node, avoiding a node transition.

Scenario 2: Backup Interrupted by Pod Workload

• BackupController is running on node-A (only ticket present)
• A Pod requiring this volume is scheduled to node-B, CSIAttacher creates a ticket for node-B
• VolumeAttachment Controller detects: interruptible ticket on node-A, workload ticket on node-B
• Volume detaches from node-A (backup interrupted), attaches to node-B (csi attacher)
• Backup will retry later automatically

Scenario 3: Detached Volume Snapshot

• Volume is detached, SnapshotController creates a ticket
• Volume attaches temporarily for snapshot creation
• After snapshot completes, ticket is removed
• Volume auto-detaches if no other tickets exist

Usage Examples​

The following examples demonstrate how VolumeAttachment resources behave in common scenarios. Each example shows the complete YAML resource state at different stages, helping you understand what to look for when troubleshooting or monitoring Longhorn operations.

Example 1: VolumeSnapshot Creation (Longhorn VolumeAttachment Only)​

VolumeSnapshot operations use only Longhorn VolumeAttachment without involving Kubernetes VolumeAttachment. This demonstrates that Longhorn VolumeAttachment can operate independently for internal operations.

┌─────────────────────────────────────────────────────────────┐
│           User Creates VolumeSnapshot via kubectl           │
│           kubectl apply -f volumesnapshot.yaml              │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Snapshot Controller                                │
│ Detects new VolumeSnapshot resource                         │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Snapshot Controller Checks Volume State                     │
│ If Volume is detached → needs attachment for snapshot       │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Snapshot Controller Creates Attachment Ticket               │
│ Updates Longhorn VolumeAttachment:                          │
│   AttachmentTickets:                                        │
│     snapshot-<snapshot-name>:                               │
│       Type: snapshot-controller                             │
│       NodeID: <volume-owner-node>                           │
│       Parameters: {disableFrontend: "false"}                │
│                                                             │
│ ❌ No K8s VolumeAttachment created                          │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn VolumeAttachment Controller                        │
│ Selects snapshot ticket → Updates Volume.Spec.NodeID        │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Volume Controller                                  │
│ Attaches volume → Starts Engine                             │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Snapshot Controller                                         │
│ Engine running → Creates snapshot via Engine API            │
│ Snapshot complete → Removes attachment ticket               │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Volume May Auto-Detach (if no other tickets exist)          │
└─────────────────────────────────────────────────────────────┘

The Longhorn VolumeAttachment YAML during snapshot creation:

During Snapshot Creation (ticket exists):

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  name: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  namespace: longhorn-system
  generation: 30
spec:
  volume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  attachmentTickets:
    # Temporary ticket created by Snapshot Controller
    snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a:
      id: snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a
      type: snapshot-controller
      nodeID: harvester-node-1
      parameters:
        disableFrontend: any
status:
  attachmentTicketStatuses:
    snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a:
      id: snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a
      satisfied: false  # Snapshot in progress
      conditions:
      - type: Satisfied
        status: "False"

After Snapshot Completes (ticket removed):

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  name: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  namespace: longhorn-system
  generation: 31  # Incremented after ticket removal
spec:
  volume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  attachmentTickets: {}  # Ticket removed after snapshot completes
status:
  attachmentTicketStatuses: {}

Key Observations:

• The snapshot-controller ticket type clearly identifies this as a Longhorn internal operation
• Unlike csi-attacher tickets (triggered by K8s), this ticket is created purely by Longhorn
• The ticket is temporary - it appears during snapshot creation and disappears when complete
• No corresponding Kubernetes VolumeAttachment exists for this operation

Example 2: VM Migration​

During VM migration, Harvester has two virt-launcher pods for the same VM: the original pod on the source node and a new pod on the target node. This multi-attach capability is enabled for RWX (ReadWriteMany) block mode volumes when the StorageClass has migratable: true parameter, which allows Longhorn to support live VM migration. In the following example, we migrate a VM from harvester-node-2 to harvester-node-0.

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  creationTimestamp: "2025-12-10T04:19:42Z"
  finalizers:
  - longhorn.io
  generation: 3
  labels:
    longhornvolume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  namespace: longhorn-system
  ownerReferences:
  - apiVersion: longhorn.io/v1beta2
    kind: Volume
    name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
    uid: 7cd2ed46-194f-4528-83f7-bbaa5945e7e3
  resourceVersion: "2736440"
  uid: b2492681-8fcb-4330-9ec6-496afa93e96b
spec:
  attachmentTickets:
    csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b:
      generation: 0
      id: csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b
      nodeID: harvester-node-2
      parameters:
        disableFrontend: "false"
        lastAttachedBy: ""
      type: csi-attacher
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      nodeID: harvester-node-0
      parameters:
        disableFrontend: "false"
        lastAttachedBy: ""
      type: csi-attacher
  volume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
status:
  attachmentTicketStatuses:
    csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b:
      conditions:
      - lastProbeTime: ""
        lastTransitionTime: "2025-12-10T04:19:49Z"
        message: ""
        reason: ""
        status: "True"
        type: Satisfied
      generation: 0
      id: csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b
      satisfied: true
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      conditions:
      - lastProbeTime: ""
        lastTransitionTime: "2025-12-10T04:21:00Z"
        message: The migrating attachment ticket is satisfied
        reason: ""
        status: "True"
        type: Satisfied
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      satisfied: true

After Migration Completes (ticket removed):

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  creationTimestamp: "2025-12-10T04:19:42Z"
  finalizers:
  - longhorn.io
  generation: 4
  labels:
    longhornvolume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  namespace: longhorn-system
  ownerReferences:
  - apiVersion: longhorn.io/v1beta2
    kind: Volume
    name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
    uid: 7cd2ed46-194f-4528-83f7-bbaa5945e7e3
  resourceVersion: "2736824"
  uid: b2492681-8fcb-4330-9ec6-496afa93e96b
spec:
  attachmentTickets:
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      nodeID: harvester-node-0
      parameters:
        disableFrontend: "false"
        lastAttachedBy: ""
      type: csi-attacher
  volume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
status:
  attachmentTicketStatuses:
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      conditions:
      - lastProbeTime: ""
        lastTransitionTime: "2025-12-10T04:21:00Z"
        message: ""
        reason: ""
        status: "True"
        type: Satisfied
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      satisfied: true

Key Observations:

• Two CSI attachment tickets coexist: One pointing to the source node (harvester-node-2) and another to the target node (harvester-node-0)
• Both tickets are csi-attacher type: Indicating they were both triggered by Kubernetes VolumeAttachment through the CSI flow
• Both tickets have satisfied: true status: This demonstrates Longhorn's support for attaching the same volume to multiple nodes simultaneously (RWX-like behavior for migration)
• Target node ticket has special message: "The migrating attachment ticket is satisfied" explicitly identifies this as a migration scenario
• Multi-attach is temporary: This dual-attachment state only exists during VM migration; the source node's ticket will be removed after migration completes

Summary​

Longhorn uses two different VolumeAttachment resources for different purposes:

Kubernetes VolumeAttachment (storage.k8s.io/v1) follows the standard CSI specification and is created only when Pods are scheduled to nodes. It represents Kubernetes' official attachment intent and is managed by K8s Attach/Detach Controller and CSI External-Attacher.

Longhorn VolumeAttachment (longhorn.io/v1beta2) extends beyond CSI to support Longhorn's advanced features. It's created for multiple scenarios, including Pod workloads, snapshots, backups, clones, and manual operations. It uses a ticket-based system to coordinate concurrent attachment requests and is managed collaboratively by multiple Longhorn controllers.

Why both are needed: K8s VolumeAttachment ensures CSI compliance with the Kubernetes ecosystem, while Longhorn VolumeAttachment enables automation for background operations without manual intervention. Importantly, not all Longhorn operations trigger K8s VolumeAttachment—for example, creating a VolumeSnapshot only creates a Longhorn VolumeAttachment ticket (snapshot-controller), not a K8s VolumeAttachment.

When troubleshooting: Check both resources. K8s VolumeAttachment shows the CSI standard workflow status, while Longhorn VolumeAttachment shows the complete picture, including all internal operations via attachment tickets. Look at the ticket type to identify the operation source: csi-attacher means triggered by the K8s VolumeAttachment (Pod workload), while snapshot-controller, backup-controller, etc. indicate Longhorn internal operations.

In this Harvester Knowledge Base article, Ivan Sim provided comprehensive guidance on using Velero to perform backup and restore operations for VMs with external storage in Harvester.

However, in certain scenarios, users may require the VM filesystem to be quiesced during Velero backup creation to prevent data corruption, especially when the VM is experiencing heavy I/O operations.

This article describes how to customize Velero Backup Hooks to implement filesystem freeze during Velero backup processing, ensuring data consistency in the backup content.

Background Knowledge​

KubeVirt's virt-freezer provides a mechanism to freeze and thaw guest filesystems. This capability can be leveraged to ensure filesystem consistency during VM backups. However, certain prerequisites must be met for filesystem freeze/thaw operations to function properly:

Prerequisites for Filesystem Freeze​

• QEMU Guest Agent must be enabled in the guest VM
  • Verify this by checking if the VMI has AgentConnected in its status
• Guest VM must be properly configured for related libvirt commands
  • When virt-freezer is triggered, KubeVirt communicates with the QEMU Guest Agent via libvirt commands such as guest-fsfreeze-freeze
  • The guest agent translates these commands to OS-specific calls:
    • Linux systems: Uses fsfreeze syscalls
    • Windows systems: Uses VSS (Volume Shadow Copy Service) APIs

Common Configuration Challenges​

Based on Harvester project experience, some guest operating systems require additional configuration:

• Linux distributions (e.g., RHEL, SLE Micro): May lack sufficient permissions for filesystem freeze operations by default, requiring custom policies
• Windows guests: Require the VSS service to be enabled for filesystem freeze functionality

Important: Filesystem freeze/thaw functionality depends on guest VM configuration, which is outside Harvester's control. Users are responsible for ensuring compatibility before implementing Velero backup hooks with filesystem freeze.

Verifying Filesystem Freeze Compatibility​

To confirm that your VM supports filesystem freeze operations:

1. Access the virtual machine's virt-launcher compute container:

   POD=$(kubectl get pods -n <VM Namespace> \
     -l vm.kubevirt.io/name=<VM Name> \
     -o jsonpath='{.items[0].metadata.name}')
   kubectl exec -it $POD -n default -c compute -- bash
   

   Copy

2. Test filesystem freeze using the virt-freezer application available in the compute container:

   virt-freezer --freeze --namespace <VM namespace> --name <VM name>
   

   Copy

3. Critical: Always verify the freeze operation result and thaw the VM filesystems before performing any other operations

Prerequisites​

All preparation steps outlined in External CSI Storage Backup and Restore With Velero are mandatory, including:

• Harvester installation and configuration
• Velero installation and setup
• S3-compatible storage configuration
• Proper networking and permissions

Implementing Filesystem Freeze Hooks for VM Backup Consistency​

Velero supports pre and post backup hooks that can be integrated with KubeVirt's virt-freezer to ensure filesystem consistency during VM backups.

Configuring VM Template Annotations​

For all VMs requiring data consistency, add the following annotations to the VM template:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-nfs
  namespace: demo
spec:
  template:
    metadata:
      annotations:
        # These annotations will be applied to the virt-launcher pod
        pre.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--freeze", "--namespace", "<VM Namespace>", "--name", "<VM Name>"']'
        pre.hook.backup.velero.io/container: compute
        pre.hook.backup.velero.io/on-error: Fail
        pre.hook.backup.velero.io/timeout: 30s
        
        post.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--unfreeze", "--namespace", "<VM Namespace>", "--name", "<VM Name>"]'
        post.hook.backup.velero.io/container: compute
        post.hook.backup.velero.io/timeout: 30s
    spec:
      # ...rest of VM spec...

These annotations will be propagated to the related virt-launcher pod and instruct Velero to:

• Freeze the VM filesystem before backup creation begins
• Thaw the VM filesystem after backup completion

Important: Replace <VM Namespace> and <VM Name> with the actual namespace and name of your VM.

Creating a Velero Backup with Filesystem Freeze​

After applying the Velero pre/post hook annotations to the VM manifest, follow the backup procedures described in External CSI Storage Backup and Restore With Velero.

Verifying Successful Hook Execution​

If the guest VM is configured correctly, the Velero backup will complete successfully with HooksAttempted indicating successful hook execution.

Check the backup status using:

velero backup describe [Backup Name] --details

Example output showing successful hook execution:

Name:         demo
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.33.3+rke2r1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=33

Phase:  Completed


Namespaces:
  Included:  demo
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          true
Data Mover:                  velero

....

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots:
    demo/vm-nfs-disk-0-au2ej:
      Data Movement:
        Operation ID: du-be5417aa-498e-4b93-b59f-e6498f95a6df.d7f97dab-3bb1-41e189381
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 5368709120
        Result: succeeded

  Pod Volume Backups: <none included>

HooksAttempted:  2
HooksFailed:     0

The output shows that Velero pre/post backup hooks completed successfully. In this case, the hooks are connected to guest VM filesystem freeze and thaw operations to ensure data consistency.

Restoring the Velero Backup​

Follow the restoration procedures described in External CSI Storage Backup and Restore With Velero to restore the namespace using Velero.

Troubleshooting​

If you encounter issues with filesystem freeze operations:

1. Verify QEMU Guest Agent status in the VMI
2. Check guest OS configuration for filesystem freeze support
3. Review Velero hook logs for specific error messages
4. Test virt-freezer manually as described in the verification section

Conclusion​

Implementing filesystem freeze hooks with Velero ensures data consistency during VM backups by quiescing the filesystem before snapshot creation. This approach is particularly valuable for VMs with high I/O activity or critical data that requires point-in-time consistency guarantees.

Harvester allows you to add disks as data volumes. However, only disks that have a World Wide Name (WWN) are displayed on the UI. This occurs because the Harvester node-disk-manager uses the ID_WWN value from udev to uniquely identify disks. The value may not exist in certain situations, particularly when the disks are connected to certain hardware RAID controllers. In these situations, you can view the disks only if you access the host using SSH and run a command such as cat /proc/partitions.

To allow extra disks without WWNs to be visible to Harvester, perform either of the following workarounds:

Workaround 1: Create a filesystem on the disk​

When you create a filesystem on a disk (for example, using the command mkfs.ext4 /dev/sda), a filesystem UUID is assigned to the disk. Harvester uses this value to identify disks without a WWN.

In Harvester versions earlier than v1.6.0, you can use this workaround for only one extra disk because of a bug in duplicate device checking.

Workaround 2: Add a udev rule for generating fake WWNs​

You can add a udev rule that generates a fake WWN for each extra disk based on the device serial number. Harvester accepts the generated WWNs because the only requirement is a unique ID_WWN value as presented by udev.

A YAML file containing the necessary udev rule must be created in the /oem directory on each host. This process can be automated across the Harvester cluster using a CloudInit Resource.

1. Create a YAML file named fake-scsi-wwn-generator.yaml with the following contents:

   apiVersion: node.harvesterhci.io/v1beta1
   kind: CloudInit
   metadata:
     name: fake-scsi-wwn-generator
   spec:
     matchSelector: {}
     filename: 90_fake_scsi_wwn_generator.yaml
     contents: |
       name: "Add udev rules to generate missing SCSI disk WWNs"
       stages:
         initramfs:
           - files:
               - path: /etc/udev/rules.d/59-fake-scsi-wwn-generator.rules
                 permissions: 420
                 owner: 0
                 group: 0
                 content: |
                   # For anything that looks like a SCSI disk (/dev/sd*),
                   # if it has a serial number, but does _not_ have a WWN,
                   # create a fake WWN based on the serial number.  We need
                   # to set both ID_WWN so Harvester's node-disk-manager
                   # can see the WWN, and ID_WWN_WITH_EXTENSION which is
                   # what 60-persistent-storage.rules uses to generate a
                   # /dev/disk/by-id/wwn-* symlink for the device.
                   ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="sd*[!0-9]", \
                     ENV{ID_SERIAL}=="?*", \
                     ENV{ID_WWN}!="?*", ENV{ID_WWN_WITH_EXTENSION}!="?*", \
                     ENV{ID_WWN}="fake.$env{ID_SERIAL}", \
                     ENV{ID_WWN_WITH_EXTENSION}="fake.$env{ID_SERIAL}"
   

   Copy

2. Apply the file's contents to the cluster by running the command kubectl apply -f fake-scsi-wwn-generator.yaml.

   The file /oem/90_fake_scsi_wwn_generator.yaml is automatically created on all cluster nodes.

3. Reboot all nodes to apply the new udev rule.

Once the rule is applied, you should be able to view and add extra disks that were previously not visible on the Harvester UI.

References​

• Harvester: Issue 7173

Harvester 1.5 introduces support for the provisioning of virtual machine root volumes and data volumes using external Container Storage Interface (CSI) drivers.

This article demonstrates how to use Velero 1.16.0 to perform backup and restore of virtual machines in Harvester.

It goes through commands and manifests to:

• Back up virtual machines in a namespace, their NFS CSI volumes, and associated namespace-scoped configuration
• Export the backup artifacts to an AWS S3 bucket
• Restore to a different namespace on the same cluster
• Restore to a different cluster

Velero is a Kubernetes-native backup and restore tool that enables users to perform scheduled and on-demand backups of virtual machines to external object storage providers such as S3, Azure Blob, or GCS, aligning with enterprise backup and disaster recovery practices.

Harvester Installation​

Refer to the Harvester documentation for installation requirements and options.

The kubeconfig file of the Harvester cluster can be retrieved following the instructions here.

Install and Configure Velero​

Download the Velero CLI.

Set the following shell variables:

BUCKET_NAME=<your-s3-bucket-name>
BUCKET_REGION=<your-s3-bucket-region>
AWS_CREDENTIALS_FILE=<absolute-path-to-your-aws-credentials-file>

Install Velero on the Harvester cluster:

velero install \
  --provider aws \
  --features=EnableCSI \
  --plugins "velero/velero-plugin-for-aws:v1.12.0,quay.io/kubevirt/kubevirt-velero-plugin:v0.7.1" \
  --bucket "${BUCKET_NAME}" \
  --secret-file "${AWS_CREDENTIALS_FILE}" \
  --backup-location-config region="${BUCKET_REGION}" \
  --snapshot-location-config region="${BUCKET_REGION}" \
  --use-node-agent

• In this setup, Velero is configured to:

  • Run in the velero namespace
  • Enable CSI volume snapshot APIs
  • Enable the built-in node agent data movement controllers and pods
  • Use the velero-plugin-for-aws plugin to manage interactions with the S3 object store
  • Use the kubevirt-velero-plugin plugin to backup and restore KubeVirt resources

Confirm that Velero is installed and running:

kubectl -n velero get po

NAME                      READY   STATUS    RESTARTS         AGE
node-agent-875mr          1/1     Running   0                1d
velero-745645565f-5dqgr   1/1     Running   0                1d

Configure the velero CLI to output the backup and restore status of CSI objects:

velero client config set features=EnableCSI

Deploy the NFS CSI and Example Server​

Follow the instructions in the NFS CSI documentation to set up the NFS CSI driver, its storage class, and an example NFS server.

The NFS CSI volume snapshotting capability must also be enabled following the instructions here.

Confirm that the NFS CSI and example server are running:

kubectl get po -A -l 'app in (csi-nfs-node,csi-nfs-controller,nfs-server)'

NAMESPACE     NAME                                  READY   STATUS    RESTARTS    AGE
default       nfs-server-b767db8c8-9ltt4            1/1     Running   0           1d
kube-system   csi-nfs-controller-5bf646f7cc-6vfxn   5/5     Running   0           1d
kube-system   csi-nfs-node-9z6pt                    3/3     Running   0           1d

The default NFS CSI storage class is named nfs-csi:

kubectl get sc nfs-csi

NAME      PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-csi   nfs.csi.k8s.io   Delete          Immediate           true                   14d

Confirm that the default NFS CSI volume snapshot class csi-nfs-snapclass is also installed:

kubectl get volumesnapshotclass csi-nfs-snapclass

NAME                DRIVER           DELETIONPOLICY   AGE
csi-nfs-snapclass   nfs.csi.k8s.io   Delete           14d

Preparing the Virtual Machine and Image​

Create a custom namespace named demo-src:

kubectl create ns demo-src

Follow the instructions in the Image Management documentation to upload the Ubuntu 24.04 raw image from https://cloud-images.ubuntu.com/minimal/releases/noble/ to Harvester.

The storage class of the image must be set to nfs-csi, per the Third-Party Storage Support documentation.

Confirm the virtual machine image is successfully uploaded to Harvester:

Follow the instructions in the third-party storage documentation to create a virtual machine with NFS root and data volumes, using the image uploaded in the previous step.

For NFS CSI snapshot to work, the NFS data volume must have the volumeMode set to Filesystem:

Backup the Source Namespace​

Use the velero CLI to create a backup of the demo-src namespace using Velero's built-in data mover:

BACKUP_NAME=backup-demo-src-`date "+%s"`

velero backup create "${BACKUP_NAME}" \
  --include-namespaces demo-src \
  --snapshot-move-data

This creates a backup of the demo-src namespace containing resources like the virtual machine created earlier, its volumes, secrets and other associated configuration.

Depending on the size of the virtual machine and its volumes, the backup may take a while to complete.

The DataUpload custom resources provide insights into the backup progress:

kubectl -n velero get datauploads -l velero.io/backup-name="${BACKUP_NAME}"

Confirm that the backup completed successfully:

velero backup get "${BACKUP_NAME}"

NAME                         STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
backup-demo-src-1747954979   Completed   0        0          2025-05-22 16:04:46 -0700 PDT   29d       default            <none>

After the backup completes, Velero removes the CSI snapshots from the storage side to free up the snapshot data space.

Restore To A Different Namespace​

This section describes how to restore the backup from the demo-src namespace to a new namespace named demo-dst.

Save the following restore modifier to a local file named modifier-data-volumes.yaml:

cat <<EOF > modifier-data-volumes.yaml
version: v1
resourceModifierRules:
- conditions:
    groupResource: persistentvolumeclaims
    matches:
    - path: /metadata/annotations/harvesterhci.io~1volumeForVirtualMachine
      value: "\"true\""
  patches:
  - operation: remove
    path: /metadata/annotations/harvesterhci.io~1volumeForVirtualMachine
EOF

This restore modifier removes the harvesterhci.io/volumeForVirtualMachine annotation from the virtual machine data volumes to ensure that the restoration do not conflict with the CDI volume import populator.

Create the restore modifier:

kubectl -n velero create cm modifier-data-volumes --from-file=modifier-data-volumes.yaml

Assign the backup name to a shell variable:

BACKUP_NAME=backup-demo-src-1747954979

Start the restore operation:

velero restore create \
  --from-backup "${BACKUP_NAME}" \
  --namespace-mappings "demo-src:demo-dst" \
  --exclude-resources "virtualmachineimages.harvesterhci.io" \
  --resource-modifier-configmap "modifier-data-volumes" \
  --labels "velero.kubevirt.io/clear-mac-address=true,velero.kubevirt.io/generate-new-firmware-uuid=true"

• During the restore:

  • The virtual machine MAC address and firmware UUID are reset to avoid potential conflicts with existing virtual machines.
  • the virtual machine image manifest is excluded because Velero restores the entire state of the virtual machine from the backup.
  • the modifier-data-volumes restore modifier is invoked to modify the virtual machine data volumes metadata to prevent conflicts with the CDI volume import populator.

While the restore operation is still in-progress, the DataDownload custom resources can be used to examine the progress of the operation:

RESTORE_NAME=backup-demo-src-1747954979-20250522164015

kubectl -n velero get datadownload -l velero.io/restore-name="${RESTORE_NAME}"

Confirm that the restore completed successfully:

velero restore get

NAME                                        BACKUP                       STATUS      STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
backup-demo-src-1747954979-20250522164015   backup-demo-src-1747954979   Completed   2025-05-22 16:40:15 -0700 PDT   2025-05-22 16:40:49 -0700 PDT   0        6          2025-05-22 16:40:15 -0700 PDT   <none>

Verify that the virtual machine and its configuration are restored to the new demo-dst namespace:

Restore To A Different Cluster​

This section extends the above scenario to demonstrate the steps to restore the backup to a different Harvester cluster.

On the target cluster, install Velero, and set up the NFS CSI and NFS server following the instructions from the Deploy the NFS CSI and Example Server section.

Once Velero is configured to use the same backup location as the source cluster, it automatically discovers the available backups:

velero backup get

NAME                         STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
backup-demo-src-1747954979   Completed   0        0          2025-05-22 16:04:46 -0700 PDT   29d       default            <none>

Follow the steps in the Restore To A Different Namespace section to restore the backup on the target cluster.

Remove the --namespace-mappings option to set the restored namespace to demo-src on the target cluster.

Confirm that the virtual machine and its configuration are restored to the demo-src namespace:

Select Longhorn Volume Snapshot Class​

To perform Velero backup and restore of virtual machines with Longhorn volumes, label the Longhorn volume snapshot class longhorn as follows:

kubectl label volumesnapshotclass longhorn velero.io/csi-volumesnapshot-class

This helps Velero to find the correct Longhorn snapshot class to use during backup and restore.

Limitations​

Enhancements related to the limitations described in this section are tracked at https://github.com/harvester/harvester/issues/8367.

• By default, Velero only supports resource filtering by resource groups and labels. In order to backup/restore a single instance of virtual machine, custom labels must be applied to the virtual machine, and its virtual machine instance, pod, data volumes, persistent volumes claim, persistent volumes and cloudinit secret resources. It's recommended to backup the entire namespace and perform resource filtering during restore to ensure that backup contains all the dependency resources required by the virtual machine.

• The restoration of virtual machine image is not fully supported yet.

In earlier versions of Harvester (v1.0.3 and prior), Longhorn volumes may get corrupted during the replica rebuilding process (reference: Analysis: Potential Data/Filesystem Corruption). In Harvester v1.1.0 and later versions, the Longhorn team has fixed this issue. This article covers manual steps you can take to scan the VM's filesystem and repair it if needed.

Stop The VM And Backup Volume​

Before you scan the filesystem, it is recommend you back up the volume first. For an example, refer to the following steps to stop the VM and backup the volume.

• Find the target VM.

• Stop the target VM.

The target VM is stopped and the related volumes are detached. Now go to the Longhorn UI to backup this volume.

• Enable Developer Tools & Features (Preferences -> Enable Developer Tools & Features).

• Click the ⋮ button and select Edit Config to edit the config page of the VM.

• Go to the Volumes tab and select Check volume details.

• Click the dropdown menu on the right side and select 'Attach' to attach the volume again.

• Select the attached node.

• Check the volume attached under Volume Details and select Take Snapshot on this volume page.

• Confirm that the snapshot is ready.

Now that you completed the volume backup, you need to scan and repair the root filesystem.

Scanning the root filesystem and repairing​

This section will introduce how to scan the filesystem (e.g., XFS, EXT4) using related tools.

Before scanning, you need to know the filesystem's device/partition.

• Identify the filesystem's device by checking the major and minor numbers of that device.

1. Obtain the major and minor numbers from the listed volume information.

   In the following example, the volume name is pvc-ea7536c0-301f-479e-b2a2-e40ddc864b58.

   harvester-node-0:~ # ls /dev/longhorn/pvc-ea7536c0-301f-479e-b2a2-e40ddc864b58 -al
   brw-rw---- 1 root root 8, 0 Oct 23 14:43 /dev/longhorn/pvc-ea7536c0-301f-479e-b2a2-e40ddc864b58
   

   Copy

   The output indicates that the major and minor numbers are 8:0.

2. Obtain the device name from the output of the lsblk command.

   harvester-node-0:~ # lsblk
   NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
   loop0    7:0    0     3G  1 loop /
   sda      8:0    0    40G  0 disk
   ├─sda1   8:1    0     2M  0 part
   ├─sda2   8:2    0    20M  0 part
   └─sda3   8:3    0    40G  0 part
   

   Copy

   The output indicates that 8:0 are the major and minor numbers of the device named sda. Therefore, /dev/sda is related to the volume named pvc-ea7536c0-301f-479e-b2a2-e40ddc864b58.

• You should now know the filesystem's partition. In the example below, sda3 is the filesystem's partition.
• Use the Filesystem toolbox image to scan and repair.

# docker run -it --rm --privileged registry.opensuse.org/isv/rancher/harvester/toolbox/main/fs-toolbox:latest -- bash

Then we try to scan with this target device.

XFS​

When scanning an XFS filesystem, use the xfs_repair command and specify the problematic partition of the device.

In the following example, /dev/sda3 is the problematic partition.

# xfs_repair -n /dev/sda3

To repair the corrupted partition, run the following command.

# xfs_repair /dev/sda3

EXT4​

When scanning a EXT4 filesystem, use the e2fsck command as follows, where the /dev/sde1 is the problematic partition of the device.

# e2fsck -f /dev/sde1

To repair the corrupted partition, run the following command.

# e2fsck -fp /dev/sde1

After using the 'e2fsck' command, you should also see logs related to scanning and repairing the partition. Scanning and repairing the corrupted partition is successful if there are no errors in these logs.

Detach and Start VM again.​

After the corrupted partition is scanned and repaired, detach the volume and try to start the related VM again.

• Detach the volume from the Longhorn UI.

• Start the related VM again from the Harvester UI.

Your VM should now work normally.

Harvester replicates volumes data across disks in a cluster. Before removing a disk, the user needs to evict replicas on the disk to other disks to preserve the volumes' configured availability. For more information about eviction in Longhorn, please check Evicting Replicas on Disabled Disks or Nodes.

Preparation​

This document describes how to evict Longhorn disks using the kubectl command. Before that, users must ensure the environment is set up correctly. There are two recommended ways to do this:

1. Log in to any management node and switch to root (sudo -i).
2. Download Kubeconfig file and use it locally
   • Install kubectl and yq program manually.
   • Open Harvester GUI, click support at the bottom left of the page and click Download KubeConfig to download the Kubeconfig file.
   • Set the Kubeconfig file's path to KUBECONFIG environment variable. For example, export KUBECONFIG=/path/to/kubeconfig.

Evicting replicas from a disk​

1. List Longhorn nodes (names are identical to Kubernetes nodes):

   kubectl get -n longhorn-system nodes.longhorn.io
   

   Copy

   Sample output:

   NAME    READY   ALLOWSCHEDULING   SCHEDULABLE   AGE
   node1   True    true              True          24d
   node2   True    true              True          24d
   node3   True    true              True          24d
   

   Copy

2. List disks on a node. Assume we want to evict replicas of a disk on node1:

   kubectl get -n longhorn-system nodes.longhorn.io node1 -o yaml | yq e '.spec.disks'
   

   Copy

   Sample output:

   default-disk-ed7af10f5b8356be:
     allowScheduling: true
     evictionRequested: false
     path: /var/lib/harvester/defaultdisk
     storageReserved: 36900254515
     tags: []
   

   Copy

3. Assume disk default-disk-ed7af10f5b8356be is the target we want to evict replicas out of.

   Edit the node:

   kubectl edit -n longhorn-system nodes.longhorn.io node1 
   

   Copy

   Update these two fields and save:

   • spec.disks.<disk_name>.allowScheduling to false
   • spec.disks.<disk_name>.evictionRequested to true

   Sample editing:

   default-disk-ed7af10f5b8356be:
     allowScheduling: false
     evictionRequested: true
     path: /var/lib/harvester/defaultdisk
     storageReserved: 36900254515
     tags: []
   

   Copy

4. Wait for all replicas on the disk to be evicted.

   Get current scheduled replicas on the disk:

   kubectl get -n longhorn-system nodes.longhorn.io node1 -o yaml | yq e '.status.diskStatus.default-disk-ed7af10f5b8356be.scheduledReplica'
   

   Copy

   Sample output:

   pvc-86d3d212-d674-4c64-b69b-4a2eb1df2272-r-7b422db7: 5368709120
   pvc-b06f0b09-f30c-4936-8a2a-425b993dd6cb-r-bb0fa6b3: 2147483648
   pvc-b844bcc6-3b06-4367-a136-3909251cb560-r-08d1ab3c: 53687091200
   pvc-ea6e0dff-f446-4a38-916a-b3bea522f51c-r-193ca5c6: 10737418240
   

   Copy

   Run the command repeatedly, and the output should eventually become an empty map:

   {}
   

   Copy

   This means Longhorn evicts replicas on the disk to other disks.

   note

   If a replica always stays in a disk, please open the Longhorn GUI and check if there is free space on other disks.