2 posts tagged with "disk"

Harvester allows you to add disks as data volumes. However, only disks that have a World Wide Name (WWN) are displayed on the UI. This occurs because the Harvester node-disk-manager uses the ID_WWN value from udev to uniquely identify disks. The value may not exist in certain situations, particularly when the disks are connected to certain hardware RAID controllers. In these situations, you can view the disks only if you access the host using SSH and run a command such as cat /proc/partitions.

To allow extra disks without WWNs to be visible to Harvester, perform either of the following workarounds:

Workaround 1: Create a filesystem on the disk​

When you create a filesystem on a disk (for example, using the command mkfs.ext4 /dev/sda), a filesystem UUID is assigned to the disk. Harvester uses this value to identify disks without a WWN.

In Harvester versions earlier than v1.6.0, you can use this workaround for only one extra disk because of a bug in duplicate device checking.

Workaround 2: Add a udev rule for generating fake WWNs​

You can add a udev rule that generates a fake WWN for each extra disk based on the device serial number. Harvester accepts the generated WWNs because the only requirement is a unique ID_WWN value as presented by udev.

A YAML file containing the necessary udev rule must be created in the /oem directory on each host. This process can be automated across the Harvester cluster using a CloudInit Resource.

1. Create a YAML file named fake-scsi-wwn-generator.yaml with the following contents:

   apiVersion: node.harvesterhci.io/v1beta1
   kind: CloudInit
   metadata:
     name: fake-scsi-wwn-generator
   spec:
     matchSelector: {}
     filename: 90_fake_scsi_wwn_generator.yaml
     contents: |
       name: "Add udev rules to generate missing SCSI disk WWNs"
       stages:
         initramfs:
           - files:
               - path: /etc/udev/rules.d/59-fake-scsi-wwn-generator.rules
                 permissions: 420
                 owner: 0
                 group: 0
                 content: |
                   # For anything that looks like a SCSI disk (/dev/sd*),
                   # if it has a serial number, but does _not_ have a WWN,
                   # create a fake WWN based on the serial number.  We need
                   # to set both ID_WWN so Harvester's node-disk-manager
                   # can see the WWN, and ID_WWN_WITH_EXTENSION which is
                   # what 60-persistent-storage.rules uses to generate a
                   # /dev/disk/by-id/wwn-* symlink for the device.
                   ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="sd*[!0-9]", \
                     ENV{ID_SERIAL}=="?*", \
                     ENV{ID_WWN}!="?*", ENV{ID_WWN_WITH_EXTENSION}!="?*", \
                     ENV{ID_WWN}="fake.$env{ID_SERIAL}", \
                     ENV{ID_WWN_WITH_EXTENSION}="fake.$env{ID_SERIAL}"
   

   Copy

2. Apply the file's contents to the cluster by running the command kubectl apply -f fake-scsi-wwn-generator.yaml.

   The file /oem/90_fake_scsi_wwn_generator.yaml is automatically created on all cluster nodes.

3. Reboot all nodes to apply the new udev rule.

Once the rule is applied, you should be able to view and add extra disks that were previously not visible on the Harvester UI.

References​

• Harvester: Issue 7173

Harvester replicates volumes data across disks in a cluster. Before removing a disk, the user needs to evict replicas on the disk to other disks to preserve the volumes' configured availability. For more information about eviction in Longhorn, please check Evicting Replicas on Disabled Disks or Nodes.

Preparation​

This document describes how to evict Longhorn disks using the kubectl command. Before that, users must ensure the environment is set up correctly. There are two recommended ways to do this:

1. Log in to any management node and switch to root (sudo -i).
2. Download Kubeconfig file and use it locally
   • Install kubectl and yq program manually.
   • Open Harvester GUI, click support at the bottom left of the page and click Download KubeConfig to download the Kubeconfig file.
   • Set the Kubeconfig file's path to KUBECONFIG environment variable. For example, export KUBECONFIG=/path/to/kubeconfig.

Evicting replicas from a disk​

1. List Longhorn nodes (names are identical to Kubernetes nodes):

   kubectl get -n longhorn-system nodes.longhorn.io
   

   Copy

   Sample output:

   NAME    READY   ALLOWSCHEDULING   SCHEDULABLE   AGE
   node1   True    true              True          24d
   node2   True    true              True          24d
   node3   True    true              True          24d
   

   Copy

2. List disks on a node. Assume we want to evict replicas of a disk on node1:

   kubectl get -n longhorn-system nodes.longhorn.io node1 -o yaml | yq e '.spec.disks'
   

   Copy

   Sample output:

   default-disk-ed7af10f5b8356be:
     allowScheduling: true
     evictionRequested: false
     path: /var/lib/harvester/defaultdisk
     storageReserved: 36900254515
     tags: []
   

   Copy

3. Assume disk default-disk-ed7af10f5b8356be is the target we want to evict replicas out of.

   Edit the node:

   kubectl edit -n longhorn-system nodes.longhorn.io node1 
   

   Copy

   Update these two fields and save:

   • spec.disks.<disk_name>.allowScheduling to false
   • spec.disks.<disk_name>.evictionRequested to true

   Sample editing:

   default-disk-ed7af10f5b8356be:
     allowScheduling: false
     evictionRequested: true
     path: /var/lib/harvester/defaultdisk
     storageReserved: 36900254515
     tags: []
   

   Copy

4. Wait for all replicas on the disk to be evicted.

   Get current scheduled replicas on the disk:

   kubectl get -n longhorn-system nodes.longhorn.io node1 -o yaml | yq e '.status.diskStatus.default-disk-ed7af10f5b8356be.scheduledReplica'
   

   Copy

   Sample output:

   pvc-86d3d212-d674-4c64-b69b-4a2eb1df2272-r-7b422db7: 5368709120
   pvc-b06f0b09-f30c-4936-8a2a-425b993dd6cb-r-bb0fa6b3: 2147483648
   pvc-b844bcc6-3b06-4367-a136-3909251cb560-r-08d1ab3c: 53687091200
   pvc-ea6e0dff-f446-4a38-916a-b3bea522f51c-r-193ca5c6: 10737418240
   

   Copy

   Run the command repeatedly, and the output should eventually become an empty map:

   {}
   

   Copy

   This means Longhorn evicts replicas on the disk to other disks.

   note

   If a replica always stays in a disk, please open the Longhorn GUI and check if there is free space on other disks.