Skip to main content

2 posts tagged with "disk"

View All Tags

· 3 min read
Tim Serong

Harvester allows you to add disks as data volumes. However, only disks that have a World Wide Name (WWN) are displayed on the UI. This occurs because the Harvester node-disk-manager uses the ID_WWN value from udev to uniquely identify disks. The value may not exist in certain situations, particularly when the disks are connected to certain hardware RAID controllers. In these situations, you can view the disks only if you access the host using SSH and run a command such as cat /proc/partitions.

To allow extra disks without WWNs to be visible to Harvester, perform either of the following workarounds:

Workaround 1: Create a filesystem on the disk

caution

Use this method only if the provisioner of the extra disk is Longhorn V1, which is filesystem-based. This method will not work correctly with LVM and Longhorn V2, which are both block device-based.

When you create a filesystem on a disk (for example, using the command mkfs.ext4 /dev/sda), a filesystem UUID is assigned to the disk. Harvester uses this value to identify disks without a WWN.

In Harvester versions earlier than v1.6.0, you can use this workaround for only one extra disk because of a bug in duplicate device checking.

Workaround 2: Add a udev rule for generating fake WWNs

note

This method works with all of the supported provisioners.

You can add a udev rule that generates a fake WWN for each extra disk based on the device serial number. Harvester accepts the generated WWNs because the only requirement is a unique ID_WWN value as presented by udev.

A YAML file containing the necessary udev rule must be created in the /oem directory on each host. This process can be automated across the Harvester cluster using a CloudInit Resource.

  1. Create a YAML file named fake-scsi-wwn-generator.yaml with the following contents:

    apiVersion: node.harvesterhci.io/v1beta1
    kind: CloudInit
    metadata:
    name: fake-scsi-wwn-generator
    spec:
    matchSelector: {}
    filename: 90_fake_scsi_wwn_generator.yaml
    contents: |
    name: "Add udev rules to generate missing SCSI disk WWNs"
    stages:
    initramfs:
    - files:
    - path: /etc/udev/rules.d/59-fake-scsi-wwn-generator.rules
    permissions: 420
    owner: 0
    group: 0
    content: |
    # For anything that looks like a SCSI disk (/dev/sd*),
    # if it has a serial number, but does _not_ have a WWN,
    # create a fake WWN based on the serial number. We need
    # to set both ID_WWN so Harvester's node-disk-manager
    # can see the WWN, and ID_WWN_WITH_EXTENSION which is
    # what 60-persistent-storage.rules uses to generate a
    # /dev/disk/by-id/wwn-* symlink for the device.
    ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="sd*[!0-9]", \
    ENV{ID_SERIAL}=="?*", \
    ENV{ID_WWN}!="?*", ENV{ID_WWN_WITH_EXTENSION}!="?*", \
    ENV{ID_WWN}="fake.$env{ID_SERIAL}", \
    ENV{ID_WWN_WITH_EXTENSION}="fake.$env{ID_SERIAL}"
  2. Apply the file's contents to the cluster by running the command kubectl apply -f fake-scsi-wwn-generator.yaml.

    The file /oem/90_fake_scsi_wwn_generator.yaml is automatically created on all cluster nodes.

  3. Reboot all nodes to apply the new udev rule.

Once the rule is applied, you should be able to view and add extra disks that were previously not visible on the Harvester UI.

References

· 2 min read
Kiefer Chang

Harvester replicates volumes data across disks in a cluster. Before removing a disk, the user needs to evict replicas on the disk to other disks to preserve the volumes' configured availability. For more information about eviction in Longhorn, please check Evicting Replicas on Disabled Disks or Nodes.

Preparation

This document describes how to evict Longhorn disks using the kubectl command. Before that, users must ensure the environment is set up correctly. There are two recommended ways to do this:

  1. Log in to any management node and switch to root (sudo -i).
  2. Download Kubeconfig file and use it locally
    • Install kubectl and yq program manually.
    • Open Harvester GUI, click support at the bottom left of the page and click Download KubeConfig to download the Kubeconfig file.
    • Set the Kubeconfig file's path to KUBECONFIG environment variable. For example, export KUBECONFIG=/path/to/kubeconfig.

Evicting replicas from a disk

  1. List Longhorn nodes (names are identical to Kubernetes nodes):

    kubectl get -n longhorn-system nodes.longhorn.io

    Sample output:

    NAME    READY   ALLOWSCHEDULING   SCHEDULABLE   AGE
    node1 True true True 24d
    node2 True true True 24d
    node3 True true True 24d
  2. List disks on a node. Assume we want to evict replicas of a disk on node1:

    kubectl get -n longhorn-system nodes.longhorn.io node1 -o yaml | yq e '.spec.disks'

    Sample output:

    default-disk-ed7af10f5b8356be:
    allowScheduling: true
    evictionRequested: false
    path: /var/lib/harvester/defaultdisk
    storageReserved: 36900254515
    tags: []
  3. Assume disk default-disk-ed7af10f5b8356be is the target we want to evict replicas out of.

    Edit the node:

    kubectl edit -n longhorn-system nodes.longhorn.io node1 

    Update these two fields and save:

    • spec.disks.<disk_name>.allowScheduling to false
    • spec.disks.<disk_name>.evictionRequested to true

    Sample editing:

    default-disk-ed7af10f5b8356be:
    allowScheduling: false
    evictionRequested: true
    path: /var/lib/harvester/defaultdisk
    storageReserved: 36900254515
    tags: []
  4. Wait for all replicas on the disk to be evicted.

    Get current scheduled replicas on the disk:

    kubectl get -n longhorn-system nodes.longhorn.io node1 -o yaml | yq e '.status.diskStatus.default-disk-ed7af10f5b8356be.scheduledReplica'

    Sample output:

    pvc-86d3d212-d674-4c64-b69b-4a2eb1df2272-r-7b422db7: 5368709120
    pvc-b06f0b09-f30c-4936-8a2a-425b993dd6cb-r-bb0fa6b3: 2147483648
    pvc-b844bcc6-3b06-4367-a136-3909251cb560-r-08d1ab3c: 53687091200
    pvc-ea6e0dff-f446-4a38-916a-b3bea522f51c-r-193ca5c6: 10737418240

    Run the command repeatedly, and the output should eventually become an empty map:

    {}

    This means Longhorn evicts replicas on the disk to other disks.

    note

    If a replica always stays in a disk, please open the Longhorn GUI and check if there is free space on other disks.