One post tagged with "priority class" | The open-source hyperconverged infrastructure solution for a cloud-native world

Configure PriorityClass on Longhorn System Components

July 25, 2023 · 7 min read

Engineer Manager

Harvester v1.2.0 introduces a new enhancement where Longhorn system-managed components in newly-deployed clusters are automatically assigned a system-cluster-critical priority class by default. However, when upgrading your Harvester clusters from previous versions, you may notice that Longhorn system-managed components do not have any priority class set.

This behavior is intentional and aimed at supporting zero-downtime upgrades. Longhorn does not allow changing the priority-class setting when attached volumes exist. For more details, please refer to Setting Priority Class During Longhorn Installation).

This article explains how to manually configure priority classes for Longhorn system-managed components after upgrading your Harvester cluster, ensuring that your Longhorn components have the appropriate priority class assigned and maintaining the stability and performance of your system.

Stop all virtual machines

Stop all virtual machines (VMs) to detach all volumes. Please back up any work before doing this.

Login to a Harvester controller node and become root.
Get all running VMs and write down their namespaces and names:
```
kubectl get vmi -A
```
Alternatively, you can get this information by backing up the Virtual Machine Instance (VMI) manifests with the following command:
```
kubectl get vmi -A -o json > vmi-backup.json
```

Shut down all VMs. Log in to all running VMs and shut them down gracefully (recommended). Or use the following command to send shutdown signals to all VMs:

kubectl get vmi -A -o json | jq -r '.items[] | [.metadata.name, .metadata.namespace] | @tsv' | while IFS=$'\t' read -r name namespace; do
      if [ -z "$name" ]; then
        break
      fi
      echo "Stop ${namespace}/${name}"
      virtctl stop $name -n $namespace
    done

note

You can also stop all VMs from the Harvester UI:

Go to the Virtual Machines page.
For each VM, select ⋮ > Stop.

Ensure there are no running VMs:
Run the command:
```
kubectl get vmi -A
```
The above command must return:
```
No resources found
```

Scale down monitoring pods

Scale down the Prometheus deployment. Run the following command and wait for all Prometheus pods to terminate:

kubectl patch -n cattle-monitoring-system prometheus/rancher-monitoring-prometheus --patch '{"spec": {"replicas": 0}}' --type merge && \
    sleep 5 && \
    kubectl rollout status --watch=true -n cattle-monitoring-system statefulset/prometheus-rancher-monitoring-prometheus

A sample output looks like this:

prometheus.monitoring.coreos.com/rancher-monitoring-prometheus patched
statefulset rolling update complete 0 pods at revision prometheus-rancher-monitoring-prometheus-cbf6bd5f7...

Scale down the AlertManager deployment. Run the following command and wait for all AlertManager pods to terminate:

kubectl patch -n cattle-monitoring-system alertmanager/rancher-monitoring-alertmanager --patch '{"spec": {"replicas": 0}}' --type merge && \
    sleep 5 && \
    kubectl rollout status --watch=true -n cattle-monitoring-system statefulset/alertmanager-rancher-monitoring-alertmanager

A sample output looks like this:

alertmanager.monitoring.coreos.com/rancher-monitoring-alertmanager patched
statefulset rolling update complete 0 pods at revision alertmanager-rancher-monitoring-alertmanager-c8c459dff...

Scale down the Grafana deployment. Run the following command and wait for all Grafana pods to terminate:

kubectl scale --replicas=0 deployment/rancher-monitoring-grafana -n cattle-monitoring-system && \
    sleep 5 && \
    kubectl rollout status --watch=true -n cattle-monitoring-system deployment/rancher-monitoring-grafana

A sample output looks like this:

deployment.apps/rancher-monitoring-grafana scaled
deployment "rancher-monitoring-grafana" successfully rolled out

Scale down vm-import-controller pods

Check if the vm-import-controller addon is enabled and configured with a persistent volume with the following command:

kubectl get pvc -n harvester-system harvester-vm-import-controller

If the above command returns an output like this, you must scale down the vm-import-controller pod. Otherwise, you can skip the following step.

NAME                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
harvester-vm-import-controller   Bound    pvc-eb23e838-4c64-4650-bd8f-ba7075ab0559   200Gi      RWO            harvester-longhorn   2m53s

Scale down the vm-import-controller pods with the following command:

kubectl scale --replicas=0 deployment/harvester-vm-import-controller -n harvester-system && \
    sleep 5 && \
    kubectl rollout status --watch=true -n harvester-system deployment/harvester-vm-import-controller

A sample output looks like this:

deployment.apps/harvester-vm-import-controller scaled
deployment "harvester-vm-import-controller" successfully rolled out

Set the `priority-class` setting

Before applying the priority-class setting, you need to verify all volumes are detached. Run the following command to verify the STATE of each volume is detached:

kubectl get volumes.longhorn.io -A

Verify the output looks like this:

NAMESPACE         NAME                                       STATE      ROBUSTNESS   SCHEDULED   SIZE           NODE   AGE
longhorn-system   pvc-5743fd02-17a3-4403-b0d3-0e9b401cceed   detached   unknown                  5368709120            15d
longhorn-system   pvc-7e389fe8-984c-4049-9ba8-5b797cb17278   detached   unknown                  53687091200           15d
longhorn-system   pvc-8df64e54-ecdb-4d4e-8bab-28d81e316b8b   detached   unknown                  2147483648            15d
longhorn-system   pvc-eb23e838-4c64-4650-bd8f-ba7075ab0559   detached   unknown                  214748364800          11m

Set the priority-class setting with the following command:

kubectl patch -n longhorn-system settings.longhorn.io priority-class --patch '{"value": "system-cluster-critical"}' --type merge

Longhorn system-managed pods will restart and then you need to check if all the system-managed components have a priority class set:

Get the value of the priority class system-cluster-critical:

kubectl get priorityclass system-cluster-critical

Verify the output looks like this:

NAME                      VALUE        GLOBAL-DEFAULT   AGE
system-cluster-critical   2000000000   false            15d

Use the following command to get pods' priority in the longhorn-system namespace:

kubectl get pods -n longhorn-system -o custom-columns="Name":metadata.name,"Priority":.spec.priority

Verify all system-managed components' pods have the correct priority. System-managed components include:
- csi-attacher
- csi-provisioner
- csi-resizer
- csi-snapshotter
- engine-image-ei
- instance-manager-e
- instance-manager-r
- longhorn-csi-plugin

Scale up vm-import-controller pods

If you scale down the vm-import-controller pods, you must scale it up again.

Scale up the vm-import-controller pod. Run the command:

kubectl scale --replicas=1 deployment/harvester-vm-import-controller -n harvester-system && \
    sleep 5 && \
    kubectl rollout status --watch=true -n harvester-system deployment/harvester-vm-import-controller

A sample output looks like this:

deployment.apps/harvester-vm-import-controller scaled
Waiting for deployment "harvester-vm-import-controller" rollout to finish: 0 of 1 updated replicas are available...
deployment "harvester-vm-import-controller" successfully rolled out

Verify vm-import-controller is running using the following command:

kubectl get pods --selector app.kubernetes.io/instance=vm-import-controller -A

A sample output looks like this, the pod's STATUS must be Running:

NAMESPACE          NAME                                              READY   STATUS    RESTARTS   AGE
harvester-system   harvester-vm-import-controller-6bd8f44f55-m9k86   1/1     Running   0          4m53s

Scale up monitoring pods

Scale up the Prometheus deployment. Run the following command and wait for all Prometheus pods to roll out:

kubectl patch -n cattle-monitoring-system prometheus/rancher-monitoring-prometheus --patch '{"spec": {"replicas": 1}}' --type merge && \
    sleep 5 && \
    kubectl rollout status --watch=true -n cattle-monitoring-system statefulset/prometheus-rancher-monitoring-prometheus

A sample output looks like:

prometheus.monitoring.coreos.com/rancher-monitoring-prometheus patched
Waiting for 1 pods to be ready...
statefulset rolling update complete 1 pods at revision prometheus-rancher-monitoring-prometheus-cbf6bd5f7...

Scale down the AlertManager deployment. Run the following command and wait for all AlertManager pods to roll out:

kubectl patch -n cattle-monitoring-system alertmanager/rancher-monitoring-alertmanager --patch '{"spec": {"replicas": 1}}' --type merge && \
    sleep 5 && \
    kubectl rollout status --watch=true -n cattle-monitoring-system statefulset/alertmanager-rancher-monitoring-alertmanager

A sample output looks like this:

alertmanager.monitoring.coreos.com/rancher-monitoring-alertmanager patched
Waiting for 1 pods to be ready...
statefulset rolling update complete 1 pods at revision alertmanager-rancher-monitoring-alertmanager-c8bd4466c...

Scale down the Grafana deployment. Run the following command and wait for all Grafana pods to roll out:

kubectl scale --replicas=1 deployment/rancher-monitoring-grafana -n cattle-monitoring-system && \
    sleep 5 && \
    kubectl rollout status --watch=true -n cattle-monitoring-system deployment/rancher-monitoring-grafana

A sample output looks like this:

deployment.apps/rancher-monitoring-grafana scaled
Waiting for deployment "rancher-monitoring-grafana" rollout to finish: 0 of 1 updated replicas are available...
deployment "rancher-monitoring-grafana" successfully rolled out

Start virtual machines

Start a VM with the command:

virtctl start $name -n $namespace

Replace $name with the VM's name and $namespace with the VM's namespace. You can list all virtual machines with the command:

kubectl get vms -A

note

You can also stop all VMs from the Harvester UI:

Go to the Virtual Machines page.
For each VM, select ⋮ > Start.

Alternatively, you can start all running VMs with the following command:

cat vmi-backup.json | jq -r '.items[] | [.metadata.name, .metadata.namespace] | @tsv' | while IFS=$'\t' read -r name namespace; do
      if [ -z "$name" ]; then
        break
      fi
      echo "Start ${namespace}/${name}"
      virtctl start $name -n $namespace || true
    done

Stop all virtual machines​

note

Scale down monitoring pods​

Scale down vm-import-controller pods​

Set the priority-class setting​

Scale up vm-import-controller pods​

Scale up monitoring pods​

Start virtual machines​

note

Stop all virtual machines

Scale down monitoring pods

Scale down vm-import-controller pods

Set the `priority-class` setting

Scale up vm-import-controller pods

Scale up monitoring pods

Start virtual machines