One post tagged with "fc"

This article explains how to install the NetApp Trident CSI driver on a Harvester cluster. Once it's installed, VMs running in Harvester can store their volumes on a NetApp ONTAP storage system.

NetApp storage is added alongside Longhorn; it does not replace it. You can use any combination of these protocols:

• NFS (ontap-nas)
• iSCSI (ontap-san, sanType: iscsi)
• NVMe/TCP (ontap-san, sanType: nvme)
• Fibre Channel (ontap-san, sanType: fcp)

This procedure was tested with Harvester v1.9.0 and Trident 26.06.1 (Helm chart 100.2606.1), using ONTAP 9.16.1. With NFS, iSCSI and NVMe/TCP, the following worked on NetApp-backed volumes: booting VMs, live migration, online volume expansion, and Harvester VM snapshots. Trident also supports Fibre Channel, and the multipath configuration below covers FC LUNs. However, FC was not tested for this article.

Prerequisites​

• A Harvester cluster and its KubeConfig. To download it, open the Harvester UI, click Support in the lower left, then click Download KubeConfig.
• A workstation with kubectl and helm that can reach the Harvester cluster.
• An ONTAP SVM with:
  • A management LIF that the Harvester nodes can reach.
  • Data LIFs for each protocol you plan to use: NFS, iSCSI, NVMe/TCP and/or FC. NVMe/TCP requires the NVMe_oF license on the ONTAP system.
  • An ONTAP login for Trident with the http and ontapi applications enabled. The examples in this article use limitAggregateUsage, which requires a cluster-scoped login with the admin role. In that case, managementLIF is the cluster management LIF. If you use an SVM-scoped vsadmin login instead, set managementLIF to the SVM management LIF and remove limitAggregateUsage from the backends. See User permissions in the Trident documentation.
• For FC only: FC HBAs in every Harvester node that mounts NetApp volumes, and fabric zoning between those HBAs and the SVM's FC LIFs.

Network design​

Trident's node plugin runs in the host network namespace, so the host's routing table decides which NIC carries storage traffic.

How Trident discovers data LIFs differs by protocol:

• iSCSI / NVMe/TCP: Trident ignores dataLIF on SAN backends and logs Specifying data LIF is no longer supported for SAN backends. It asks ONTAP for every SAN LIF in the SVM, and each node logs in to all of them, so multipath works automatically. You only need managementLIF and svm.
• NFS: dataLIF takes a single address. If you leave it out, Trident uses the first NFS LIF it finds in the SVM. Set it explicitly to choose the network that carries NFS traffic.
• FC: traffic runs over the FC fabric, not the IP network, so the rest of this section does not apply. Only the management LIF must be reachable over IP.

Optional: a dedicated storage interface with HostNetworkConfig​

This takes two steps: create a cluster network on the storage NICs, then give each host an IP address on it with a HostNetworkConfig.

Step 1: create the cluster network. A cluster network groups the storage NICs into a bridge on each node. See Create a New Cluster Network for details.

1. In the Harvester UI, go to Networks > ClusterNetworks/Configs and click Create. Name it, for example cn-storage.
2. Click Create Network Config on the new cluster network.
3. On the Node Selector tab, select the nodes that mount NetApp volumes. Select all nodes does not work if the cluster has a witness node. Select nodes by name, or by a label such as netapp-trident=enabled (see the note further down).
4. On the Uplink tab, select the storage NICs, the bond mode, and the MTU. The MTU must match the switch ports and the ONTAP LIFs, for example 9000 for jumbo frames.
5. Click Save.

The switch ports for these NICs must be trunk ports that carry the storage VLAN tagged.

Step 2: give the hosts an IP address on it. On its own, a cluster network carries only VM traffic, because the hosts have no IP address on it. A HostNetworkConfig (Networks > Host Networks) creates a VLAN sub-interface on the cluster network's bridge on each node and gives it an IP address. Traffic to the ONTAP data LIFs on that subnet then leaves through that interface instead of the management network. The ONTAP data LIFs must be on the same VLAN and subnet.

Example with static addresses on VLAN 150:

apiVersion: network.harvesterhci.io/v1beta1
kind: HostNetworkConfig
metadata:
  name: netapp-storage
spec:
  clusterNetwork: cn-storage
  vlanID: 150
  mode: static
  nodeSelector:
    matchLabels:
      netapp-trident: enabled
  ips:
    <node-1>: 192.168.150.11/24
    <node-2>: 192.168.150.12/24

Check the result on a node with ip addr show cn-storage-br.150. Each node's state is reported in the resource's status.nodeStatus.

For iSCSI, you can add a second cluster network (or a second VLAN) with its own HostNetworkConfig and LIFs, so multipath has two independent paths.

1. Configure multipathd (iSCSI and FC)​

iSCSI and FC LUNs need multipathd, which is not enabled on Harvester by default. Both protocols present NetApp LUNs with vendor NETAPP and product LUN, so the same configuration covers them. The configuration below lets multipathd claim only NetApp LUNs. Every other device is blacklisted, so Longhorn and the local disks are never touched. Harvester's own /etc/multipath/conf.d/99-longhorn.conf is left as is.

The following CloudInit resource writes /etc/multipath/conf.d/98-netapp.conf and enables multipathd on every node. It is re-applied on every boot. Save it as 01-cloudinit-netapp-multipathd.yaml:

# Persists across reboots: writes /etc/multipath/conf.d/98-netapp.conf (multipath ONLY claims NETAPP LUNs)
apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: netapp-multipathd
spec:
  matchSelector: {}
  filename: 98_netapp_multipathd.yaml
  contents: |
    name: "enable multipathd for NetApp Trident iSCSI/FC (NETAPP LUNs only)"
    stages:
      initramfs:
        - files:
            - path: /etc/multipath/conf.d/98-netapp.conf
              permissions: 0644
              owner: 0
              group: 0
              content: |
                defaults {
                    user_friendly_names yes
                    find_multipaths no
                    # lets libmpathpersist (qemu-pr-helper) track SCSI-3 PR keys across paths, for shared disks (e.g. WSFC)
                    reservation_key file
                    # don't create kpartx maps for partitions that guests create on the LUNs
                    skip_kpartx yes
                }
                blacklist {
                    device {
                        vendor ".*"
                        product ".*"
                    }
                }
                blacklist_exceptions {
                    device {
                        vendor "NETAPP"
                        product "LUN.*"
                    }
                }
      network:
        - name: "enable and start multipathd"
          systemctl:
            enable:
              - multipathd
            start:
              - multipathd

kubectl apply -f 01-cloudinit-netapp-multipathd.yaml

The CloudInit resource takes effect the next time each node boots. To apply it now without rebooting, run the following on each node as root. It writes the same file and starts multipathd, and stops at the first failure:

set -e
mkdir -p /etc/multipath/conf.d
cat > /etc/multipath/conf.d/98-netapp.conf <<'EOF'
defaults {
    user_friendly_names yes
    find_multipaths no
    reservation_key file
    skip_kpartx yes
}
blacklist {
    device {
        vendor ".*"
        product ".*"
    }
}
blacklist_exceptions {
    device {
        vendor "NETAPP"
        product "LUN.*"
    }
}
EOF
systemctl enable --now multipathd
multipath -ll   # must not list any local disk

Check that every node has a unique iSCSI initiator name. Nodes cloned from the same image can end up with the same name in /etc/iscsi/initiatorname.iscsi. ONTAP then treats those nodes as a single host. If you change a node's IQN, restart that node's trident-node-linux pod afterwards. Trident caches the IQN, and with a stale one volume staging fails with no devices present yet.

2. Install the Trident operator​

helm repo add netapp-trident https://netapp.github.io/trident-helm-chart
helm repo update
helm search repo netapp-trident --versions | head

helm upgrade --install trident netapp-trident/trident-operator --version 100.2606.1 \
  --namespace trident --create-namespace --wait

Wait until the trident-controller pod and one trident-node-linux pod per node are Running:

kubectl -n trident get pods -o wide

kubectl label node <node-1> <node-2> netapp-trident=enabled --overwrite

helm upgrade --install trident netapp-trident/trident-operator --version 100.2606.1 \
  --namespace trident --create-namespace --wait \
  --set tridentNodePluginNodeSelector.netapp-trident=enabled

3. Create the ONTAP credentials Secret​

kubectl -n trident create secret generic netapp-creds \
  --from-literal=username=<ontap user> \
  --from-literal=password='<ontap password>'

4. Create the backends​

Create one TridentBackendConfig per protocol you use; you can drop any you don't need.

The examples below use these settings:

• Thin provisioning: spaceReserve: none and snapshotReserve: "0".
• limitAggregateUsage: "80%": Trident stops creating new volumes once the aggregate passes 80% used.

If several Kubernetes or Harvester clusters share one SVM, give each cluster a unique backendName and set storagePrefix (for example storagePrefix: hv1_). That keeps volume names from colliding and makes each cluster's volumes easy to identify in ONTAP.

For Fibre Channel, copy the iSCSI backend below, then change name and backendName (for example to netapp-fc) and set sanType: fcp. Then add a matching StorageClass in step 5 with storagePools: "netapp-fc:.*". FC was not tested for this article.

Save as 03-backends.yaml, replacing the <...> values:

# NFS
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-nas
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-nas
  backendName: netapp-nas
  managementLIF: <management LIF IP>
  dataLIF: <NFS data LIF IP>          # optional; defaults to the first NFS LIF in the SVM
  svm: <svm name>
  autoExportPolicy: true
  autoExportCIDRs: ["<storage network CIDR>"]
  nfsMountOptions: "nfsvers=4.1"
  limitAggregateUsage: "80%"
  credentials:
    name: netapp-creds
  defaults:
    spaceReserve: none
    snapshotReserve: "0"
---
# iSCSI (all iSCSI LIFs in the SVM are discovered; no dataLIF)
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-iscsi
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-san
  sanType: iscsi
  backendName: netapp-iscsi
  managementLIF: <management LIF IP>
  svm: <svm name>
  limitAggregateUsage: "80%"
  credentials:
    name: netapp-creds
  defaults:
    spaceReserve: none
    snapshotReserve: "0"
---
# NVMe/TCP (all NVMe LIFs in the SVM are discovered; no dataLIF)
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: netapp-nvme
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-san
  sanType: nvme
  backendName: netapp-nvme
  managementLIF: <management LIF IP>
  svm: <svm name>
  limitAggregateUsage: "80%"
  credentials:
    name: netapp-creds
  defaults:
    spaceReserve: none
    snapshotReserve: "0"

kubectl apply -f 03-backends.yaml
kubectl -n trident get tridentbackendconfigs

Every backend must report PHASE Bound and STATUS Success. If one doesn't, kubectl -n trident describe tridentbackendconfig <name> shows the error.

5. Create the StorageClasses and VolumeSnapshotClass​

Trident StorageClasses have no sanType parameter. A StorageClass with only backendType: ontap-san can place a volume on any SAN backend: iSCSI, NVMe/TCP or FC. To prevent that, pin each class to its backend with storagePools: "<backendName>:.*".

Save as 04-storageclasses.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-nfs
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-nas"
  storagePools: "netapp-nas:.*"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-iscsi
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-san"
  storagePools: "netapp-iscsi:.*"
  fsType: "ext4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-nvme
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-san"
  storagePools: "netapp-nvme:.*"
  fsType: "ext4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: trident-snapshotclass
driver: csi.trident.netapp.io
deletionPolicy: Delete

kubectl apply -f 04-storageclasses.yaml

Live migration requires ReadWriteMany volumes. CDI already knows Trident's defaults:

• ontap-san (iSCSI, NVMe/TCP and FC): ReadWriteMany with volumeMode: Block.
• ontap-nas: ReadWriteMany with volumeMode: Filesystem.

So these StorageClasses don't need the cdi.harvesterhci.io/storageProfileVolumeModeAccessModes annotation.

6. Register Trident for Harvester VM snapshots​

Harvester needs to know which VolumeSnapshotClass to use for Trident volumes.

1. In the Harvester UI, go to Advanced > Settings, find csi-driver-config, and select ⋮ > Edit Setting.
2. Click Add, then fill in the fields:
   • Provisioner: csi.trident.netapp.io
   • Volume Snapshot Class Name: trident-snapshotclass
3. Save.

Leave Backup Volume Snapshot Class Name empty. In Harvester v1.9, VM backups only work with Longhorn volumes. VM snapshots of Trident volumes work. See Third-Party Storage Support.

The resulting entry in the setting's JSON value is:

"csi.trident.netapp.io": {
  "volumeSnapshotClassName": "trident-snapshotclass",
  "backupVolumeSnapshotClassName": ""
}

7. Use it​

The new StorageClasses now appear when you create a volume, a VM image or a VM in the Harvester UI. A quick check:

1. Create a VM whose root disk uses netapp-iscsi, netapp-nvme or netapp-nfs.
2. Start the VM, live-migrate it to another node, and take a VM snapshot.
3. On a node running the VM, multipath -ll (iSCSI) or ls /sys/class/nvme-subsystem (NVMe) shows the NetApp devices.

Troubleshooting​

• Backends go offline after the ONTAP system was unreachable. If they don't recover on their own, restart the controller: kubectl -n trident rollout restart deploy/trident-controller.
• Deleted volumes still use space on ONTAP. ONTAP keeps deleted volumes in the volume recovery queue for 12 hours. While a deleted FlexClone (such as a VM disk cloned from an image) sits in the queue, its parent snapshot stays busy, so the parent volume and its VolumeSnapshot can't be deleted. To free them immediately, run on ONTAP: set -privilege advanced; volume recovery-queue purge-all -vserver <svm name>.
• multipath -ll lists local disks. multipathd was started without the NetApp-only configuration. Put 98-netapp.conf in place, flush the unused maps with multipath -F, and restart multipathd.
• iSCSI volumes fail to stage with no devices present yet. Check for duplicate initiator names across nodes, and for a stale IQN in Trident (see step 1).

References​

• NetApp Trident documentation
• Deploy the Trident operator using Helm
• ONTAP SAN driver and ONTAP NAS driver
• Harvester third-party storage support