This article explains how to install the NetApp Trident CSI driver on a Harvester cluster. Once it's installed, VMs running in Harvester can store their volumes on a NetApp ONTAP storage system.
NetApp storage is added alongside Longhorn; it does not replace it. You can use any combination of these protocols:
- NFS (
ontap-nas) - iSCSI (
ontap-san,sanType: iscsi) - NVMe/TCP (
ontap-san,sanType: nvme) - Fibre Channel (
ontap-san,sanType: fcp)
This procedure was tested with Harvester v1.9.0 and Trident 26.06.1 (Helm chart 100.2606.1), using ONTAP 9.16.1. With NFS, iSCSI and NVMe/TCP, the following worked on NetApp-backed volumes: booting VMs, live migration, online volume expansion, and Harvester VM snapshots. Trident also supports Fibre Channel, and the multipath configuration below covers FC LUNs. However, FC was not tested for this article.
note
This article replaces an earlier version (August 2023) written for Harvester v1.2.0 and Trident v23.07. The main changes:
- Multipath is now configured with a Harvester
CloudInitresource, instead of files edited by hand on every node. - Backends are now
TridentBackendConfigresources, sotridentctlis no longer needed. - NFS and NVMe/TCP are now covered.
- The extra
ClusterRolefor PSA (NetApp/trident#839) was not needed in testing. - Third-party volumes can now be used as VM boot volumes.
Prerequisites
- A Harvester cluster and its KubeConfig. To download it, open the Harvester UI, click Support in the lower left, then click Download KubeConfig.
- A workstation with
kubectlandhelmthat can reach the Harvester cluster. - An ONTAP SVM with:
- A management LIF that the Harvester nodes can reach.
- Data LIFs for each protocol you plan to use: NFS, iSCSI, NVMe/TCP and/or FC. NVMe/TCP requires the
NVMe_oFlicense on the ONTAP system. - An ONTAP login for Trident with the
httpandontapiapplications enabled. The examples in this article uselimitAggregateUsage, which requires a cluster-scoped login with theadminrole. In that case,managementLIFis the cluster management LIF. If you use an SVM-scopedvsadminlogin instead, setmanagementLIFto the SVM management LIF and removelimitAggregateUsagefrom the backends. See User permissions in the Trident documentation.
- For FC only: FC HBAs in every Harvester node that mounts NetApp volumes, and fabric zoning between those HBAs and the SVM's FC LIFs.
Network design
Trident's node plugin runs in the host network namespace, so the host's routing table decides which NIC carries storage traffic.
tip
Avoid sending storage traffic over the management network. It's shared with the cluster control plane and the pod network, and adds overhead. For production, use a dedicated cluster network on its own NICs, as described in the next section.
How Trident discovers data LIFs differs by protocol:
- iSCSI / NVMe/TCP: Trident ignores
dataLIFon SAN backends and logsSpecifying data LIF is no longer supported for SAN backends. It asks ONTAP for every SAN LIF in the SVM, and each node logs in to all of them, so multipath works automatically. You only needmanagementLIFandsvm. - NFS:
dataLIFtakes a single address. If you leave it out, Trident uses the first NFS LIF it finds in the SVM. Set it explicitly to choose the network that carries NFS traffic. - FC: traffic runs over the FC fabric, not the IP network, so the rest of this section does not apply. Only the management LIF must be reachable over IP.
Optional: a dedicated storage interface with HostNetworkConfig
This takes two steps: create a cluster network on the storage NICs, then give each host an IP address on it with a HostNetworkConfig.
Step 1: create the cluster network. A cluster network groups the storage NICs into a bridge on each node. See Create a New Cluster Network for details.
- In the Harvester UI, go to Networks > ClusterNetworks/Configs and click Create. Name it, for example
cn-storage. - Click Create Network Config on the new cluster network.
- On the Node Selector tab, select the nodes that mount NetApp volumes. Select all nodes does not work if the cluster has a witness node. Select nodes by name, or by a label such as
netapp-trident=enabled(see the note further down). - On the Uplink tab, select the storage NICs, the bond mode, and the MTU. The MTU must match the switch ports and the ONTAP LIFs, for example
9000for jumbo frames. - Click Save.
The switch ports for these NICs must be trunk ports that carry the storage VLAN tagged.
Step 2: give the hosts an IP address on it. On its own, a cluster network carries only VM traffic, because the hosts have no IP address on it. A HostNetworkConfig (Networks > Host Networks) creates a VLAN sub-interface on the cluster network's bridge on each node and gives it an IP address. Traffic to the ONTAP data LIFs on that subnet then leaves through that interface instead of the management network. The ONTAP data LIFs must be on the same VLAN and subnet.
Example with static addresses on VLAN 150:
apiVersion: network.harvesterhci.io/v1beta1
kind: HostNetworkConfig
metadata:
name: netapp-storage
spec:
clusterNetwork: cn-storage
vlanID: 150
mode: static
nodeSelector:
matchLabels:
netapp-trident: enabled
ips:
<node-1>: 192.168.150.11/24
<node-2>: 192.168.150.12/24
Check the result on a node with ip addr show cn-storage-br.150. Each node's state is reported in the resource's status.nodeStatus.
Static mode and witness nodes
In static mode without a nodeSelector, the webhook requires an IP for every node in the cluster, witness included. Otherwise it rejects the resource with static IP not found for node <witness>. You have two options:
- Add a
nodeSelector, as in the example above. Only the matching nodes then need an entry inips. The selector can use the samenetapp-trident=enabledlabel that restricts the Trident node plugin. This way the witness is excluded from the storage network and from Trident consistently. - Give the witness an IP as well. It costs one address on the storage subnet. The witness never mounts volumes, so the address goes unused.
In static mode, new nodes are not configured automatically. Add their entry to ips (and the label, if you use one) when they join.
For iSCSI, you can add a second cluster network (or a second VLAN) with its own HostNetworkConfig and LIFs, so multipath has two independent paths.
1. Configure multipathd (iSCSI and FC)
iSCSI and FC LUNs need multipathd, which is not enabled on Harvester by default. Both protocols present NetApp LUNs with vendor NETAPP and product LUN, so the same configuration covers them. The configuration below lets multipathd claim only NetApp LUNs. Every other device is blacklisted, so Longhorn and the local disks are never touched. Harvester's own /etc/multipath/conf.d/99-longhorn.conf is left as is.
caution
Never start multipathd before the NetApp-only configuration is in place. With its default configuration, multipathd claims local disks, including Longhorn disks, as dm multipath devices.
The following CloudInit resource writes /etc/multipath/conf.d/98-netapp.conf and enables multipathd on every node. It is re-applied on every boot. Save it as 01-cloudinit-netapp-multipathd.yaml:
# Persists across reboots: writes /etc/multipath/conf.d/98-netapp.conf (multipath ONLY claims NETAPP LUNs)
apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
name: netapp-multipathd
spec:
matchSelector: {}
filename: 98_netapp_multipathd.yaml
contents: |
name: "enable multipathd for NetApp Trident iSCSI/FC (NETAPP LUNs only)"
stages:
initramfs:
- files:
- path: /etc/multipath/conf.d/98-netapp.conf
permissions: 0644
owner: 0
group: 0
content: |
defaults {
user_friendly_names yes
find_multipaths no
# lets libmpathpersist (qemu-pr-helper) track SCSI-3 PR keys across paths, for shared disks (e.g. WSFC)
reservation_key file
# don't create kpartx maps for partitions that guests create on the LUNs
skip_kpartx yes
}
blacklist {
device {
vendor ".*"
product ".*"
}
}
blacklist_exceptions {
device {
vendor "NETAPP"
product "LUN.*"
}
}
network:
- name: "enable and start multipathd"
systemctl:
enable:
- multipathd
start:
- multipathd
kubectl apply -f 01-cloudinit-netapp-multipathd.yaml
The CloudInit resource takes effect the next time each node boots. To apply it now without rebooting, run the following on each node as root. It writes the same file and starts multipathd, and stops at the first failure:
set -e
mkdir -p /etc/multipath/conf.d
cat > /etc/multipath/conf.d/98-netapp.conf <<'EOF'
defaults {
user_friendly_names yes
find_multipaths no
reservation_key file
skip_kpartx yes
}
blacklist {
device {
vendor ".*"
product ".*"
}
}
blacklist_exceptions {
device {
vendor "NETAPP"
product "LUN.*"
}
}
EOF
systemctl enable --now multipathd
multipath -ll # must not list any local disk
note
- NVMe/TCP does not use
dm-multipath. It uses native NVMe multipath, which is already enabled in the Harvester kernel, and doesn't neednvme-clion the host. - NFS needs no host configuration.
Check that every node has a unique iSCSI initiator name. Nodes cloned from the same image can end up with the same name in /etc/iscsi/initiatorname.iscsi. ONTAP then treats those nodes as a single host. If you change a node's IQN, restart that node's trident-node-linux pod afterwards. Trident caches the IQN, and with a stale one volume staging fails with no devices present yet.
2. Install the Trident operator
helm repo add netapp-trident https://netapp.github.io/trident-helm-chart
helm repo update
helm search repo netapp-trident --versions | head
helm upgrade --install trident netapp-trident/trident-operator --version 100.2606.1 \
--namespace trident --create-namespace --wait
Wait until the trident-controller pod and one trident-node-linux pod per node are Running:
kubectl -n trident get pods -o wide
Optional: run the node plugin only on selected nodes
By default, the trident-node-linux DaemonSet tolerates all NoSchedule and NoExecute taints, so it runs on every node, including a witness node. A witness node only runs etcd and never mounts volumes, so you may want to keep Trident off it. The same approach works for keeping Trident on any subset of nodes that host NetApp-backed VMs.
Label the nodes that should mount NetApp volumes, then pass the label to the chart:
kubectl label node <node-1> <node-2> netapp-trident=enabled --overwrite
helm upgrade --install trident netapp-trident/trident-operator --version 100.2606.1 \
--namespace trident --create-namespace --wait \
--set tridentNodePluginNodeSelector.netapp-trident=enabled
When you add a node to the cluster later, label it too. Otherwise VMs scheduled on it cannot mount NetApp volumes.
If you use a dedicated storage interface, use the same label as the HostNetworkConfig nodeSelector. Then the nodes that run Trident and the nodes with an IP on the storage network are always the same set.
VMs using NetApp volumes must only run on the labelled nodes. A Harvester witness node never runs VMs, so this needs no extra configuration in the witness case.
3. Create the ONTAP credentials Secret
kubectl -n trident create secret generic netapp-creds \
--from-literal=username=<ontap user> \
--from-literal=password='<ontap password>'
4. Create the backends
Create one TridentBackendConfig per protocol you use; you can drop any you don't need.
The examples below use these settings:
- Thin provisioning:
spaceReserve: noneandsnapshotReserve: "0". limitAggregateUsage: "80%": Trident stops creating new volumes once the aggregate passes 80% used.
If several Kubernetes or Harvester clusters share one SVM, give each cluster a unique backendName and set storagePrefix (for example storagePrefix: hv1_). That keeps volume names from colliding and makes each cluster's volumes easy to identify in ONTAP.
For Fibre Channel, copy the iSCSI backend below, then change name and backendName (for example to netapp-fc) and set sanType: fcp. Then add a matching StorageClass in step 5 with storagePools: "netapp-fc:.*". FC was not tested for this article.
Save as 03-backends.yaml, replacing the <...> values:
# NFS
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
name: netapp-nas
namespace: trident
spec:
version: 1
storageDriverName: ontap-nas
backendName: netapp-nas
managementLIF: <management LIF IP>
dataLIF: <NFS data LIF IP> # optional; defaults to the first NFS LIF in the SVM
svm: <svm name>
autoExportPolicy: true
autoExportCIDRs: ["<storage network CIDR>"]
nfsMountOptions: "nfsvers=4.1"
limitAggregateUsage: "80%"
credentials:
name: netapp-creds
defaults:
spaceReserve: none
snapshotReserve: "0"
---
# iSCSI (all iSCSI LIFs in the SVM are discovered; no dataLIF)
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
name: netapp-iscsi
namespace: trident
spec:
version: 1
storageDriverName: ontap-san
sanType: iscsi
backendName: netapp-iscsi
managementLIF: <management LIF IP>
svm: <svm name>
limitAggregateUsage: "80%"
credentials:
name: netapp-creds
defaults:
spaceReserve: none
snapshotReserve: "0"
---
# NVMe/TCP (all NVMe LIFs in the SVM are discovered; no dataLIF)
apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
name: netapp-nvme
namespace: trident
spec:
version: 1
storageDriverName: ontap-san
sanType: nvme
backendName: netapp-nvme
managementLIF: <management LIF IP>
svm: <svm name>
limitAggregateUsage: "80%"
credentials:
name: netapp-creds
defaults:
spaceReserve: none
snapshotReserve: "0"
kubectl apply -f 03-backends.yaml
kubectl -n trident get tridentbackendconfigs
Every backend must report PHASE Bound and STATUS Success. If one doesn't, kubectl -n trident describe tridentbackendconfig <name> shows the error.
5. Create the StorageClasses and VolumeSnapshotClass
Trident StorageClasses have no sanType parameter. A StorageClass with only backendType: ontap-san can place a volume on any SAN backend: iSCSI, NVMe/TCP or FC. To prevent that, pin each class to its backend with storagePools: "<backendName>:.*".
Save as 04-storageclasses.yaml:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: netapp-nfs
provisioner: csi.trident.netapp.io
parameters:
backendType: "ontap-nas"
storagePools: "netapp-nas:.*"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: netapp-iscsi
provisioner: csi.trident.netapp.io
parameters:
backendType: "ontap-san"
storagePools: "netapp-iscsi:.*"
fsType: "ext4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: netapp-nvme
provisioner: csi.trident.netapp.io
parameters:
backendType: "ontap-san"
storagePools: "netapp-nvme:.*"
fsType: "ext4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: trident-snapshotclass
driver: csi.trident.netapp.io
deletionPolicy: Delete
kubectl apply -f 04-storageclasses.yaml
Live migration requires ReadWriteMany volumes. CDI already knows Trident's defaults:
ontap-san(iSCSI, NVMe/TCP and FC):ReadWriteManywithvolumeMode: Block.ontap-nas:ReadWriteManywithvolumeMode: Filesystem.
So these StorageClasses don't need the cdi.harvesterhci.io/storageProfileVolumeModeAccessModes annotation.
6. Register Trident for Harvester VM snapshots
Harvester needs to know which VolumeSnapshotClass to use for Trident volumes.
- In the Harvester UI, go to Advanced > Settings, find csi-driver-config, and select ⋮ > Edit Setting.
- Click Add, then fill in the fields:
- Provisioner:
csi.trident.netapp.io - Volume Snapshot Class Name:
trident-snapshotclass
- Provisioner:
- Save.
Leave Backup Volume Snapshot Class Name empty. In Harvester v1.9, VM backups only work with Longhorn volumes. VM snapshots of Trident volumes work. See Third-Party Storage Support.
The resulting entry in the setting's JSON value is:
"csi.trident.netapp.io": {
"volumeSnapshotClassName": "trident-snapshotclass",
"backupVolumeSnapshotClassName": ""
}
7. Use it
The new StorageClasses now appear when you create a volume, a VM image or a VM in the Harvester UI. A quick check:
- Create a VM whose root disk uses
netapp-iscsi,netapp-nvmeornetapp-nfs. - Start the VM, live-migrate it to another node, and take a VM snapshot.
- On a node running the VM,
multipath -ll(iSCSI) orls /sys/class/nvme-subsystem(NVMe) shows the NetApp devices.
Troubleshooting
- Backends go
offlineafter the ONTAP system was unreachable. If they don't recover on their own, restart the controller:kubectl -n trident rollout restart deploy/trident-controller. - Deleted volumes still use space on ONTAP. ONTAP keeps deleted volumes in the volume recovery queue for 12 hours. While a deleted FlexClone (such as a VM disk cloned from an image) sits in the queue, its parent snapshot stays busy, so the parent volume and its
VolumeSnapshotcan't be deleted. To free them immediately, run on ONTAP:set -privilege advanced; volume recovery-queue purge-all -vserver <svm name>. multipath -lllists local disks.multipathdwas started without the NetApp-only configuration. Put98-netapp.confin place, flush the unused maps withmultipath -F, and restartmultipathd.- iSCSI volumes fail to stage with
no devices present yet. Check for duplicate initiator names across nodes, and for a stale IQN in Trident (see step 1).
