4 posts tagged with "network"

You can integrate Harvester with the DNS servers that already exist in your environment. Specifically, you can configure the DNS settings that virtual machines (VMs) receive, and you can use ExternalDNS to publish DNS records for Harvester load balancers (LBs) and for LoadBalancer services in guest clusters.

This procedure was tested with Harvester v1.9.0, ExternalDNS v0.15.0, BIND 9.20, and PowerDNS Authoritative Server 4.9. The guest cluster load balancer example was tested with an RKE2 v1.36.5 guest cluster that uses the Harvester cloud provider v0.2.8.

User Stories​

• An infrastructure team runs application VMs behind Harvester load balancers. Application owners want to reach each application by a name in the corporate DNS zone (for example, web.vm.example.com) instead of an IP address. The DNS record must be created when the load balancer is created and removed when the load balancer is deleted, without a manual request to the DNS team. The VMs must also resolve internal names by using the corporate DNS servers.

• A platform team runs RKE2 guest clusters on Harvester. Developers expose applications with LoadBalancer services and want each service to be resolvable by name in the corporate DNS zone as soon as it receives an IP address.

Architecture​

The following diagram shows the components involved and which party manages each component.

Harvester and your existing infrastructure have the following responsibilities:

ExternalDNS is a third-party component that is not installed or managed by Harvester. ExternalDNS is available in the SUSE Application Collection. You can run ExternalDNS in the Harvester cluster or outside it, provided that it can read services from the Harvester Kubernetes API. When ExternalDNS runs outside the cluster, a service account with read-only access to services is sufficient.

DNS Integration Options​

The following table outlines the available options:

Configure DNS Settings for VMs​

VMs obtain DNS settings from the network configuration that they receive. You can use any of the following methods:

• Managed DHCP: Specify the dns, domainName, and domainSearch fields in the IPPool resource. The Managed DHCP agent sends these values to VMs as DHCP options. For more information, see Managed DHCP.

• Kube-OVN DHCP: Specify the DNS server in the dhcpV4Options field of the subnet (for example, dns_server=10.0.0.53). For more information, see Virtual Private Cloud (VPC).

• External DHCP server or static configuration: Configure the DNS settings on your DHCP server, or specify the DNS servers in the cloud-init network data when you create the VM.

Limitations​

• Harvester does not automatically create DNS records for the IP addresses of individual VMs. You can use ExternalDNS to publish DNS records only for load balancers.

• The Managed DHCP add-on does not send the hostname option (option 12) and does not process the client FQDN option (option 81). Consequently, DHCP-based dynamic DNS updates are not supported for VMs that obtain IP addresses from Managed DHCP.

Publish DNS Records for VM Load Balancers​

Each VM load balancer is backed by a Kubernetes Service of type LoadBalancer. The service has the following characteristics:

• It has the same name and namespace as the load balancer.
• It has the label loadbalancer.harvesterhci.io/servicelb: "true".
• Its status.loadBalancer.ingress field contains the IP address assigned to the load balancer.

This applies to load balancers that use either the IP pool or the DHCP IPAM mode.

Because the service is a standard Kubernetes resource, ExternalDNS can watch it and automatically create or update the corresponding DNS records on your DNS server.

Prerequisites​

Before you begin, ensure that the following requirements are met:

• The DNS server supports dynamic updates (RFC 2136) and is reachable from the location where ExternalDNS runs.

• A TSIG key is configured on the DNS server, and the key is allowed to update the target zone. For a BIND configuration example, see the ExternalDNS RFC 2136 tutorial.

The examples in this section use the following settings:

• DNS server: 192.168.100.53 (BIND)
• Zone: vm.example.com
• TSIG key name: externaldns-key
• TSIG algorithm: hmac-sha256

Deploy ExternalDNS​

1. Create the namespace for ExternalDNS.

   kubectl create namespace external-dns
   

   Copy

2. Create a secret that contains the TSIG secret.

   kubectl create secret generic external-dns-tsig \
     --namespace external-dns \
     --from-literal=tsig-secret='<base64-tsig-secret>'
   

   Copy

3. Deploy ExternalDNS with the RBAC resources described in the ExternalDNS RFC 2136 tutorial, and configure the container as follows:

   containers:
   - name: external-dns
     image: registry.k8s.io/external-dns/external-dns:v0.15.0
     args:
     - --source=service
     # Watch only the services created by the Harvester load balancer
     - --label-filter=loadbalancer.harvesterhci.io/servicelb=true
     # Generate a DNS name for each load balancer
     - --fqdn-template={{ .Name }}.{{ .Namespace }}.vm.example.com
     - --domain-filter=vm.example.com
     - --provider=rfc2136
     - --rfc2136-host=192.168.100.53
     - --rfc2136-port=53
     - --rfc2136-zone=vm.example.com
     - --rfc2136-tsig-keyname=externaldns-key
     - --rfc2136-tsig-secret-alg=hmac-sha256
     - --rfc2136-tsig-axfr
     - --registry=txt
     - --txt-owner-id=harvester-cluster-1
     - --policy=upsert-only
     env:
     - name: EXTERNAL_DNS_RFC2136_TSIG_SECRET
       valueFrom:
         secretKeyRef:
           name: external-dns-tsig
           key: tsig-secret
   

   Copy

   With this configuration, ExternalDNS publishes a DNS record named <load-balancer-name>.<namespace>.vm.example.com for each VM load balancer.

4. Create a VM load balancer.

5. Verify that the DNS record resolves to the IP address of the load balancer.

   dig +short <load-balancer-name>.<namespace>.vm.example.com @192.168.100.53
   

   Copy

Assign a Custom Hostname​

To use a specific hostname instead of the name generated by --fqdn-template, add the external-dns.alpha.kubernetes.io/hostname annotation to the load balancer's service.

Example:

kubectl annotate service <load-balancer-name> --namespace <namespace> \
  external-dns.alpha.kubernetes.io/hostname=web.vm.example.com

DNS Record Lifecycle​

ExternalDNS manages the DNS records of each load balancer according to the value of the --policy flag:

In both modes, ExternalDNS only modifies the records that it owns, and does not change records that were created by other means.

Publish DNS Records for Guest Cluster Load Balancers​

LoadBalancer services in guest clusters that use the Harvester cloud provider are standard Kubernetes services. To publish DNS records for these services, deploy ExternalDNS in the guest cluster and configure it for your DNS server. You can then add the external-dns.alpha.kubernetes.io/hostname annotation to each service.

The IP address of a guest cluster load balancer appears in the service's status.loadBalancer.ingress field only after the service has at least one ready endpoint. Until then, ExternalDNS does not publish a DNS record for the service.

For more information, see the ExternalDNS documentation.

DNS Server Integrations Included in ExternalDNS​

ExternalDNS includes integrations (providers) for a wide range of DNS servers. These integrations are developed and maintained by the ExternalDNS project. The following table lists providers that are commonly used in on-premises environments:

For a complete list of providers, including public cloud DNS services, see the ExternalDNS documentation.

We have the default SSH user rancher, but other users may be required. Creating users with useradd will result in their deletion upon restarting the Harvester node; therefore, follow the steps below to create persistent SSH users.

Public Key Authentication​

Create cloud-init.yaml​

Create cloud-init.yaml with the following content.

• Modify the matchSelector if you want to create an SSH user only on specific nodes.
• #cloud-config should be written exactly as shown.
• User must be in the admin group.
• Specify the public key in ssh_authorized_keys.
• Modify the contents according to your environment.

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: add-test-user
spec:
  matchSelector: {} # applies to all nodes
  filename: 99_add_test_user.yaml
  contents: |
    #cloud-config
    users:
      - name: test-user
        gecos: "admin_user"
        groups: [users, admin]
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        ssh_authorized_keys:
        - ssh-rsa AAAA.... # <--insert full authorized key here, e.g. from your ~/.ssh/id_rsa.pub file

Password Authentication​

Create password hash​

Use the following command to create a password hash. Replace test with your actual password.

$ openssl passwd -6 'test'
$6$zF26pcXOS2eaivX8$6ySoTzQC2cToz29mGFC0DuG5cVWTv3Mktc3k/g1KXTtrG2BhsFh8xs3N0zBmNx0D/H4f1W48a45vI1RK8Rzs.0

Create cloud-init.yaml​

Create cloud-init.yaml with the following content.

• Modify the matchSelector if you want to create an SSH user only on specific nodes.
• #cloud-config should be written exactly as shown.
• User must be in the admin group.
• Specify the hash created earlier in passwd.
• Modify the contents according to your environment.

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: add-test-user
spec:
  matchSelector: {} # applies to all nodes
  filename: 99_add_test_user.yaml
  contents: |
    #cloud-config
    users:
      - name: test-user
        gecos: "admin_user"
        groups: [users, admin]
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        lock_passwd: false
        passwd: $6$zF26pcXOS2eaivX8$6ySoTzQC2cToz29mGFC0DuG5cVWTv3Mktc3k/g1KXTtrG2BhsFh8xs3N0zBmNx0D/H4f1W48a45vI1RK8Rzs.0

Apply the YAML file​

Apply the YAML with this command.

kubectl apply -f cloud-init.yaml

Trailing file will be created.

$ cat /oem/99_add_test_user.yaml
#cloud-config
users:
  - name: test-user
    gecos: "admin_user"
    groups: users, admin
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ...

Reboot Harvester nodes​

1. Run Enable Maintenance Mode on the Harvester UI.
2. Wait until the state changes to Maintenance.
3. Reboot the Harvester node.
4. Run Disable Maintenance Mode on the Harvester UI.

After that, SSH login will be possible.

Tested on: Harvester 1.6.0, 1.5.1.

NIC Naming Scheme changed after upgrading to v1.0.1​

systemd in OpenSUSE Leap 15.3 which is the base OS of Harvester is upgraded to 246.16-150300.7.39.1. In this version, systemd will enable additional naming scheme sle15-sp3 which is v238 with bridge_no_slot. When there is a PCI bridge associated with NIC, systemd will never generate ID_NET_NAME_SLOT and naming policy in /usr/lib/systemd/network/99-default.link will fallback to ID_NET_NAME_PATH. According to this change, NIC names might be changed in your Harvester nodes during the upgrade process from v1.0.0 to v1.0.1-rc1 or above, and it will cause network issues that are associated with NIC names.

Effect Settings and Workaround​

Startup Network Configuration​

NIC name changes will need to update the name in /oem/99_custom.yaml. You could use migration script to change the NIC names which are associated with a PCI bridge.

You could simply execute the script with root account in v1.0.0 via

# python3 udev_v238_sle15-sp3.py

It will output the patched configuration to the screen and you could compare it to the original one to ensure there is no exception. (e.g. We could use vimdiff to check the configuration)

# python3 udev_v238_sle15-spe3.py > /oem/test
# vimdiff /oem/test /oem/99_custom.yaml

After checking the result, we could execute the script with --really-want-to-do to override the configuration. It will also back up the original configuration file with a timestamp before patching it.

# python3 udev_v238_sle15-sp3.py --really-want-to-do

Harvester VLAN Network Configuration​

If your VLAN network is associated with NIC name directly without bonding, you will need to migrate ClusterNetwork and NodeNetwork with the previous section together.

Modify ClusterNetworks​

You need to modify ClusterNetworks via

$ kubectl edit clusternetworks vlan

search this pattern

config:
  defaultPhysicalNIC: <Your NIC name>

and change to new NIC name

Modify NodeNetworks​

You need to modify NodeNetworks via

$ kubectl edit nodenetworks <Node name>-vlan

search this pattern

spec:
  nic: <Your NIC name>

and change to new NIC name

What is the default behavior of a VM with multiple NICs​

In some scenarios, you'll setup two or more NICs in your VM to serve different networking purposes. If all networks are setup by default with DHCP, you might get random connectivity issues. And while it might get fixed after rebooting the VM, it still will lose connection randomly after some period.

How-to identify connectivity issues​

In a Linux VM, you can use commands from the iproute2 package to identify the default route.

In your VM, execute the following command:

ip route show default

The output of this command will only show the default route with the gateway and VM IP of the primary network interface (eth0 in the example below).

default via <Gateway IP> dev eth0 proto dhcp src <VM IP> metric 100

Here is the full example:

$ ip route show default
default via 192.168.0.254 dev eth0 proto dhcp src 192.168.0.100 metric 100

However, if the issue covered in this KB occurs, you'll only be able to connect to the VM via the VNC or serial console.

Once connected, you can run again the same command as before:

$ ip route show default

However, this time you'll get a default route with an incorrect gateway IP. For example:

default via <Incorrect Gateway IP> dev eth0 proto dhcp src <VM's IP> metric 100

Why do connectivity issues occur randomly​

In a standard setup, cloud-based VMs typically use DHCP for their NICs configuration. It will set an IP and a gateway for each NIC. Lastly, a default route to the gateway IP will also be added, so you can use its IP to connect to the VM.

However, Linux distributions start multiple DHCP clients at the same time and do not have a priority system. This means that if you have two or more NICs configured with DHCP, the client will enter a race condition to configure the default route. And depending on the currently running Linux distribution DHCP script, there is no guarantee which default route will be configured.

As the default route might change in every DHCP renewing process or after every OS reboot, this will create network connectivity issues.

How to avoid the random connectivity issues​

You can easily avoid these connectivity issues by having only one NIC attached to the VM and having only one IP and one gateway configured.

However, for VMs in more complex infrastructures, it is often not possible to use just one NIC. For example, if your infrastructure has a storage network and a service network. For security reasons, the storage network will be isolated from the service network and have a separate subnet. In this case, you must have two NICs to connect to both the service and storage networks.

You can choose a solution below that meets your requirements and security policy.

Disable DHCP on secondary NIC​

As mentioned above, the problem is caused by a race condition between two DHCP clients. One solution to avoid this problem is to disable DHCP for all NICs and configure them with static IPs only. Likewise, you can configure the secondary NIC with a static IP and keep the primary NIC enabled with DHCP.

1. To configure the primary NIC with a static IP (eth0 in this example), you can edit the file /etc/sysconfig/network/ifcfg-eth0 with the following values:

BOOTPROTO='static'
IPADDR='192.168.0.100'
NETMASK='255.255.255.0'

Alternatively, if you want to reserve the primary NIC using DHCP (eth0 in this example), use the following values instead:

BOOTPROTO='dhcp'
DHCLIENT_SET_DEFAULT_ROUTE='yes'

2. You need to configure the default route by editing the file /etc/sysconfig/network/ifroute-eth0 (if you configured the primary NIC using DHCP, skip this step):

# Destination  Dummy/Gateway  Netmask  Interface
default        192.168.0.254  -        eth0

3. Finally, configure a static IP for the secondary NIC by editing the file /etc/sysconfig/network/ifcfg-eth1:

BOOTPROTO='static'
IPADDR='10.0.0.100'
NETMASK='255.255.255.0'

Cloud-Init config​

network:
  version: 1
  config:
    - type: physical
      name: eth0
      subnets:
        - type: dhcp
    - type: physical
      name: eth1
      subnets:
        - type: static
          address: 10.0.0.100/24

Disable secondary NIC default route from DHCP​

If your secondary NIC requires to get its IP from DHCP, you'll need to disable the secondary NIC default route configuration.

1. Confirm that the primary NIC configures its default route in the file /etc/sysconfig/network/ifcfg-eth0:

BOOTPROTO='dhcp'
DHCLIENT_SET_DEFAULT_ROUTE='yes'

2. Disable the secondary NIC default route configuration by editing the file /etc/sysconfig/network/ifcfg-eth1:

BOOTPROTO='dhcp'
DHCLIENT_SET_DEFAULT_ROUTE='no'

Cloud-Init config​

This solution is not available in Cloud-Init. Cloud-Init didn't allow any option for DHCP.