NIC teaming on Centos 7

I've heard a lot about link aggregation, teaming or bonding but I've never tried it. Until now. In this post I will demonstrate how to setup NIC teaming on CentOS 7 virtual host.

TOC

• Overview
• Limit network bandwidth for guest NIC
• Add another NIC to guest
• Setup NIC teaming on guest
• Redundancy test
• Sources

Limit network bandwidth for guest NIC

At first I will test network speed on guest OS without any bandwidth limitation. Network device configuration of my centos-team guest looks pretty normal:

    . 
    .
    .
    <interface type='network'>
      <mac address='52:54:00:c6:b8:9f'/>
      <source network='default'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    .
    .
    .

After booting up the guest I will check transfer speed with copying big file over scp. As you can see it is around 50MB/s:

[root@kra ~]$ scp centos-7.img root@centos-team:.
centos-7.img                                   10%  940MB  51.4MB/s   02:43 ETA
^CKilled by signal 2.

Now I will limit guest NIC bandwidth to 10MB/s with <bandwidth> tag:

    .
    .
    .
    <interface type='network'>
      <mac address='52:54:00:c6:b8:9f'/>
      <source network='default'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <bandwidth>
        <inbound average='10240'/>
        <outbound average='10240'/>
      </bandwidth> 
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    .
    .
    .

It looks it is working properly. Transfer speed dropped to around 10MB/s:

[root@kra ~]$ scp centos-7.img root@centos-team:.
centos-7.img                                    2%  189MB   9.6MB/s   15:52 ETA
^CKilled by signal 2.

Add another NIC to guest

This step is really simple. I've just put another network device to <devices> part of guest domain:

    .
    .
    .
    <interface type='network'>
      <mac address='52:54:00:c6:b8:9f'/>
      <source network='default'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <bandwidth>
        <inbound average='10240'/>
        <outbound average='10240'/>
      </bandwidth>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='network'>
      <source network='default'/>
      <target dev='vnet2'/>
      <model type='virtio'/>
      <bandwidth>
        <inbound average='10240'/>
        <outbound average='10240'/>
      </bandwidth>
    </interface> 
    .
    .
    .

After reboot I can see 2 network cards (both with IP address assigned by DHCP):

[root@centos-team ~]# nmcli device
DEVICE  TYPE      STATE      CONNECTION         
eth0    ethernet  connected  System eth0        
eth1    ethernet  connected  Wired connection 1 
lo      loopback  unmanaged  --                 
[root@centos-team ~]# nmcli connection
NAME                UUID                                  TYPE            DEVICE 
Wired connection 1  b709aa19-505f-4b7a-8513-a0bf8bfb3ff8  802-3-ethernet  eth1   
System eth0         4628dca0-dc29-4c47-9f4a-743a08010ea5  802-3-ethernet  eth0   
[root@centos-team ~]# ip address list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:c6:b8:9f brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.41/24 brd 192.168.122.255 scope global dynamic eth0
       valid_lft 3536sec preferred_lft 3536sec
    inet6 fe80::5054:ff:fec6:b89f/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:ee:47:16 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.187/24 brd 192.168.122.255 scope global dynamic eth1
       valid_lft 3538sec preferred_lft 3538sec
    inet6 fe80::5054:ff:feee:4716/64 scope link 
       valid_lft forever preferred_lft forever

Setup NIC teaming on guest

Now I want to team these to NICs together. I will assign static IP adress to teamed device (team0). First step is to backup current network configuration:

[root@centos-team ~]# cd /etc/sysconfig/network-scripts/
[root@centos-team network-scripts]# mkdir backup
[root@centos-team network-scripts]# mv ifcfg-eth* backup/

I've created two configuration for each NIC - ifcfg-eth0 and ifcfg-eth1:

[root@centos-team network-scripts]# cat ifcfg-eth0
DEVICE="eth0"
DEVICETYPE="TeamPort"
ONBOOT="yes"
BOOTPROTO="none"
TEAM_MASTER="team0"
[root@centos-team network-scripts]# cat ifcfg-eth1
DEVICE="eth1"
DEVICETYPE="TeamPort"
ONBOOT="yes"
BOOTPROTO="none"
TEAM_MASTER="team0"

DEVICETYPE line defines that this devices is part of team. TEAM_MASTER defines to which master device it belongs. Next step is to define team device:

[root@centos-team network-scripts]# cat ifcfg-team0 
DEVICE="team0"
DEVICETYPE="Team"
ONBOOT="yes"
BOOTPROTO="none"
NETMASK=255.255.255.0
IPADDR=192.168.122.200
TEAM_CONFIG='{"runner": {"name": "roundrobin"}}'

Device will be configured with IP address of 192.168.122.200. Parameter TEAM_CONFIG stores teamd configuration. For more information see man teamd.conf. This config says that load on team device (team0) will be split on slave devices (eth0 and eth1) in round-robin fashion. It will basically join these slave interfaces together to achieve double network bandwidth. After reboot it looks like this:

[root@centos-team ~]# ip address list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP qlen 1000
    link/ether 52:54:00:c6:b8:9f brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP qlen 1000
    link/ether 52:54:00:c6:b8:9f brd ff:ff:ff:ff:ff:ff
4: team0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 52:54:00:c6:b8:9f brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.200/24 brd 192.168.122.255 scope global team0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fec6:b89f/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
[root@centos-team ~]# nmcli connection
NAME         UUID                                  TYPE            DEVICE 
Team team0   702de3eb-2e80-897c-fd52-cd0494dd8123  team            team0  
System eth1  9c92fad9-6ecb-3e6c-eb4d-8a47c6f50c04  802-3-ethernet  eth1   
System eth0  5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03  802-3-ethernet  eth0   
[root@centos-team ~]# nmcli device
DEVICE  TYPE      STATE      CONNECTION  
eth0    ethernet  connected  System eth0 
eth1    ethernet  connected  System eth1 
team0   team      connected  Team team0  
lo      loopback  unmanaged  --          

I can see team daemon running:

[root@centos-team ~]# systemctl status NetworkManager
NetworkManager.service - Network Manager
   Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled)
   Active: active (running) since Wed 2014-07-16 17:58:10 CEST; 16h ago
 Main PID: 673 (NetworkManager)
   CGroup: /system.slice/NetworkManager.service
           |-673 /usr/sbin/NetworkManager --no-daemon
           \-843 /usr/bin/teamd -o -n -U -D -t team0 -c {"runner": {"name": "roundrobin"}}

Let's try if teaming works. I will copy file from host to guest again. This time I'm expecting nearly double transfer speed:

[root@kra ~]$ scp centos-7.img root@192.168.122.200:.
centos-7.img                                                                                                                                                                         4%  441MB  18.5MB/s   08:00 ETA
^CKilled by signal 2.

My expectation was right. I can use teaming not only for increasing network bandwidth but also to achieve redundancy.

Redundancy test

Team device (team0)) consists of two devices eth0, eth1). If I cut down one device, team device should work without any error. Of course, speed will drop down to half. I can use ip link set <device> up/down to emulate plug in/off of the network cable. And with teamdctl I can see state of team device. This is how it looks during normal operation:

[root@centos-team ~]# ip link show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP mode DEFAULT qlen 1000
    link/ether 52:54:00:c6:b8:9f brd ff:ff:ff:ff:ff:ff
[root@centos-team ~]# ip link show dev eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP mode DEFAULT qlen 1000
    link/ether 52:54:00:c6:b8:9f brd ff:ff:ff:ff:ff:ff
[root@centos-team ~]# teamdctl team0 state view
setup:
  runner: roundrobin
ports:
  eth0
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
  eth1
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up

I will copy file to guest again and this time I will keep it running. During file transfer I will plug off network cable of eth0 device:

[root@centos-team ~]# ip link set eth0 down
[root@centos-team ~]# ip link show eth0
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master team0 state DOWN mode DEFAULT qlen 1000
    link/ether 52:54:00:c6:b8:9f brd ff:ff:ff:ff:ff:ff

I will check state of team0 device and It notices link state change:

[root@centos-team ~]# teamdctl team0 state view
setup:
  runner: roundrobin
ports:
  eth0
    link watches:
      link summary: down
      instance[link_watch_0]:
        name: ethtool
        link: down
  eth1
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up

If I check file transfer, I will see transfer speed is slowly decreasing up to 10MB/s, which is maximum transfer speed of remaining NIC. Connection is working with only one network card hence redundancy test was passed successfully.

Sources

• Libvirt documentation
• TeamDriver fedora wiki