Configuring static routing

In the guide below, we will demonstrate static routing using a virtual networking lab using vmm.

In order to save on costs, we will be using the reserved IP address prefix 10/8 and then translate the range to a public IP address. Here is the topology:

    R1 <-----> Internet
 (10.1/16)
 10.2.1.1
    ^
    |
    |
   veb12
    |
    |
    V
 10.2.2.1
    R2  10.3.2.1 <-veb23-> 10.3.2.2 R3 10.5.3.1 <--veb35--> 10.5.5.1 R5
 (10.2/16)                       (10.3/16)      (10.5/16)
 10.4.2.1                        10.3.4.2
     ^                              ^
      \                            /
     veb24                        /
        \                       veb34
         \                      /
          ------>   R4    <-----
       10.4.4.1  (10.4/16)  10.3.4.1

Each R# in the above topology represents a router, which we simulate with a virtual machine with multiple interfaces. We will use veb(4) to link two routers together. Because there are multiple interfaces on each router, no three nodes will ever share the same broadcast domain. This ensures that routing will be required to traverse the network.

In the hypervisor, we configure the proper interfaces:

host# cat /etc/vm.conf
socket owner :vmdusers

switch "switch12" {
    locked lladdr
    interface veb12
}

switch "switch23" {
    locked lladdr
    interface veb23
}

switch "switch24" {
    locked lladdr
    interface veb24
}

switch "switch34" {
    locked lladdr
    interface veb34
}

switch "switch35" {
    locked lladdr
    interface veb35
}

bsdiso="/home/iso/install75.iso"

vm "r2" {
    owner $USER
    memory 1G
    cdrom $bsdiso
    disk /home/$USER/r2.qcow2 format qcow2
    interface tap22 {
        locked lladdr e8:8b:22:22:22:22
        switch "switch12"
    }
    interface tap32 {
        locked lladdr e8:8b:32:32:32:32
        switch "switch23"
    }
    interface tap42 {
        locked lladdr e8:8b:42:42:42:42
        switch "switch24"
    }
}

vm "r3" {
    owner $USER
    memory 1G
    cdrom $bsdiso
    disk /home/$USER/r3.qcow2 format qcow2
    interface tap322 { 
        locked lladdr e8:8b:32:02:32:02
        switch "switch23"
    }
    interface tap342 { 
        locked lladdr e8:8b:34:02:34:02
        switch "switch34"
    }
    interface tap53 {
        locked lladdr e8:8b:53:53:53:53
        switch "switch35"
    }
}

vm "r4" {
    owner $USER
    memory 1G
    cdrom $bsdiso
    disk /home/$USER/r4.qcow2 format qcow2
    interface tap44 { 
        switch "switch24"
        locked lladdr e8:8b:44:44:44:44
    }
    interface tap34 { 
        locked lladdr e8:8b:34:34:34:34
        switch "switch34"
    }
}

vm "r5" {
    owner $USER
    memory 1G
    cdrom $bsdiso
    disk /home/$USER/r5.qcow2 format qcow2
    interface tap55 { 
        switch "switch35"
        locked lladdr e8:8b:55:55:55:55
    }
}

NOTE: You will need to create necessary tap devices:

# cd /dev
# sh MAKEDEV tap{22,32,42,322,342,53,44,34,55}

We also create the qcow2 images and install the system:

$ vmctl create -s 20G $HOME/r2.qcow2
$ vmctl create -s 20G $HOME/r3.qcow2
$ vmctl create -s 20G $HOME/r4.qcow2
$ vmctl create -s 20G $HOME/r5.qcow2

In /etc/pf.conf?, we add a rule similar to the following for performing NAT:

match out on egress from !(egress:network) to any nat-to (egress:0)

Now all IPs from 10/8 will be NAT'd to our host's public IP address.

Then reload the ruleset:

host# pfctl -f /etc/pf.conf

We configure the proper interfaces on the host and sysctl.conf(5):

host# cat /etc/hostname.veb12
add vport21
up
host# cat /etc/hostname.veb23
up
host# cat /etc/hostname.veb24
up
host# cat /etc/hostname.veb34
up
host# cat /etc/hostname.veb35
up
host# cat /etc/hostname.vport11
inet 10.1.1.1 0xffff0000
up
host# cat /etc/hostname.vport21
inet 10.2.1.1 0xffff0000
!route add -inet 10/8 10.2.2.1
up
host# cat /etc/sysctl.conf
net.inet.ip.arpq.maxlen=1024
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

Next, we configure the virtual machines:

r2# cat /etc/hostname.vio0
inet 10.2.2.1 0xffff0000
!route add -inet 10.1/16 10.2.1.1
!route add -inet default 10.2.1.1
up
r2# cat /etc/hostname.vio1
inet 10.3.2.1 0xffffff00
!route add -inet 10.3/16 10.3.2.2
!route add -inet 10.5/16 10.3.2.2
up
r2# cat /etc/hostname.vio2
inet 10.4.2.1 0xffff0000
!route add -inet 10.4/16 10.4.4.1
up
r2# cat /etc/sysctl.conf
net.inet.ip.forwarding=1

r3# cat /etc/hostname.vio0
inet 10.3.2.2 0xffffff00
!route add -inet 10.2/16 10.3.2.1
!route add -inet default 10.3.2.1
up
r3# cat /etc/hostname.vio1
inet 10.3.4.2 0xffffff00
!route add -inet 10.4/16 10.3.4.1
up
r3# cat /etc/hostname.vio2
inet 10.5.3.1 0xffff0000
!route add -inet 10.5/16 10.5.5.1
up
r3# cat /etc/sysctl.conf
net.inet.ip.forwarding=1

r4# cat /etc/hostname.vio0
inet 10.4.4.1 0xffff0000
!route add -inet 10.2/16 10.4.2.1
!route add -inet default 10.4.2.1
up
r4# cat /etc/hostname.vio1
inet 10.3.4.1 0xffffff00
!route add -inet 10.3/16 10.3.4.2
!route add -inet 10.5/16 10.3.4.2
up
r4# cat /etc/sysctl.conf
net.inet.ip.forwarding=1

r5# cat /etc/hostname.vio0
inet 10.5.5.1 0xffff0000
!route add -inet 10.3/16 10.5.3.1
!route add -inet default 10.5.3.1
up
r5# cat /etc/sysctl.conf
net.inet.ip.forwarding=1

Note: Be careful to input the correct subnet masks. Interfaces connecting to 10.3/16 have a longer mask.

In this basic setup, we add static routes to allow packets to be forwarded inside the network.

Notice the routes that are chosen. If R5 runs:

r5# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=53 time=4.181 ms
64 bytes from 1.1.1.1: icmp_seq=1 ttl=53 time=4.353 ms

You can run tcpdump on each of the routers' interfaces to see which interfaces receive the packet:

# tcpdump -ne -i if0 'icmp and host 1.1.1.1'

Replace if0 with vio0, vio1, or vio2. You can also run:

To get from R5 to the internet, R5 passes through R3, R2, and R1, but does not pass through R4:

r3# tcpdump -ne -i vio0 'icmp and host 1.1.1.1'
tcpdump: listening on vio0, link-type EN10MB
08:47:30.937682 e8:8b:32:02:32:02 e8:8b:32:32:32:32 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request
08:47:30.941063 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]
08:47:31.937602 e8:8b:32:02:32:02 e8:8b:32:32:32:32 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request
08:47:31.941035 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]

r4# tcpdump -ne -i vio0 'icmp and host 1.1.1.1'
tcpdump: listening on vio0, link-type EN10MB
^C
0 packets received by filter
0 packets dropped by kernel

The route command can also tell you where packets will be sent:

r5# route -n get 1.1.1.1
   route to: 1.1.1.1
destination: 0.0.0.0
       mask: 0.0.0.0
    gateway: 10.5.3.1
  interface: vio0
 if address: 10.5.5.1
   priority: 8 (static)
      flags: <UP,GATEWAY,DONE,STATIC>
     use       mtu    expire
      34         0         0

Route priorities

By changing the routing tables, we can change the path that packets take.

First, notice how R3 can take two paths to reach R2. It can reach R2 directly, or it can first send packets to R4 which then forwards them to R2.

Let's have R3 send packets to R4 in order to reach R2. However, let's have R2 send packets to R3 directly. This will create asymmetric routing.

r2# cat /etc/hostname.vio0
inet 10.2.2.1 0xffff0000
!route add -inet 10.1/16 10.2.1.1
!route add -inet default 10.2.1.1
up
r2# cat /etc/hostname.vio1
inet 10.3.2.1 0xffffff00
!route add -inet -priority 2 -mpath 10.3/16 10.3.2.2
!route add -inet -priority 2 -mpath 10.5/16 10.3.2.2
up
r2# cat /etc/hostname.vio2
inet 10.4.2.1 0xffff0000
!route add -inet -mpath 10.3/16 10.4.4.1
!route add -inet 10.4/16 10.4.4.1
!route add -inet -mpath 10.5/16 10.4.4.1
up
r2# cat /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet.ip.multipath=1

r3# cat /etc/hostname.vio0
inet 10.3.2.2 0xffffff00
!route add -inet -mpath 10.2/16 10.3.2.1
!route add -inet -mpath default 10.3.2.1
up
r3# cat /etc/hostname.vio1
inet 10.3.4.2 0xffffff00
!route add -inet -priority 2 -mpath 10.4/16 10.3.4.1
!route add -inet -priority 2 -mpath default 10.3.4.1
up
r3# cat /etc/hostname.vio2
inet 10.5.3.1 0xffff0000
!route add -inet 10.5/16 10.5.5.1
up
r3# cat /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet.ip.multipath=1

r4# cat /etc/hostname.vio0
inet 10.4.4.1 0xffff0000
!route add -inet 10.1/16 10.4.2.1
!route add -inet -priority 2 -mpath 10.2/16 10.4.2.1
!route add -inet -priority 2 -mpath default 10.4.2.1
up
r4# cat /etc/hostname.vio1
inet 10.3.4.1 0xffffff00
!route add -inet 10.3/16 10.3.4.2
!route add -inet 10.5/16 10.3.4.2
!route add -inet -mpath default 10.3.4.2
up
r4# cat /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet.ip.multipath=1

r5# cat /etc/hostname.vio0
inet 10.5.5.1 0xffff0000
!route add -inet 10.3/16 10.5.3.1
!route add -inet default 10.5.3.1
up
r5# cat /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet.ip.multipath=1

Note: we must add the net.inet.ip.multipath sysctl in order to handle multipath routing.

Note: route priority 1 is reserved and cannot be used, so we use the next lowest priority, 2.

From route(8):

If no priority is specified, the kernel will set a priority depending on the RTF_STATIC flag to either RTP_STATIC or RTP_DEFAULT.

From ifconfig(8):

The default priority of new static routes added to the kernel is calculated by adding 8 (RTP_STATIC) to the interface priority.

On R5, try running:

r5# ping 1.1.1.1

You can see ping requests coming from R3 (10.4.2.1) to R4 vio0 (10.4.4.1), but no ping replies; and you can see ping replies coming from R2 (10.3.2.1) to R3 (10.3.2.2), but no ping requests:

r3# tcpdump -ne -i vio0 'icmp and host 1.1.1.1'
tcpdump: listening on vio0, link-type EN10MB
21:08:40.272715 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]
21:08:41.272550 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]
21:08:42.272799 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]

r4# tcpdump -ne -i vio0 'icmp and host 1.1.1.1'
tcpdump: listening on vio0, link-type EN10MB
21:08:44.270061 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request
21:08:45.270192 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request
21:08:46.270045 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request

Let's fix up the routes so that we have symmetric routing and multiple paths defined. The routers will choose routes based on their priorities.

r2# cat /etc/hostname.vio0
inet 10.2.2.1 0xffff0000
!route add -inet 10.1/16 10.2.1.1
!route add -inet default 10.2.1.1
up
r2# cat /etc/hostname.vio1 
inet 10.3.2.1 0xffffff00
!route add -inet -priority 2 -mpath 10.3/16 10.3.2.2
!route add -inet -priority 2 -mpath 10.5/16 10.3.2.2
up
r2# cat /etc/hostname.vio2 
inet 10.4.2.1 0xffff0000
!route add -inet -mpath 10.3/16 10.4.4.1
!route add -inet 10.4/16 10.4.4.1
!route add -inet -mpath 10.5/16 10.4.4.1
up
r2# cat /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet.ip.multipath=1

r3# cat /etc/hostname.vio0 
inet 10.3.2.2 0xffffff00
!route add -inet -mpath -priority 2 10.1/16 10.2.3.1
!route add -inet -mpath -priority 2 10.2/16 10.3.2.1
!route add -inet -mpath -priority 2 default 10.3.2.1
up
r3# cat /etc/hostname.vio1 
inet 10.3.4.2 0xffffff00
!route add -inet -mpath 10.1/16 10.3.4.1
!route add -inet -mpath 10.2/16 10.3.4.1
!route add -inet 10.4/16 10.3.4.1
!route add -inet -mpath default 10.3.4.1
up
r3# cat /etc/hostname.vio2 
inet 10.5.3.1 0xffff0000
!route add -inet 10.5/16 10.5.5.1
up
r3# cat /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet.ip.multipath=1

r4# cat /etc/hostname.vio0
inet 10.4.4.1 0xffff0000
!route add -inet 10.1/16 10.4.2.1
!route add -inet 10.2/16 10.4.2.1
!route add -inet default 10.4.2.1
up
r4# cat /etc/hostname.vio1 
inet 10.3.4.1 0xffffff00
!route add -inet 10.3/16 10.3.4.2
!route add -inet 10.5/16 10.3.4.2
up
r4# cat /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet.ip.multipath=1

r5# cat /etc/hostname.vio0
inet 10.5.5.1 0xffff0000
!route add -inet 10.3/16 10.5.3.1
!route add -inet default 10.5.3.1
up
r5# cat /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet.ip.multipath=1

Now, on R5, run ping:

r5# ping 1.1.1.1

Run tcpdump to notice that no packets pass through R4, and all packets pass through R3, as expected:

r3# tcpdump -ne -i vio0 'icmp and host 1.1.1.1'
tcpdump: listening on vio0, link-type EN10MB
22:52:10.308964 e8:8b:32:02:32:02 e8:8b:32:32:32:32 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request
22:52:10.312342 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]
22:52:11.308979 e8:8b:32:02:32:02 e8:8b:32:32:32:32 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request
22:52:11.312279 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]
22:52:12.308964 e8:8b:32:02:32:02 e8:8b:32:32:32:32 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request
22:52:12.312357 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]

r4# tcpdump -ne -i vio0 'icmp and host 1.1.1.1'
tcpdump: listening on vio0, link-type EN10MB
^C
0 packets received by filter
0 packets dropped by kernel

Now, let's see what happens when one of the links breaks. We will bring down the vio0 interface on R3, to break the link between R2 and R3:

r3# ifconfig vio0 down

On R2, we change route priorities:

r2# route change -inet -priority 10 10.5/16 10.3.2.1
change net 10.5/16: gateway 10.3.2.1
r2# route change -inet -priority 2 10.5/16 10.4.4.1
change net 10.5/16: gateway 10.4.4.1

On R5, ping should still work:

r5# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=52 time=4.831 ms
64 bytes from 1.1.1.1: icmp_seq=1 ttl=52 time=4.349 ms

However, it now takes a different route. Notice that R4 now receives all packets:

r4# tcpdump -ne -i vio0 'icmp and host 1.1.1.1'
tcpdump: listening on vio0, link-type EN10MB
00:25:59.811394 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request
00:25:59.814762 e8:8b:42:42:42:42 e8:8b:44:44:44:44 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]
00:26:00.811231 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request
00:26:00.814731 e8:8b:42:42:42:42 e8:8b:44:44:44:44 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]
00:26:01.811288 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request
00:26:01.814664 e8:8b:42:42:42:42 e8:8b:44:44:44:44 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]