OSPF and IP Unnumbered over different types of links

This post is about using OSPF to link routers where link endpoints are on completely different IP networks.

Below is the topology used in this post to capture the main link types we'll come up against in the CCIE lab - Frame Relay, PPP and Ethernet.

We shall start with the most straight forward - direct PPP over serial between R1 and R3  Each router is using Loopback0 as the only IP address on the router which is bound to Serial0/1


R1

hostname R1
interface Loopback0
 ip address 1.1.1.1 255.255.255.0
 ip ospf 1 area 0
!
interface Serial0/1
 ip unnumbered Loopback0
 encapsulation ppp
 ip ospf 1 area 0
 clock rate 2000000

 no shutdown
!

R3

hostname R3
interface Loopback0
 ip address 3.3.3.3 255.255.255.0
 ip ospf 1 area 0
!
interface Serial0/1
 ip unnumbered Loopback0
 encapsulation ppp
 ip ospf 1 area 0

 shutdown
!

Let's enable R3 S0/1 and verify that OSPF comes up



R3#deb ip ospf adj
OSPF adjacency events debugging is on
R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#int s0/1
R3(config-if)#no shut
R3(config-if)#
*Mar  1 00:02:19.543: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
*Mar  1 00:02:20.555: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, changed state to up
*Mar  1 00:02:20.559: OSPF: Interface Serial0/1 going Up
*Mar  1 00:02:20.587: OSPF: 2 Way Communication to 1.1.1.1 on Serial0/1, state 2WAY
*Mar  1 00:02:20.587: OSPF: Send DBD to 1.1.1.1 on Serial0/1 seq 0x1DAE opt 0x52 flag 0x7 len 32
*Mar  1 00:02:20.587: OSPF: Rcv DBD from 1.1.1.1 on Serial0/1 seq 0x105C opt 0x52 flag 0x7 len 32  mtu 1500 state EXSTART
*Mar  1 00:02:20.587: OSPF: First DBD and we are not SLAVE
*Mar  1 00:02:20.591: OSPF: Rcv DBD from 1.1.1.1 on Serial0/1 seq 0x1DAE opt 0x52 flag 0x2 len 52  mtu 1500 state EXSTART
*Mar  1 00:02:20.591: OSPF: NBR Negotiation Done. We are the MASTER
*Mar  1 00:02:20.591: OSPF: Send DBD to 1.1.1.1 on Serial0/1 seq 0x1DAF opt 0x52 flag 0x3 len 52
*Mar  1 00:02:20.591: OSPF: Rcv DBD from 1.1.1.1 on Serial0/1 seq 0x1DAF opt 0x52 flag 0x0 len 32  mtu 1500 state EXCHANGE
*Mar  1 00:02:20.595: OSPF: Send DBD to 1.1.1.1 on Serial0/1 seq 0x1DB0 opt 0x52 flag 0x1 len 32
*Mar  1 00:02:20.595: OSPF: Send LS REQ to 1.1.1.1 length 12 LSA count 1
*Mar  1 00:02:20.599: OSPF: Rcv LS REQ from 1.1.1.1 on Serial0/1 length 36 LSA count 1
*Mar  1 00:02:20.599: OSPF: Send UPD to 1.1.1.1 on Serial0/1 length 40 LSA count 1
*Mar  1 00:02:20.599: OSPF: Rcv DBD from 1.1.1.1 on Serial0/1 seq 0x1DB0 opt 0x52 flag 0x0 len 32  mtu 1500 state EXCHANGE
*Mar  1 00:02:20.599: OSPF: Exchange Done with 1.1.1.1 on Serial0/1
*Mar  1 00:02:20.599: OSPF: Rcv LS UPD from 1.1.1.1 on Serial0/1 length 64 LSA count 1
*Mar  1 00:02:20.603: OSPF: Synchronized with 1.1.1.1 on Serial0/1, state FULL
*Mar  1 00:02:20.603: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.1 on Serial0/1 from LOADING to FULL, Loading Done
*Mar  1 00:02:21.059: OSPF: Build router LSA for area 0, router ID 3.3.3.3, seq 0x80000002, process 1
*Mar  1 00:02:21.075: OSPF: Rcv LS UPD from 1.1.1.1 on Serial0/1 length 76 LSA count 1


So no problems here, everything came up, let's validate our IP interface and OSPF interface/neighbors

R3(config-if)#do sh ip interface Serial0/1 | i Interface|Peer
  Interface is unnumbered. Using address of Loopback0 (3.3.3.3)
  Peer address is 1.1.1.1
R3(config-if)#do sh ip ospf int brief
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Se0/1        1     0               0.0.0.0/0          64    P2P   1/1
Lo0          1     0               3.3.3.3/24         1     LOOP  0/0
R3(config-if)#do sh ip ospf neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1           0   FULL/  -        00:00:35    1.1.1.1         Serial0/1

R1#sh ip interface Serial0/1 | i Interface|Peer
  Interface is unnumbered. Using address of Loopback0 (1.1.1.1)
  Peer address is 3.3.3.3
R1#sh ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
3.3.3.3           0   FULL/  -        00:00:39    3.3.3.3         Serial0/1
R1#sh ip ospf int bri
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Se0/1        1     0               0.0.0.0/0          64    P2P   1/1
Lo0          1     0               1.1.1.1/24         1     LOOP  0/0


Okay, this looks good, lets check the routing table


R1#sh ip route ospf



Hmm, we cant see anything here, we should have R3 Lo0's stub network in our ospf database..

R1#sh ip ospf data router 3.3.3.3 | s Stub
    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 3.3.3.3
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metrics: 1


We do but it's not being installed in the routing table, is something else already there?

R1#sh ip route 3.3.3.3
Routing entry for 3.3.3.3/32
  Known via "connected", distance 0, metric 0 (connected, via interface)
  Routing Descriptor Blocks:
  * directly connected, via Serial0/1
      Route metric is 0, traffic share count is 1


By default, PPP installs a connected route to our peer device, not necessarily a bad thing but we would prefer to see this route originated by ospf, so lets fix that on both sides



R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#int s0/1
R3(config-if)#no peer neighbor-route
R3(config-if)#end
 

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int s0/1
R1(config-if)#no peer neighbor-route

Making these changes doesn't cause the static neighbor route to be removed, so we need to bounce the link to make that happen

R1(config-if)#shut
*Mar  1 00:16:45.727: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/1 from FULL to DOWN, Neighbor Down: Interface down or detached
*Mar  1 00:16:47.715: %LINK-5-CHANGED: Interface Serial0/1, changed state to administratively down
*Mar  1 00:16:48.715: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, changed state to down
R1(config-if)#no shut
*Mar  1 00:16:53.495: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
*Mar  1 00:16:54.531: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, changed state to up
*Mar  1 00:16:54.603: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/1 from LOADING to FULL, Loading Done

R1(config-if)#do sh ip int s0/1 | i Interface|Peer
  Interface is unnumbered. Using address of Loopback0 (1.1.1.1)
  Peer address is 3.3.3.3
 

Let's see how if the route comes through from ospf and is installed into the routing table

R1(config-if)#do sh ip route ospf
     3.0.0.0/32 is subnetted, 1 subnets
O       3.3.3.3 [110/65] via 3.3.3.3, 00:00:00, Serial0/1


R1(config-if)#do ping 3.3.3.3 so 1.1.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms


OSPF IP Unnumbered over Frame Relay


Between R1 and R2 we have a Frame Relay link, we are going to take advantage of implementing PPP over FR to make this happen


On R1 and R2 create the virtual-template required for PPPoFR to operate, again it's an unnumbered link associated with loopback0


R1

interface Virtual-Template1
 ip unnumbered Loopback0
 ip ospf 1 area 0
 no peer neighbor-route
!

R2

hostname R2
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
 ip ospf 1 area 0
!

interface Virtual-Template1
 ip unnumbered Loopback0
 ip ospf 1 area 0
 no peer neighbor-route
!

Then we configure the serial interface for frame relay encapsulation and bind the virtual-template to the FR PVC.

R1

interface Serial0/0
 encapsulation frame-relay
 frame-relay interface-dlci 102 ppp Virtual-Template1
 no frame-relay inverse-arp
 no shutdown
!

R2

interface Serial0/0
 encapsulation frame-relay
 frame-relay interface-dlci 201 ppp Virtual-Template1
 no frame-relay inverse-arp
 no shutdown
!


After a short period of time R1/R2s PVC will come up and the PPP session will go active and then OSPF should start

R1#

*Mar  1 00:35:51.403: %LINK-3-UPDOWN: Interface Serial0/0, changed state to up
*Mar  1 00:36:02.407: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, changed state to up
*Mar  1 00:36:51.423: %LINK-3-UPDOWN: Interface Virtual-Access2, changed state to up
*Mar  1 00:36:53.515: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to up
*Mar  1 00:36:53.575: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Virtual-Access2 from LOADING to FULL, Loading Done


R1#sh ip ospf int brie
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Vi2          1     0               0.0.0.0/0          1     P2P   1/1
Vt1          1     0               0.0.0.0/0          1     DOWN  0/0
Se0/1        1     0               0.0.0.0/0          64    P2P   1/1
Lo0          1     0               1.1.1.1/24         1     LOOP  0/0

Of potential interest, the PPPoFR link has a lower ospf cost than the native PPP link


R1#sh ip ospf nei


Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2           0   FULL/  -        00:00:37    2.2.2.2         Virtual-Access2
3.3.3.3           0   FULL/  -        00:00:38    3.3.3.3         Serial0/1

R1#sh ip route ospf
     2.0.0.0/32 is subnetted, 1 subnets
O       2.2.2.2 [110/2] via 2.2.2.2, 00:02:30, Virtual-Access2
     3.0.0.0/32 is subnetted, 1 subnets
O       3.3.3.3 [110/65] via 3.3.3.3, 00:22:27, Serial0/1


The final link type we're working with is ethernet between R2 and R3

SW1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW1(config)#int range fa1/2 - 3
SW1(config-if-range)#switchport mode access
SW1(config-if-range)#switchport access vlan 23

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int fa0/0
R2(config-if)#ip unnumbered lo0
Point-to-point (non-multi-access) interfaces only


Okay, that's not so good, I guess we can go back to our friend PPP again and go for PPPoE (R2 will be our PPPoE Client)

R2(config-if)#int di1
R2(config-if)#ip unnumbered lo0

R2(config-if)#encapsulation ppp
R2(config-if)#no peer neighbor-route
R2(config-if)#dialer pool 1
R2(config-if)#dialer persistent
R2(config-if)#ip ospf 1 area 0
R2(config-if)#int fa0/0
R2(config-if)#pppoe-client dial-pool-number 1
R2(config-if)#no shut

Let's set up R3 as the PPPoE Server


R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#int virtual-template 1
R3(config-if)#ip unnumbered lo0
R3(config-if)#no peer neighbor-route
R3(config-if)#ip ospf 1 area 0
R3(config-if)#int fa0/0
R3(config-if)#pppoe enable
R3(config-if)#bba-group pppoe global
R3(config-bba-group)#virtual-template 1
R3(config-bba-group)#int fa0/0
R3(config-if)#no shut


After a short period of time we can see that the PPPoE link comes up but eventially we see this:


*Mar  1 00:59:06.723: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Virtual-Access2.1 from EXSTART to DOWN, Neighbor Down: Too many retransmissions


A typical cause of this is an MTU mismatch, lets see if this is the case:




R3#sh ip int vi2.1 | i MTU
  MTU is 1492 bytes


Remember the PPPoE overhead is 8 bytes....



R2#sh ip int di1 | i MTU
  MTU is 1500 bytes


Okay, so that didn't happen here, let's fix this up



R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int di1
R2(config-if)#ip mtu 1492
R2(config-if)#end

We could have also fixed this with ip ospf mtu-ignore... After a few seconds on R2 we see:

*Mar  1 01:00:35.827: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Dialer1 from LOADING to FULL, Loading Done


So there you go, a few different ways of implementing OSPF adjacencies between routers using peers with different IP addresses and subnets, just remember that different link types will have different link cost metrics applied..

Stateful Network Address Translation

Stateful NAT is something that can be tied to HSRP in order to ensure that session based flows are maintained when there is a link/router failure by ensuring HSRP peers share their NAT database.

This 4 router topology below will be used to demonstrate how it can be implemented.

Starting Configurations:

R1
hostname R1
interface FastEthernet0/0
 ip address 10.1.123.1 255.255.255.0
 no shut
!
ip route 0.0.0.0 0.0.0.0 10.1.123.23

R2
hostname R2
interface FastEthernet0/0
 ip address 10.1.123.2 255.255.255.0
 standby 1 ip 10.1.123.23
 standby 1 priority 105
 standby 1 preempt
 standby 1 name HSRP-123
 standby 1 track FastEthernet0/1
 no shut
!
interface FastEthernet0/1
 ip address 20.1.234.2 255.255.255.0
 standby 1 ip 20.1.234.23
 standby 1 priority 105
 standby 1 preempt
 standby 1 track FastEthernet0/0
 no shut
!

R3
hostname R3
interface FastEthernet0/0
 ip address 10.1.123.3 255.255.255.0
 standby 1 ip 10.1.123.23
 standby 1 preempt
 standby 1 name HSRP-123
 no shut
!
interface FastEthernet0/1
 ip address 20.1.234.3 255.255.255.0
 standby 1 ip 20.1.234.23
 standby 1 preempt
 no shut
!

R4
hostname R4
service tcp-small-servers
interface FastEthernet0/1
 ip address 20.1.234.4 255.255.255.0
 no shut
!
ip route 0.0.0.0 0.0.0.0 20.1.234.23

R2 by default will be the active HSRP router on Fa0/0 and Fa0/1 due to the priority being set to 105.  Should the corresponding interface go down, the priority will be decremented to 95 meaning R3 will take over

R2#sh standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa0/0       1    105 P Active  local           10.1.123.3      10.1.123.23
Fa0/1       1    105 P Active  local           20.1.234.3      20.1.234.23

R3#sh standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa0/0       1    100 P Standby 10.1.123.2      local           10.1.123.23
Fa0/1       1    100 P Standby 20.1.234.2      local           20.1.234.23

So lets test it (we aren't doing NAT yet but both R1 and R4 have default routes pointing to the Virtual-IP on their LAN segments)

R1#ping 20.1.234.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 20.1.234.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/8/12 ms

Okay, lets bring down R2 Fa0/1 - this should cause R3 on both Fa0/0 and Fa0/1 to take over.  R3 Fa0/1 since R2's hellos will have stopped and, R3 Fa0/0 since R2 will decrement its Fa0/0 priority to be less than R3s and we have preempt enabled.

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int fa0/1
R2(config-if)#shut
*Mar  1 00:23:56.335: %HSRP-5-STATECHANGE: FastEthernet0/1 Grp 1 state Active -> Init
*Mar  1 00:23:58.343: %LINK-5-CHANGED: Interface FastEthernet0/1, changed state to administratively down
*Mar  1 00:23:59.343: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down

R2(config-if)#do sh standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa0/0       1    95  P Standby 10.1.123.3      local           10.1.123.23
Fa0/1       1    105 P Init    unknown         unknown         20.1.234.23


R3#sh standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa0/0       1    100 P Active  local           10.1.123.2      10.1.123.23
Fa0/1       1    100 P Active  local           unknown         20.1.234.23


R1#ping 20.1.234.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 20.1.234.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/8/12 ms

Okay, so HSRP is working as we want it.  Let's introduce stateful NAT.

In order for us to use stateful nat we need to configure a stateful nat instance on our routers and we need to give each router a unique id.  In this example I'm just using the router number as the id.

R2(config-if)#ip nat stateful id 2

next we specify the name of the HSRP group that we are tracking, in this case it's "HSRP-123" which is we configured on Fa0/0 on R2 and R3

R2(config-ipnat-snat)#redundancy HSRP-123

then we need to set a mapping-id which is global for synchronisation.

R2(config-ipnat-snat-red)#mapping-id 23

Now we create a typical dynamic NAT entry.  I tend to find that using a route-map with an ACL works more often than directly using the ACL

R2(config)#ip access-list standard R1
R2(config-std-nacl)#permit host 10.1.123.1
R2(config-std-nacl)#exit
R2(config)#route-map NAT-R1
R2(config-route-map)#match ip address R1
R2(config-route-map)#exit

Then we set up our NAT pool and then the NAT statement, notice we set the mapping-id to be the same as the one in our stateful NAT entry

R2(config)#ip nat pool NAT-POOL 222.222.222.222 222.222.222.222 prefix-length 24
R2(config)#ip nat inside source route-map NAT-R1 pool NAT-POOL mapping-id 23

Now set the inside and outside nat interfaces

R2(config)#int fa0/0
R2(config-if)#ip nat inside
R2(config)#int fa0/1
R2(config-if)#ip nat outside
R2(config)#end

The stateful NAT configuration is the exactly the same procedure for R3, however the stateful id must be unique.

R3(config-if)#ip nat stateful id 3
R3(config-ipnat-snat)#redundancy HSRP-123
R3(config-ipnat-snat-red)#mapping-id 23
R3(config)#ip access-list standard R1
R3(config-std-nacl)#permit host 10.1.123.1
R3(config-std-nacl)#exit
R3(config)#route-map NAT-R1
R3(config-route-map)#match ip address R1
R3(config-route-map)#exit
R3(config)#ip nat pool NAT-POOL 222.222.222.222 222.222.222.222 prefix-length 24
R3(config)#ip nat inside source route-map NAT-R1 pool NAT-POOL mapping-id 23
R3(config)#int fa0/0
R3(config-if)#ip nat inside
R3(config)#int fa0/1
R3(config-if)#ip nat outside
R3(config)#end

Shortly after that we'll see a message about stateful NAT converging...

*Mar  1 00:21:27.587: %SNAT-5-PROCESS: Id 3, System start converging
*Mar  1 00:21:46.271: %SNAT-5-PROCESS: Id 3, System fully converged

We can check that redundancy is correct by invoking "show ip snat distributed"

R2#sh ip snat distributed

Stateful NAT Connected Peers

SNAT: Mode IP-REDUNDANCY :: ACTIVE
    : State READY
    : Local Address 10.1.123.2
    : Local NAT id 2
    : Peer Address 10.1.123.3
    : Peer NAT id 3
    : Mapping List 23

R3#sh ip snat distributed

Stateful NAT Connected Peers

SNAT: Mode IP-REDUNDANCY :: STANDBY
    : State READY
    : Local Address 10.1.123.3
    : Local NAT id 3
    : Peer Address 10.1.123.2
    : Peer NAT id 2
    : Mapping List 23

So let's test things:

R4#deb ip icmp
ICMP packet debugging is on

R1#ping 20.1.234.4 repeat 1

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 20.1.234.4, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 12/12/12 ms

R4#
*Mar  1 00:30:45.951: ICMP: echo reply sent, src 20.1.234.4, dst 222.222.222.222

R4 thinks it was pinged by 222.222.222.222 so NAT is working fine.

If we check the translation entries we can see that R2 and R3 are in alignment

R2#sh ip nat tr
Pro Inside global      Inside local       Outside local      Outside global
icmp 222.222.222.222:8 10.1.123.1:8       20.1.234.4:8       20.1.234.4:8

R3#sh ip nat tr
Pro Inside global      Inside local       Outside local      Outside global
icmp 222.222.222.222:8 10.1.123.1:8       20.1.234.4:8       20.1.234.4:8

So let's test that the stateful NAT is working.  The easiest way to verify it is to have a TCP stream between R1 and R4.  R4 has service tcp-small-servers in its configuration which means the chargen service on port 19 is listening.

R1#telnet 20.1.234.4 19
Trying 20.1.234.4, 19 ... Open
 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefg
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefgh
"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghi
#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghij
$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijk
........

R4 will keep sending traffic forever...


So lets check the NAT tables

R2#sh ip nat tr
Pro Inside global      Inside local       Outside local      Outside global
tcp 222.222.222.222:61143 10.1.123.1:61143 20.1.234.4:19     20.1.234.4:19

R3#sh ip nat tr
Pro Inside global      Inside local       Outside local      Outside global
tcp 222.222.222.222:61143 10.1.123.1:61143 20.1.234.4:19     20.1.234.4:19

Let's shutdown interface fa0/1 on R2 which should cause R3 to takeover in HSRP and cause a stateful NAT takeover as well..

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int fa0/1
R2(config-if)#shut
SNAT: interface FastEthernet0/1 with address 20.1.234.2 is down
R2(config-if)#
%HSRP-5-STATECHANGE: FastEthernet0/1 Grp 1 state Active -> Init
%TRACKING-5-STATE: 2 interface Fa0/1 line-protocol Up->Down
%HSRP-5-STATECHANGE: FastEthernet0/0 Grp 1 state Active -> Speak
%SNAT-5-PROCESS: Id 2, System start converging
SNAT (Receive): CONVERGENCE Message from Router-Id: 3 for  Router-Id: 0's entries
%SNAT-5-PROCESS: Id 2, System fully converged
%LINK-5-CHANGED: Interface FastEthernet0/1, changed state to administratively down
%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
%HSRP-5-STATECHANGE: FastEthernet0/0 Grp 1 state Speak -> Standby

%HSRP-5-STATECHANGE: FastEthernet0/1 Grp 1 state Standby -> Active
SNAT Redundancy(prempt): we are in middle of FOR_ALL HSRP-123 HSRP-123
%HSRP-5-STATECHANGE: FastEthernet0/0 Grp 1 state Standby -> Active
%SNAT-5-PROCESS: Id 3, System start converging
%SNAT-5-PROCESS: Id 3, System fully converged
SNAT (Receive): CONVERGENCE Message from Router-Id: 2 for  Router-Id: 0's entries

R2(config-if)#do sh standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa0/0       1    95  P Standby 10.1.123.3      local           10.1.123.23
Fa0/1       1    105 P Init    unknown         unknown         20.1.234.23
 
R2(config-if)#do sh ip nat tr
Pro Inside global      Inside local       Outside local      Outside global
tcp 222.222.222.222:61143 10.1.123.1:61143 20.1.234.4:19     20.1.234.4:19

R2(config-if)#do sh ip snat dist

Stateful NAT Connected Peers

SNAT: Mode IP-REDUNDANCY :: STANDBY
    : State READY
    : Local Address 10.1.123.2
    : Local NAT id 2
    : Peer Address 10.1.123.3
    : Peer NAT id 3
    : Mapping List 23

R2 is not the active NAT device here...


R3#sh standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa0/0       1    100 P Active  local           10.1.123.2      10.1.123.23
Fa0/1       1    100 P Active  local           unknown         20.1.234.23

R3#sh ip nat tr
Pro Inside global      Inside local       Outside local      Outside global
tcp 222.222.222.222:61143 10.1.123.1:61143 20.1.234.4:19     20.1.234.4:19

R3#sh ip snat dist

Stateful NAT Connected Peers

SNAT: Mode IP-REDUNDANCY :: ACTIVE
    : State READY
    : Local Address 10.1.123.3
    : Local NAT id 3
    : Peer Address 10.1.123.2
    : Peer NAT id 2
    : Mapping List 23


And the R1/R4 NATed session is still in operation

WXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?
XYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@
YZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@A
Z[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@AB
[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABC
\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCD
]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDE

BGP with Unequal Cost paths

This posting covers how we can handle load sharing with unequal cost paths between two BGP Autonmous Systems.

This 6 router topology has two BGP Autonomous Systems (AS 123 and AS 456) with two exit points between them, one has a 128kbps path (R2-R4), and the other a 192kbps path (R3-R5)

 Below are our initial configurations:

R1
hostname R1
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
 ip ospf 1 area 123
!
interface FastEthernet0/0
 ip address 10.1.111.1 255.255.255.0
 ip ospf 1 area 123
 no shutdown
!
router bgp 123
 no bgp default ipv4-unicast
 neighbor 2.2.2.2 remote-as 123
 neighbor 2.2.2.2 update-source Loopback0
 neighbor 3.3.3.3 remote-as 123
 neighbor 3.3.3.3 update-source Loopback0
 !
 address-family ipv4
  neighbor 2.2.2.2 activate
  neighbor 3.3.3.3 activate
  no auto-summary
  no synchronization
  network 1.1.1.1 mask 255.255.255.255
 exit-address-family
!

R2
hostname R2
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
 ip ospf 1 area 123
!
interface FastEthernet0/0
 ip address 10.1.111.2 255.255.255.0
 ip ospf 1 area 123
 no shutdown
!
interface Serial0/0
 no ip address
 encapsulation frame-relay
 no frame-relay inverse-arp
 no shutdown
!
interface Serial0/0.24 point-to-point
 bandwidth 128
 ip address 10.1.24.2 255.255.255.0
 snmp trap link-status
 frame-relay interface-dlci 204
!
router bgp 123
 no bgp default ipv4-unicast
 neighbor 1.1.1.1 remote-as 123
 neighbor 1.1.1.1 update-source Loopback0
 neighbor 3.3.3.3 remote-as 123
 neighbor 3.3.3.3 update-source Loopback0
 neighbor 10.1.24.4 remote-as 456
 !
 address-family ipv4
  neighbor 1.1.1.1 activate
  neighbor 1.1.1.1 next-hop-self
  neighbor 3.3.3.3 activate
  neighbor 3.3.3.3 next-hop-self
  neighbor 10.1.24.4 activate
  no auto-summary
  no synchronization
  network 2.2.2.2 mask 255.255.255.255
 exit-address-family
!

R3
hostname R3
interface Loopback0
 ip address 3.3.3.3 255.255.255.255
 ip ospf 1 area 123
!
interface FastEthernet0/0
 ip address 10.1.111.3 255.255.255.0
 ip ospf 1 area 123
 no shutdown
!
interface Serial0/0
 no ip address
 encapsulation frame-relay
 no frame-relay inverse-arp
 no shutdown
!
interface Serial0/0.35 point-to-point
 bandwidth 192
 ip address 10.1.35.3 255.255.255.0
 snmp trap link-status
 frame-relay interface-dlci 305
!
router bgp 123
 no bgp default ipv4-unicast
 neighbor 1.1.1.1 remote-as 123
 neighbor 1.1.1.1 update-source Loopback0
 neighbor 2.2.2.2 remote-as 123
 neighbor 2.2.2.2 update-source Loopback0
 neighbor 10.1.35.5 remote-as 456
 !
 address-family ipv4
  neighbor 1.1.1.1 activate
  neighbor 1.1.1.1 next-hop-self
  neighbor 2.2.2.2 activate
  neighbor 2.2.2.2 next-hop-self
  neighbor 10.1.35.5 activate
  no auto-summary
  no synchronization
  network 3.3.3.3 mask 255.255.255.255
 exit-address-family
!

R4
hostname R4
interface Loopback0
 ip address 4.4.4.4 255.255.255.255
 ip ospf 1 area 456
!
interface FastEthernet0/0
 ip address 10.1.222.4 255.255.255.0
 ip ospf 1 area 456
 no shutdown
!
interface Serial0/0
 no ip address
 encapsulation frame-relay
 no frame-relay inverse-arp
 no shutdown
!
interface Serial0/0.42 point-to-point
 bandwidth 128
 ip address 10.1.24.4 255.255.255.0
 snmp trap link-status
 frame-relay interface-dlci 402
!
router bgp 456
 no bgp default ipv4-unicast
 neighbor 5.5.5.5 remote-as 456
 neighbor 5.5.5.5 update-source Loopback0
 neighbor 6.6.6.6 remote-as 456
 neighbor 6.6.6.6 update-source Loopback0
 neighbor 10.1.24.2 remote-as 123
 !
 address-family ipv4
  neighbor 5.5.5.5 activate
  neighbor 5.5.5.5 next-hop-self
  neighbor 6.6.6.6 activate
  neighbor 6.6.6.6 next-hop-self
  neighbor 10.1.24.2 activate
  no auto-summary
  no synchronization
  network 4.4.4.4 mask 255.255.255.255
 exit-address-family
!

R5

hostname R5
interface Loopback0
 ip address 5.5.5.5 255.255.255.255
 ip ospf 1 area 456
!
interface FastEthernet0/0
 ip address 10.1.222.5 255.255.255.0
 ip ospf 1 area 456
 no shutdown
!
interface Serial0/0
 no ip address
 encapsulation frame-relay
 no frame-relay inverse-arp
 no shutdown
!
interface Serial0/0.53 point-to-point
 bandwidth 192
 ip address 10.1.35.5 255.255.255.0
 snmp trap link-status
 frame-relay interface-dlci 503
!
router bgp 456
 no bgp default ipv4-unicast
 neighbor 4.4.4.4 remote-as 456
 neighbor 4.4.4.4 update-source Loopback0
 neighbor 6.6.6.6 remote-as 456
 neighbor 6.6.6.6 update-source Loopback0
 neighbor 10.1.35.3 remote-as 123
 !
 address-family ipv4
  neighbor 4.4.4.4 activate
  neighbor 4.4.4.4 next-hop-self
  neighbor 6.6.6.6 activate
  neighbor 6.6.6.6 next-hop-self
  neighbor 10.1.35.3 activate
  no auto-summary
  no synchronization
  network 5.5.5.5 mask 255.255.255.255
 exit-address-family
!



R6
hostname R6
interface Loopback0
 ip address 6.6.6.6 255.255.255.255
 ip ospf 1 area 456
!
interface FastEthernet0/0
 ip address 10.1.222.6 255.255.255.0
 ip ospf 1 area 456
 no shutdown
!
router bgp 456
 no bgp default ipv4-unicast
 neighbor 4.4.4.4 remote-as 456
 neighbor 4.4.4.4 update-source Loopback0
 neighbor 5.5.5.5 remote-as 456
 neighbor 5.5.5.5 update-source Loopback0
 !
 address-family ipv4
  neighbor 4.4.4.4 activate
  neighbor 5.5.5.5 activate
  no auto-summary
  no synchronization
  network 6.6.6.6 mask 255.255.255.255
 exit-address-family
!

Lets check the bgp table and routing tables on R1 and R6

R1#sh ip bgp | b Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 1.1.1.1/32       0.0.0.0                  0         32768 i
r>i2.2.2.2/32       2.2.2.2                  0    100      0 i
r>i3.3.3.3/32       3.3.3.3                  0    100      0 i
* i4.4.4.4/32       3.3.3.3                  0    100      0 456 i
*>i                 2.2.2.2                  0    100      0 456 i
*>i5.5.5.5/32       2.2.2.2                  0    100      0 456 i
* i                 3.3.3.3                  0    100      0 456 i
*>i6.6.6.6/32       2.2.2.2                  0    100      0 456 i
* i                 3.3.3.3                  0    100      0 456 i
R1#sh ip route bgp
     4.0.0.0/32 is subnetted, 1 subnets
B       4.4.4.4 [200/0] via 2.2.2.2, 00:03:02
     5.0.0.0/32 is subnetted, 1 subnets
B       5.5.5.5 [200/0] via 2.2.2.2, 00:03:02
     6.0.0.0/32 is subnetted, 1 subnets
B       6.6.6.6 [200/0] via 2.2.2.2, 00:01:03

R6#sh ip bgp | b Network
   Network          Next Hop            Metric LocPrf Weight Path
*>i1.1.1.1/32       4.4.4.4                  0    100      0 123 i
* i                 5.5.5.5                  0    100      0 123 i
*>i2.2.2.2/32       4.4.4.4                  0    100      0 123 i
* i                 5.5.5.5                  0    100      0 123 i
*>i3.3.3.3/32       4.4.4.4                  0    100      0 123 i
* i                 5.5.5.5                  0    100      0 123 i
r>i4.4.4.4/32       4.4.4.4                  0    100      0 i
r>i5.5.5.5/32       5.5.5.5                  0    100      0 i
*> 6.6.6.6/32       0.0.0.0                  0         32768 i
R6#sh ip route bgp
     1.0.0.0/32 is subnetted, 1 subnets
B       1.1.1.1 [200/0] via 4.4.4.4, 00:01:28
     2.0.0.0/32 is subnetted, 1 subnets
B       2.2.2.2 [200/0] via 4.4.4.4, 00:01:28
     3.0.0.0/32 is subnetted, 1 subnets
B       3.3.3.3 [200/0] via 4.4.4.4, 00:01:28

Right now we can see only one BGP route is installed in our routing table because that is the default behaviour.  Right now we are interested in R1 and R6 being able to use the other exit points from their autonomous systems, so lets enable bgp multipath (since R1 and R6 only have ibgp peers we need to enable this for ibgp)

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#router bgp 123
R1(config-router)#address-family ipv4
R1(config-router-af)#maximum-paths ibgp 2
R1(config-router-af)#end

R1#sh ip route bgp
     4.0.0.0/32 is subnetted, 1 subnets
B       4.4.4.4 [200/0] via 3.3.3.3, 00:00:30
                [200/0] via 2.2.2.2, 00:04:55
     5.0.0.0/32 is subnetted, 1 subnets
B       5.5.5.5 [200/0] via 3.3.3.3, 00:00:30
                [200/0] via 2.2.2.2, 00:04:55
     6.0.0.0/32 is subnetted, 1 subnets
B       6.6.6.6 [200/0] via 3.3.3.3, 00:00:30
                [200/0] via 2.2.2.2, 00:02:57

R6#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R6(config)#router bgp 456
R6(config-router)#address-family ipv4
R6(config-router-af)#maximum-paths ibgp 2
R6(config-router-af)#end
R6#
R6#sh ip route bgp
     1.0.0.0/32 is subnetted, 1 subnets
B       1.1.1.1 [200/0] via 5.5.5.5, 00:00:06
                [200/0] via 4.4.4.4, 00:03:55
     2.0.0.0/32 is subnetted, 1 subnets
B       2.2.2.2 [200/0] via 5.5.5.5, 00:00:06
                [200/0] via 4.4.4.4, 00:03:55
     3.0.0.0/32 is subnetted, 1 subnets
B       3.3.3.3 [200/0] via 5.5.5.5, 00:00:06
                [200/0] via 4.4.4.4, 00:03:55

Okay, now R1 and R6 are able to utilise both exit points however to get maximum efficiency of these links it would be worthwhile to share the load across the links relatively fairly based on the static link bandwidths.

We do this by:

1. activating the bgp dmzlink-bw function
2. Ensuring that we send our extended bgp communities to our IBGP peers
3. Associate the dmzlink-bw with our EBGP neighbor

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#router bgp 123
R2(config-router)#address-family ipv4 unicast
R2(config-router-af)#bgp dmzlink-bw
R2(config-router-af)#neighbor 1.1.1.1 send-community extended
R2(config-router-af)#neighbor 3.3.3.3 send-community extended
R2(config-router-af)#neighbor 10.1.24.4 dmzlink-bw
R2(config-router-af)#end

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#router bgp 123
R3(config-router)#address-family ipv4 unicast
R3(config-router-af)#bgp dmzlink-bw
R3(config-router-af)#neighbor 1.1.1.1 send-community extended
R3(config-router-af)#neighbor 2.2.2.2 send-community extended
R3(config-router-af)#neighbor 10.1.35.5 dmzlink-bw
R3(config-router-af)#end

R1#sh ip bgp 6.6.6.6
BGP routing table entry for 6.6.6.6/32, version 16
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Multipath: iBGP
Flag: 0x8820
  Not advertised to any peer
  456
    3.3.3.3 (metric 11) from 3.3.3.3 (3.3.3.3)
      Origin IGP, metric 0, localpref 100, valid, internal, multipath
      DMZ-Link Bw 24 kbytes
  456
    2.2.2.2 (metric 11) from 2.2.2.2 (2.2.2.2)
      Origin IGP, metric 0, localpref 100, valid, internal, multipath, best
      DMZ-Link Bw 16 kbytes


R1#sh ip route 6.6.6.6
Routing entry for 6.6.6.6/32
  Known via "bgp 123", distance 200, metric 0
  Tag 456, type internal
  Last update from 2.2.2.2 00:00:05 ago
  Routing Descriptor Blocks:
  * 3.3.3.3, from 3.3.3.3, 00:00:05 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 456
    2.2.2.2, from 2.2.2.2, 00:00:05 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 456

We can see that the dmz-linkbw community is being received by R1 but it's not doing anything with it...

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#router bgp 123
R1(config-router)#address-family ipv4 unicast
R1(config-router-af)#bgp dmzlink-bw
R1(config-router-af)#end

Now to bounce our BGP peering

R1#clear ip bgp *
R1#
*Mar  1 00:33:47.999: %BGP-5-ADJCHANGE: neighbor 2.2.2.2 Down User reset
*Mar  1 00:33:47.999: %BGP-5-ADJCHANGE: neighbor 3.3.3.3 Down User reset
*Mar  1 00:34:18.059: %BGP-5-ADJCHANGE: neighbor 2.2.2.2 Up
*Mar  1 00:34:18.063: %BGP-5-ADJCHANGE: neighbor 3.3.3.3 Up
R1#sh ip route 6.6.6.6
Routing entry for 6.6.6.6/32
  Known via "bgp 123", distance 200, metric 0
  Tag 456, type internal
  Last update from 2.2.2.2 00:00:05 ago
  Routing Descriptor Blocks:
  * 3.3.3.3, from 3.3.3.3, 00:00:05 ago
      Route metric is 0, traffic share count is 2
      AS Hops 1
      Route tag 456
    2.2.2.2, from 2.2.2.2, 00:00:05 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 456

That's more like it

Let's do the equivalent for our routers in AS 456

R4#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R4(config)#router bgp 456
R4(config-router)#address-family ipv4 unicast
R4(config-router-af)#bgp dmzlink-bw
R4(config-router-af)#neighbor 5.5.5.5 send-community extended
R4(config-router-af)#neighbor 6.6.6.6 send-community extended
R4(config-router-af)#neighbor 10.1.24.2 dmzlink-bw
R4(config-router-af)#end


R5#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R5(config)#router bgp 456
R5(config-router)#address-family ipv4 unicast
R5(config-router-af)#bgp dmzlink-bw
R5(config-router-af)#neighbor 4.4.4.4 send-community extended
R5(config-router-af)#neighbor 6.6.6.6 send-community extended
R5(config-router-af)#neighbor 10.1.35.3 dmzlink-bw
R5(config-router-af)#end


R6#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R6(config)#router bgp 456
R6(config-router)#address-family ipv4 unicast
R6(config-router-af)#bgp dmzlink-bw
R6(config-router-af)#end

R6#clear ip bgp *
R6#
*Mar  1 00:49:11.923: %BGP-5-ADJCHANGE: neighbor 4.4.4.4 Down User reset
*Mar  1 00:49:11.927: %BGP-5-ADJCHANGE: neighbor 5.5.5.5 Down User reset
*Mar  1 00:49:25.943: %BGP-5-ADJCHANGE: neighbor 4.4.4.4 Up
*Mar  1 00:49:25.947: %BGP-5-ADJCHANGE: neighbor 5.5.5.5 Up
R6#sh ip bgp 1.1.1.1
BGP routing table entry for 1.1.1.1/32, version 10
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Multipath: iBGP
Flag: 0x8820
  Not advertised to any peer
  123
    5.5.5.5 (metric 11) from 5.5.5.5 (5.5.5.5)
      Origin IGP, metric 0, localpref 100, valid, internal, multipath
      DMZ-Link Bw 24 kbytes
  123
    4.4.4.4 (metric 11) from 4.4.4.4 (4.4.4.4)
      Origin IGP, metric 0, localpref 100, valid, internal, multipath, best
      DMZ-Link Bw 16 kbytes

R6#sh ip route 1.1.1.1
Routing entry for 1.1.1.1/32
  Known via "bgp 456", distance 200, metric 0
  Tag 123, type internal
  Last update from 4.4.4.4 00:00:43 ago
  Routing Descriptor Blocks:
  * 5.5.5.5, from 5.5.5.5, 00:00:43 ago
      Route metric is 0, traffic share count is 2
      AS Hops 1
      Route tag 123
    4.4.4.4, from 4.4.4.4, 00:00:43 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 123

Let's set up some basic metering using cbqos and test if load balancing works

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#policy-map TST
R2(config-pmap)#class class-default
R2(config-pmap-c)#int s0/0.24
R2(config-subif)#service-policy input TST
R2(config-subif)#service-policy output TST
R2(config-subif)#int s0/0
R2(config-if)#load-interval 30
R2(config-if)#end

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#policy-map TST
R3(config-pmap)#class class-default
R3(config-pmap-c)#int s0/0.35
R3(config-subif)#service-policy input TST
R3(config-subif)#service-policy output TST
R3(config-subif)#int s0/0
R3(config-if)#load-interval 30
R3(config-if)#end

Now we will do an extended ping between R1 Lo0 and R6 Lo0, we'll use the record route option so we can see that the path doesn't remain the same for all of the pings.

R1#ping
Protocol [ip]:
Target IP address: 6.6.6.6
Repeat count [5]: 1000
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 1.1.1.1
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]: r
Number of hops [ 9 ]: 8
Loose, Strict, Record, Timestamp, Verbose[RV]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
Packet has IP options:  Total option bytes= 39, padded length=40
 Record route: <*>
   (0.0.0.0)
   (0.0.0.0)
   (0.0.0.0)
   (0.0.0.0)
   (0.0.0.0)
   (0.0.0.0)
   (0.0.0.0)
   (0.0.0.0)

Reply to request 0 (20 ms).  Received packet has options
 Total option bytes= 36, padded length=36
 Record route:
   (10.1.111.1)
   (10.1.35.3)
   (10.1.222.5)
   (6.6.6.6)
   (10.1.222.6)
   (10.1.35.5)
   (10.1.111.3)
   (1.1.1.1)
   <*>
 End of list

Reply to request 1 (8 ms).  Received packet has options
 Total option bytes= 36, padded length=36
 Record route:
   (10.1.111.1)
   (10.1.35.3)
   (10.1.222.5)
   (6.6.6.6)
   (10.1.222.6)
   (10.1.35.5)
   (10.1.111.3)
   (1.1.1.1)
   <*>
 End of list

Reply to request 2 (12 ms).  Received packet has options
 Total option bytes= 36, padded length=36
 Record route:
   (10.1.111.1)
   (10.1.24.2)
   (10.1.222.4)
   (6.6.6.6)
   (10.1.222.6)
   (10.1.24.4)
   (10.1.111.2)
   (1.1.1.1)
   <*>
 End of list

Reply to request 3 (8 ms).  Received packet has options
 Total option bytes= 36, padded length=36
 Record route:
   (10.1.111.1)
   (10.1.35.3)
   (10.1.222.5)
   (6.6.6.6)
   (10.1.222.6)
   (10.1.35.5)
   (10.1.111.3)
   (1.1.1.1)
   <*>
 End of list

Reply to request 4 (16 ms).  Received packet has options
 Total option bytes= 36, padded length=36
 Record route:
   (10.1.111.1)
   (10.1.35.3)
   (10.1.222.5)
   (6.6.6.6)
   (10.1.222.6)
   (10.1.35.5)
   (10.1.111.3)
   (1.1.1.1)
   <*>
 End of list

..............
 
Reply to request 997 (8 ms).  Received packet has options
 Total option bytes= 36, padded length=36
 Record route:
   (10.1.111.1)
   (10.1.35.3)
   (10.1.222.5)
   (6.6.6.6)
   (10.1.222.6)
   (10.1.35.5)
   (10.1.111.3)
   (1.1.1.1)
   <*>
 End of list

Reply to request 998 (8 ms).  Received packet has options
 Total option bytes= 36, padded length=36
 Record route:
   (10.1.111.1)
   (10.1.24.2)
   (10.1.222.4)
   (6.6.6.6)
   (10.1.222.6)
   (10.1.24.4)
   (10.1.111.2)
   (1.1.1.1)
   <*>
 End of list

Reply to request 999 (12 ms).  Received packet has options
 Total option bytes= 36, padded length=36
 Record route:
   (10.1.111.1)
   (10.1.35.3)
   (10.1.222.5)
   (6.6.6.6)
   (10.1.222.6)
   (10.1.35.5)
   (10.1.111.3)
   (1.1.1.1)
   <*>
 End of list

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/10/20 ms



We can see that the path alternates somewhat in a pattern that looks close to 2:1 but lets check it by looking at the traffic counters

R2#sh policy-map int s0/0.24

 Serial0/0.24

  Service-policy input: TST

    Class-map: class-default (match-any)
      335 packets, 34739 bytes
      30 second offered rate 8000 bps, drop rate 0 bps
      Match: any

  Service-policy output: TST

    Class-map: class-default (match-any)
      335 packets, 38351 bytes
      30 second offered rate 9000 bps, drop rate 0 bps
      Match: any

R3#sh policy-map int s0/0.35

 Serial0/0.35

  Service-policy input: TST

    Class-map: class-default (match-any)
      668 packets, 69431 bytes
      30 second offered rate 12000 bps, drop rate 0 bps
      Match: any

  Service-policy output: TST

    Class-map: class-default (match-any)
      670 packets, 76471 bytes
      30 second offered rate 13000 bps, drop rate 0 bps
      Match: any

The packet load appears to pretty much align with the traffic share count.