External BGP Next Hop

Here's an interesting BGP discovery (well at least it's interesting to me...)

A common belief that is not quite true is that EBGP peers set the advertised next hop to be themselves. If we have a look at RFC 1771 section 5.1.3 (NEXT_HOP)

A BGP speaker can advertise any internal border router as the next hop provided that the interface associated with the IP address of this border router (as specified in the NEXT_HOP path attribute) shares a common subnet with both the local and remote BGP speakers. A BGP speaker can advertise any external border router as the next hop, provided that the IP address of this border router was learned from one of the BGP speaker's peers, and the interface associated with the IP address of this border router (as specified in the NEXT_HOP path attribute) shares a common subnet with the local and remote BGP speakers.

Take this example to verify it:

We have 3 routers each in a separate BGP AS.  R1 is peered with R2, and R3 is peered with R2:

R1 and R3 are connected to a switch belonging to R2 and Fa0/0 of R1/R2/R3 are all in the same VLAN.

Below are the relevant configs:

R1
hostname R1
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
!
interface FastEthernet0/0
 ip address 10.1.123.1 255.255.255.0
!
router bgp 1
 no synchronization
 network 1.1.1.1 mask 255.255.255.255
 neighbor 10.1.123.2 remote-as 2
 no auto-summary

R2
hostname R2
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
!
interface FastEthernet0/0
 ip address 10.1.123.2 255.255.255.0
!
router bgp 2
 no synchronization
 network 2.2.2.2 mask 255.255.255.255
 neighbor 10.1.123.1 remote-as 1
 neighbor 10.1.123.3 remote-as 3
 no auto-summary

R3
hostname R3
interface Loopback0
 ip address 3.3.3.3 255.255.255.255
interface FastEthernet0/0
 ip address 10.1.123.3 255.255.255.0
!
router bgp 3
 no synchronization
 bgp log-neighbor-changes
 network 3.3.3.3 mask 255.255.255.255
 neighbor 10.1.123.2 remote-as 2
 no auto-summary

Lets make sure everyone has BGP connectivity and has learnt prefixes:

R1#sh ip bgp summ | b Neighbor
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.123.2      4     2      26      20       28    0    0 00:13:46        2

R2#sh ip bgp summ | b Neighbor
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.123.1      4     1      20      26       28    0    0 00:13:48        1
10.1.123.3      4     3      20      27       28    0    0 00:12:44        1

R3#sh ip bgp summ | b Neighbor
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.123.2      4     2      27      20       28    0    0 00:12:31        2


Looks reasonable, lets check what we routes have learnt from BGP


R1#sh ip route bgp
     2.0.0.0/32 is subnetted, 1 subnets
B       2.2.2.2 [20/0] via 10.1.123.2, 00:11:57
     3.0.0.0/32 is subnetted, 1 subnets
B       3.3.3.3 [20/0] via 10.1.123.3, 00:10:56


R2#sh ip route bgp
     1.0.0.0/32 is subnetted, 1 subnets
B       1.1.1.1 [20/0] via 10.1.123.1, 00:11:45
     3.0.0.0/32 is subnetted, 1 subnets
B       3.3.3.3 [20/0] via 10.1.123.3, 00:11:45


R3#sh ip route bgp
     1.0.0.0/32 is subnetted, 1 subnets
B       1.1.1.1 [20/0] via 10.1.123.1, 00:12:05
     2.0.0.0/32 is subnetted, 1 subnets
B       2.2.2.2 [20/0] via 10.1.123.2, 00:13:06


The interesting thing here is that R1 has 3.3.3.3/32 reachable via R3 Fa0/0 and R3 has see 1.1.1.1/32 reachable via R1 Fa0/0 even though neither of these guys have a direct BGP peering with each other.

It appears that R2 is smart enough to realise since R1 and R3 are on the same subnet rather than having R2 get in the way, it will not change the advertised next-hop IP address to be itself even though R1 and R2 are in different autonomous systems

R1#sh ip bgp
BGP table version is 28, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 1.1.1.1/32       0.0.0.0                  0         32768 i
*> 2.2.2.2/32       10.1.123.2               0             0 2 i
*> 3.3.3.3/32       10.1.123.3                             0 2 3 i

R2#sh ip bgp
BGP table version is 28, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 1.1.1.1/32       10.1.123.1               0             0 1 i
*> 2.2.2.2/32       0.0.0.0                  0         32768 i
*> 3.3.3.3/32       10.1.123.3               0             0 3 i

R3#sh ip bgp
BGP table version is 28, local router ID is 3.3.3.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 1.1.1.1/32       10.1.123.1                             0 2 1 i
*> 2.2.2.2/32       10.1.123.2               0             0 2 i
*> 3.3.3.3/32       0.0.0.0                  0         32768 i

We can see that R2's AS is still included within the AS-PATH for the routing information even if it is not in the traffic forwarding path

Troubleshooting Workbook

As my lab attempt revealed, my troubleshooting technique isn't as good as I thought it was.  As such I've been going through the Micronics Troubleshooting workbook which was written by Narbik and Dan Shechter.  So far I find it pretty helpful.

It comes with 10 Troubleshooting labs following the Narbik topology,  it ships with two Locklizard PDFs - one is just questions only, and the other with the answers for verification or when you finally give up.  Each lab has around 15 tickets which encompass routers and switches.  Currently I've been allocating myself 2 hours per lab.  Some tickets I can resolve very quickly, others take a bit of time to identify and resolve the problem - it's good though.

Although its a very small thing, I have stopped using Putty Connection Manager and have gone back to untabbed Putty windows with the "Always on top" and "Right Click = menu" which is what the lab environment uses - when I encountered this in the lab I found it a little disconcerting particulary during the troubleshooting when there were on the order of 30 possible devices to work with.  It wasn't the cause of me failing that section, it just help fluster me a little.  Already I am taking this into my workflow and learning to close down sessions that I don't actively require, while a very small thing on the scheme of things.  I think it might give a slight advantage next time I sit the lab (BTW it is possible during the lab to change the settings of Putty but I would rather save those few minutes of terminal configuration to be used for solving tickets)

If you're interested in trying out some of the tickets from the workbook without risking your cash - checkout some of Dan's mini TS Lab tickets to get a sense of the style and the whole of Lab #3 is available for free for you to evaluate.

Status Update

It's been two weeks since I attempted the CCIE R&S Version 4.0 Lab in Sydney (I have just returned from a vacation in Europe which started right after my lab attempt).  I found the lab pretty tough but doable.  For me by the time lunch time came around I was pretty sure that by the middle of the day, I was doing the famous expensive lunch (which at the Sydney campus was not bad to be honest)  My technique for troubleshooting just let me down - I have no excuses in this area, my speed and technique did not do me any favours here.  It was difficult to keep going once that section closed down with a number of outstanding tickets.  The configuration section was a different story, I found it relatively straight forward - a few variations of things I didn't necessarily expect but the experience gained using training materials from IPexpert and Narbik did arm me with the capability to deal with them and with the config section I achieved a pass.

I guess a little distance between reporting the status here allows the emotions to settle, and the time away from the routers and mailing lists are able to let me have a little break to regroup, re-energise and know that this is definitely something I can pass and that I intend to face the lab again in the next couple of months.  I intend to mainly focus on my increasing TS capability to be much faster in problem diagrnose and repair (but not ignoring the config portions) as to when I set my next date it will depend on when I believe I am ready and my workload (returning from a long vacation may mean I have a lot of catch up there)

For those interested in what the lab is like, the lab exam demo is very much on the money and may be helpful.

EIGRP Unequal Cost Paths

This post is going to be about how to perform unequal cost load balancing using EIGRP.  There's going to be a little bit of mathematics here but nothing beyond some basic algebra.


The topology for this post is shown below:  We have 4 routers, with multiple paths between them, each has a loopback interface and all interfaces are in EIGRP AS 100.



R1--FR-256kbps--R2--FR-256kbps--R4
 \                              /
  \-FR-512kbps-R3-FastEthernet-/



The initial configs are shown below:
R1
hostname R1
interface Loopback0
 ip address 1.1.1.1 255.0.0.0
!
interface Serial0/0
 description to Frame Switch
 no ip address
 encapsulation frame-relay
 no frame-relay inverse-arp
!
interface Serial0/0.12 point-to-point
 description R1-R2
 bandwidth 256
 ip address 10.1.12.1 255.255.255.0
 frame-relay interface-dlci 102
!
interface Serial0/0.13 point-to-point
 description R1-R3
 bandwidth 512
 ip address 10.1.13.1 255.255.255.0
 frame-relay interface-dlci 103
!
router eigrp 100
 network 1.1.1.1 0.0.0.0
 network 10.1.12.1 0.0.0.0
 network 10.1.13.1 0.0.0.0
 no auto-summary
!

R2
hostname R2
interface Loopback0
 ip address 2.2.2.2 255.0.0.0
!
interface Serial0/0
 description to Frame Switch
 no ip address
 encapsulation frame-relay
 no frame-relay inverse-arp
!
interface Serial0/0.21 point-to-point
 bandwidth 256
 description R2-R1
 ip address 10.1.12.2 255.255.255.0
 frame-relay interface-dlci 201
!
interface Serial0/0.24 point-to-point
 bandwidth 256
 description R2-R4
 ip address 10.1.24.2 255.255.255.0
 frame-relay interface-dlci 204
!
router eigrp 100
 network 2.2.2.2 0.0.0.0
 network 10.1.12.2 0.0.0.0
 network 10.1.24.2 0.0.0.0
 no auto-summary
!

R3
hostname R3
interface Loopback0
 ip address 3.3.3.3 255.0.0.0
!
interface FastEthernet0/0
 description to R3-R4
 ip address 10.1.34.3 255.255.255.0
 speed 100
 full-duplex
!
interface Serial0/0
 description to Frame Switch
 no ip address
 encapsulation frame-relay
 no frame-relay inverse-arp
!
interface Serial0/0.31 point-to-point
 description R3-R1
 ip address 10.1.13.3 255.255.255.0
 frame-relay interface-dlci 301
!
router eigrp 100
 network 3.3.3.3 0.0.0.0
 network 10.1.13.3 0.0.0.0
 network 10.1.34.3 0.0.0.0
 no auto-summary
!

R4
hostname R4
interface Loopback0
 ip address 4.4.4.4 255.0.0.
!
interface FastEthernet0/0
 description to R4-R3
 ip address 10.1.34.4 255.255.255.0
 speed 100
 full-duplex
!
interface Serial0/0
 description to Frame Switch
 no ip address
 encapsulation frame-relay
 no frame-relay inverse-arp
!
interface Serial0/0.42 point-to-point
 bandwidth 256
 description R4-R2
 ip address 10.1.24.4 255.255.255.0
 frame-relay interface-dlci 402
!
router eigrp 100
 network 4.4.4.4 0.0.0.0
 network 10.1.24.4 0.0.0.0
 network 10.1.34.4 0.0.0.0
 no auto-summary
!


As you can see we are using the default metric weights here: since we haven't changed anything K1 (Bandwidth) and K3 (Delay) are set to 1, with K2 , K4 and K5 left at 0

Therefor EIGRP metrics are calculated by 256 * ( 10^7 / WorstPathBandwidth + (TotalPathDelay/10) ) Where WorstPathBandwidth is in kbps and TotalPathDelay is in usecs.

From R1 if we were to work out the EIGRP Metric to 4.0.0.0 (Loopback 0 on R4) we can see two paths based on the topology - one via R2 and one via R3

Path 1 Metric calculation For R1 to 4.0.0.0 (R4 Lo0) via R2

Using the default metric weights, we can follow the same process EIGRP uses to calculate metrics by following the paths hop by hope and obtain the interface bandwidth (BW) and delay (DLY) values from show interface | inc DLY

Before we directly see what metric R1 has to 4.0.0.0, we need to calculate the metric R2 has to reach 4.0.0.0 as this is used by EIGRP to determine the feasibility condition on R1.  I'll get into the importance of this a little further down but right now it should be enough to know that our router of interest (R1) needs to know the advertised metric from our EIGRP upstream neighbor as well as computing the metric locally.

R2's Metric to 4.0.0.0 (R4 Lo0)

From R2 S0/0.24 to R4 S0/0.24 (Bandwidth 256 kbps and 20000 usec Delay)
From R4 To R4 Lo (Bandwidth 8000000 kbps and 5000 usec Delay)

B = round (10 ^ 7 / WorstPathBandwidth) = 39062
D = TotalPathDelay / 10 = 2500

Metric = 256 * (B + D)

  = 10639872

This metric (which R1 will use as part of the feasbility calculations) as well as the worst bandwidth and total delay will be advertised to R1 in order for it to compute the end metric to 4.0.0.0


R1's Metric to 4.0.0.0 (R4 Lo0) via R2



R1 will look at its connection to R2 (R1 S0/0.12 to R2 S0/0.21) indentifying the link bandwidth is 256 kbps and 20000 usec Delay and factor this into it's metric calculation, including the bandwidth and delay values received from R2:

WorstPathBandwidth = 256 kbps and TotalPathDelay = 45000 usec

R1's computed Metric to 4.0.0.0 via R2 is therefore


Metric = 256 * (B + D)
Where:
    B = round (10 ^ 7 / WorstPathBandwidth) = 39062
    D = TotalPathDelay / 10 = 4500

Which results in 11151872


Path 2 Metric calculation For R1 to 4.0.0.0 (R4 Lo0) via R3

This is a higher bandwidth path so we expect to see a lower metric here.

R3's Metric to 4.0.0.0 (R4 Lo0)

From R3 Fa0/0 to R4 Fa0/0 (Bandwidth 100000 kbps and 100 usec Delay)
To R4 Lo (Bandwidth 8000000 kbps and 5000 usec Delay)

Metric = 256 * (B + D)
Where:   B = round (10 ^ 7 / WorstPathBandwidth) = 100
   D = TotalPathDelay / 10 = 510

Which results in 156160

R1's Metric to 4.0.0.0 (R4 Lo0) via R3



R1 will look at its connection to R3 (R1 S0/0.13 to R3 S0/0.31) indentifying the link Bandwidth 512 kbps and 20000 usec Delay and factor this into it's metric calculation, including the bandwidth and delay values advertised from R3:

WorstPathBandwidth = 512 kbps and TotalPathDelay = 25100 usec

Metric = 256 * (B + D)
Where:   B = round (10 ^ 7 / WorstPathBandwidth) = 19531
   D = TotalPathDelay / 10 = 2510

Which results in 5642496

Okay, so we can work out metric calculations by hand, lets see if R1 agrees with our findings:

R1#sh ip eigrp topology 4.0.0.0

IP-EIGRP (AS 100): Topology entry for 4.0.0.0/8

  State is Passive, Query origin flag is 1, 1 Successor(s), FD is 5642496

  Routing Descriptor Blocks:

  10.1.13.3 (Serial0/0.13), from 10.1.13.3, Send flag is 0x0

      Composite metric is (5642496/156160), Route is Internal

      Vector metric:

        Minimum bandwidth is 512 Kbit

        Total delay is 25100 microseconds

        Reliability is 255/255

        Load is 1/255

        Minimum MTU is 1500

        Hop count is 2

  10.1.12.2 (Serial0/0.12), from 10.1.12.2, Send flag is 0x0

      Composite metric is (11151872/10639872), Route is Internal

      Vector metric:

        Minimum bandwidth is 256 Kbit

        Total delay is 45000 microseconds

        Reliability is 255/255

        Load is 1/255

        Minimum MTU is 1500

        Hop count is 2

We can see that the minimum bandwidth aligns with our WorstPathBandwidth and that the total delay is also in agreement with our manual calculations.

When we look at the composite metrics we can see two values (A/B) where A represents the computed metric to 4.0.0.0 (R4 Lo0) as far as R1 is concerned for a particular next-hop and B represents the metric that the next-hop has to reach 4.0.0.0

Task

Make R1 load balance traffic destined to R4 Lo0 with a traffic-share of 2:1 (matching with our PVC bandwidths)

Obviously this means that we use the eigrp "variance" command, however on its own this will not be enough to achieve the goal:

If we look at the metrics associated with both paths by examing the EIGRP topology database:

R1#sh ip eigrp topology 4.0.0.0 | inc from|Composite|bandwidth|delay

  10.1.13.3 (Serial0/0.13), from 10.1.13.3, Send flag is 0x0

      Composite metric is (5642496/156160), Route is Internal

        Minimum bandwidth is 512 Kbit

        Total delay is 25100 microseconds

  10.1.12.2 (Serial0/0.12), from 10.1.12.2, Send flag is 0x0

      Composite metric is (11151872/10639872), Route is Internal

        Minimum bandwidth is 256 Kbit

        Total delay is 45000 microseconds

Although we need to consider that while 11151872 and 5642496 doesn't exactly equal 2, more importantly in order for EIGRP install another route into the routing table for the same destination, the feasbility condition has to be met.  In this case it means that for the path via 10.1.12.2 to be considered, the advertised metric from R2 (10639872) must be less than R1's computed metric via R3 (5642496) which at this point in time is not the case.

With the default K values, we only have bandwidth and delay to play with and usually you do not want to change bandwidth values because that might break a number of other things such as QoS.

If we compare the two paths, we can see the link via R3 has a delay of 25100 usec and via R2 a delay of 45000 usec.  If we equalise the delay, the bandwidth values are already two to one, maybe that should make things work?

On R1 on the link facing R3, lets increase the delay by 45000 - 25100 = 19900 usecs

Firstly lets confirm the current delay

R1#sh int s0/0.13 | inc DLY

  MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,

So obviously the new delay should be 39900 usec but remember we set delay in 10s of usec

R1(config)#int s0/0.13

R1(config-subif)# delay 3990

Did this work? Lets set the variance and see what appears in the routing table

R1(config)#router eigrp 100

R1(config-router)#variance 2

R1(config-router)#do sh ip route 4.0.0.0

Routing entry for 4.0.0.0/8

  Known via "eigrp 100", distance 90, metric 6151936, type internal

  Redistributing via eigrp 100

  Last update from 10.1.13.3 on Serial0/0.13, 00:00:05 ago

  Routing Descriptor Blocks:

  * 10.1.13.3, from 10.1.13.3, 00:00:05 ago, via Serial0/0.13

      Route metric is 6151936, traffic share count is 1

      Total delay is 45000 microseconds, minimum bandwidth is 512 Kbit

      Reliability 255/255, minimum MTU 1500 bytes

      Loading 1/255, Hops 2

No, it doesn't appear to have worked - what do we see in the topology database?

R1(config-router)#do sh ip eigrp topology 4.0.0.0 | inc from|Composite|bandwidth|delay

  10.1.13.3 (Serial0/0.13), from 10.1.13.3, Send flag is 0x0

      Composite metric is (6151936/156160), Route is Internal

        Minimum bandwidth is 512 Kbit

        Total delay is 45000 microseconds

  10.1.12.2 (Serial0/0.12), from 10.1.12.2, Send flag is 0x0

      Composite metric is (11151872/10639872), Route is Internal

        Minimum bandwidth is 256 Kbit

        Total delay is 45000 microseconds

Well while the bandwidth and delay values are in alignment with the R1 computed metrics to 4.0.0.0 with the ratio of 2:1 the feasability condition is not being met, so only the route via R3 can be used

One way to satisfy the FC is to make the computed metric for R3 to exceed 10639872 (the advertised metric from R2)

Metric = 256 * (B + D)
Where:    B = round (10 ^ 7 / WorstPathBandwidth)
   D = TotalPathDelay / 10

10639872 = 256 * (10^7/512 + D/10)

10639872 / 256 - 10^7/512 = D/10

22031 = D/10

D = 220310 but 5100 usec of delay already exist, so D has to be greater than 215210 in order for the FC to pass...

R1(config-subif)#int s0/0.13

R1(config-subif)#delay 21522

So will EIGRP now install this route?

R1(config-subif)#do sh ip route 4.0.0.0

Routing entry for 4.0.0.0/8

  Known via "eigrp 100", distance 90, metric 10640128, type internal

  Redistributing via eigrp 100

  Last update from 10.1.12.2 on Serial0/0.12, 00:00:17 ago

  Routing Descriptor Blocks:

  * 10.1.13.3, from 10.1.13.3, 00:00:17 ago, via Serial0/0.13

      Route metric is 10640128, traffic share count is 240

      Total delay is 220320 microseconds, minimum bandwidth is 512 Kbit

      Reliability 255/255, minimum MTU 1500 bytes

      Loading 1/255, Hops 2

    10.1.12.2, from 10.1.12.2, 00:00:17 ago, via Serial0/0.12

      Route metric is 11151872, traffic share count is 229

      Total delay is 45000 microseconds, minimum bandwidth is 256 Kbit

      Reliability 255/255, minimum MTU 1500 bytes

      Loading 1/255, Hops 2

Yes, we have two routes to 4.0.0.0 but the traffic share count is wrong (it should be 2 to 1 not 240 to 229)

For this to work we need to play with delay again but this time on the R2 facing link so that the computed metric of the path using R2 is double that using R3

Metric = 256 * (B + D)
Where:    B = round (10 ^ 7 / WorstPathBandwidth)
   D = TotalPathDelay / 10



2 * 10640128 = 256 * (10^7/256 + D/10)

83126 = 39062 + D/10

D = 440640 usec (the current delay is 45000 usec, requiring an extra 395640 usec of added delay)

R1(config-subif)#do sh int s0/0.12 | inc DLY

  MTU 1500 bytes, BW 256 Kbit/sec, DLY 20000 usec,

R1(config-subif)#int s0/0.12

R1(config-subif)#delay 41564

So does this sort it out?

R1(config-subif)#do sh ip route 4.0.0.0

Routing entry for 4.0.0.0/8

  Known via "eigrp 100", distance 90, metric 10640128, type internal

  Redistributing via eigrp 100

  Last update from 10.1.12.2 on Serial0/0.12, 00:00:03 ago

  Routing Descriptor Blocks:

  * 10.1.13.3, from 10.1.13.3, 00:00:03 ago, via Serial0/0.13

      Route metric is 10640128, traffic share count is 2

      Total delay is 220320 microseconds, minimum bandwidth is 512 Kbit

      Reliability 255/255, minimum MTU 1500 bytes

      Loading 1/255, Hops 2

    10.1.12.2, from 10.1.12.2, 00:00:03 ago, via Serial0/0.12

      Route metric is 21280256, traffic share count is 1

      Total delay is 440640 microseconds, minimum bandwidth is 256 Kbit

      Reliability 255/255, minimum MTU 1500 bytes

      Loading 1/255, Hops 2

Okay, the from a routing point of view this looks good, lets make sure that CEF is active and enable per-packet load balancing

R1(config-subif)#ip cef

R1(config)#int s0/0.12

R1(config-subif)#ip load-sharing per-packet

R1(config-subif)#int s0/0.13

R1(config-subif)#ip load-sharing per-packet

To see that things are going to plan, we shall set up a service policy to match icmp traffic and attach it to both exit interfaces:

R1(config-subif)#int s0/0

R1(config-if)#load-interval 30

R1(config-if)#access-list 100 permit icmp any any

R1(config)#class-map ICMP

R1(config-cmap)#match access-group 100

R1(config-cmap)#policy-map OUT

R1(config-pmap)#class ICMP

R1(config-pmap-c)#int s0/0.12

R1(config-subif)#service-policy output OUT

R1(config-subif)#int s0/0.13

R1(config-subif)#service-policy output OUT

R1(config-subif)#do show policy-map int | s Serial|ICMP
 Serial0/0.12
    Class-map: ICMP (match-all)
      0 packets, 0 bytes
      30 second offered rate 0 bps
      Match: access-group 100

 Serial0/0.13
    Class-map: ICMP (match-all)
      0 packets, 0 bytes
      30 second offered rate 0 bps
      Match: access-group 100

R1(config-subif)#do ping 4.4.4.4 source 1.1.1.1 repeat 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 12/50/100 ms

Since the load is meant to be 2:1, we should see roughly 34 packets on S0/0.12 and 66 on S0/0.13


R1(config-subif)#do show policy-map int | s Serial|ICMP
 Serial0/0.12
    Class-map: ICMP (match-all)
      35 packets, 3640 bytes
      30 second offered rate 2000 bps
      Match: access-group 100

 Serial0/0.13
    Class-map: ICMP (match-all)
      65 packets, 6760 bytes
      30 second offered rate 2000 bps
      Match: access-group 100


Seems close enough, how is CEF handling the load sharing?

R1(config-subif)#do sh ip cef 4.0.0.0 internal

4.0.0.0/8, version 691, epoch 0, per-packet sharing

0 packets, 0 bytes

  via 10.1.13.3, Serial0/0.13, 0 dependencies

    traffic share 2

    next hop 10.1.13.3, Serial0/0.13

    valid adjacency

  via 10.1.12.2, Serial0/0.12, 0 dependencies

    traffic share 1, current path

    next hop 10.1.12.2, Serial0/0.12

    valid adjacency

  0 packets, 0 bytes switched through the prefix

  tmstats: external 0 packets, 0 bytes

           internal 0 packets, 0 bytes

  Load distribution: 0 1 0 1 0 1 0 1 0 1 0 0 0 0 0 (refcount 1)

  Hash  OK  Interface                 Address         Packets

  1     Y   Serial0/0.13              point2point           0

  2     Y   Serial0/0.12              point2point           0

  3     Y   Serial0/0.13              point2point           0

  4     Y   Serial0/0.12              point2point           0

  5     Y   Serial0/0.13              point2point           0

  6     Y   Serial0/0.12              point2point           0

  7     Y   Serial0/0.13              point2point           0

  8     Y   Serial0/0.12              point2point           0

  9     Y   Serial0/0.13              point2point           0

  10    Y   Serial0/0.12              point2point           0

  11    Y   Serial0/0.13              point2point           0

  12    Y   Serial0/0.13              point2point           0

  13    Y   Serial0/0.13              point2point           0

  14    Y   Serial0/0.13              point2point           0

  15    Y   Serial0/0.13              point2point           0

  refcount 6

Interestingly it appears that CEF alternates between both links for the first few packets and then concentrates on just using S0/0.13 towards the end of the sequence

Drinking from the Firehose (Narbik's Bootcamp Review)

Caution: Big wall of Text Warning!  If you're a TL:DR (Too long: Didn't Read) type of person maybe skip this post or just jump down to the conclusion.
 
At the time of writing (the Saturday after the bootcamp) I'm still recovering from the marathon Routing and Switching bootcamp that was run by Narbik from Micronics Training in Sydney, Australia between May 23 2011 and May 27 2011 but I wanted to give some impressions as to what I thought about it because when the opportunity to take a bootcamp came up, I wanted to be sure that I was spending my money (and time) wisely.  This is obviously all my opinion about my experience but I figure I'll try to give as much relevant information as to the bootcamp and the surrounding environent as possible because if you're going to be spending a lot of money on a bootcamp especially if its your own you want to know what your in for and what you get and what you need.  Your experience may be slightly different even if you do it in Sydney.

The class size with around 14 students with slightly more depending.  The reason for the slightly more is that once you have attended one of Narbik's bootcamps you are welcome to re-attend it for free.  Please note that this doesn't cover the Cisco 360 material (I'll get into that shortly) nor the use of Narbik's lab equipment (however you can rent a lab for what appears to be a very modest fee indeed, use your own rack at home if you have remote access, or simulate the topology using dynamips but accepting that you wont be able to do the advanced Cat 3560 features).  So during the bootcamp there were a few guys who popped in for a couple of days to brush up on some of the topics.   Actually when I asked the re-attendees if what was being presented was exactly the same, they mentioned that the information itself didn't really change but that in some areas Narbik was presented in a slightly different but more easily to understand manner which suggests that he's continuously refining his delivery method based on feedback.

Our class was located right in the Sydney Central Business District - initially on the website (before the class was 100% confirmed - I believe that there is a minimum of 7 new students required before a class can take place - so if you're looking at trying to get a bootcamp happening in your area if you can get close to 6 like minded mates together, it might be worth seeing if something can be arranged) it mentioned North Sydney and listed some hotels in the area.  I had booked a hotel in North Sydney (wanting to be in easy walking distance to the training centre) so was a little surprised when the venue had changed to the Sydney CBD itself (fortunately I didn't pre-pay my accomodation so I was able to re-book somewhere else without paying a fee) I would strongly suggest that rather than just reading the location on the website that you send an email to the people at Micronics to check.  It still may be worth foregoing any pre-payment discount just in case for whatever reason the course gets relocated or postponed)  I actually think that the different location was much better because North Sydney seems to shutdown not long after 5pm, where in the CBD is right near entertainment venues and many resturants etc so it was easy to get food at any time and there was a major grocery store (Coles) in very close proximity so getting your own supplies was easy.

Part of the way that the bootcamp costs are controlled is that you need to bring your own equipment - bring your own laptop - pretty much you just need something with a web browser and a telnet client.  While you could probably get by with linux, some of the material you get access to are locklizard protected PDFs so it might be a problem.  The material from Narbik can be read on Mac OSX but for some reason the Cisco 360 Assessment Lab Answer keys cannot be read on OSX (requiring a virtualised XP/Windows 7 instance or bootcamp to read it)  Network access provided was wireless (not so good most likely because other businesses in the building with their own wireless networks going) so I used the wired network which was very reliable and fast.  Besides bringing your own laptop, any meals, snacks, note pads, pens etc - bring that yourself.  As I said due to the location it was easy enough to be able to get your own stuff pretty much at any time (Coles for instance would open at 6am and close as midnight) with plenty of fast food options nearby with China town not that far of a jump for a more substantial feed at the end of the day if you so chose.  Actually I know a number of guys brought lunch/dinner from home and were able to use the fridge/microwave at the training centre to keep their food, so if you're on a tight budget that might help you some too.

A number of the guys in the class were Sydney based and took public transport to the site which just because they lived a fair way out could take over an hour in travel in each direction and worse outside of peak travel times.  The hours of the bootcamp are going to be longer than a typical workday, so if you are able to, it could be worth looking at getting a hotel room nearby just so you can cut down on that (there is some homework that you don't have to but if you can you should be doing - knocking out 2 hours of travel so you can do that may be a good idea, so you can get some sleep in - it's not essential but it may help you get that little bit more out of your bootcamp experience)


Okay, so you've decided you're going to do the bootcamp - you book the class, and pay your money.  Not too long later you'll get some workbooks "The Foundation Volumes 1 and 2" from Janet (Narbik's wife) that are in your interests to go through prior to attending the bootcamp.  These are the first of the locklizard protected workbooks you get.  If you have performed these labs or are familiar with the topics, you are going to get the most out of the bootcamp.  To be honest I looked through these but didn't do the labs (I have been doing my lab prepation with another training vendor's materials)  If you come to this bootcamp towards the beginning of your CCIE study journey rather than towards the end like myself, I think this material will be very beneficial and you may not need to necessarily look elsewhere.  So while there is probably an advantage to booking onto the bootcamp early to get access to these workbooks early to prepare yourself for the bootcamp, I did ask Janet if I purchased the workbooks on their own - if I would get a discount from the bootcamp fee and she indicated that this was the case.  I guess if you wanted to dip your feet in the water and de-risk things for yourself, I guess you can start with just purchasing the foundation workbooks, see if you like the labs and style (the bootcamp labs use the same topology) and then sign up for the bootcamp.   Please note that the foundation workbooks themselves are not actually used or referred to in the bootcamp and are simply there to help ensure you have the fundamentals down in order to get the most out of the bootcamp.  Coming up to the bootcamp you'll need to have a valid cisco.com account, Janet will ask you for your details since the Cisco 360 program uses those credentials to send you your relevant information and setting things up for Cisco 360 rack access.  During the bootcamp you will have access to two racks.  One rack will be a Cisco 360 rack which you really only use for the Assessment Labs and another being Narbik's.  Everyone has their own gear, there's no sharing.  Access to the labs is via a terminal server so if you prefer to have a separate telnet/ssh session for each device or just go through the terminal server and using the control shift 6 (or whatever it is - that's not the way I work my lab) you can go that way.

The day before or the day of the bootcamp, you'll get the access credentials from Cisco 360 and you'll get a bunch of other workbooks from Janet.  You'll get the Advanced Routing and Switching 4.0 Workbooks Volumes 1 and 2 - which have the worked out labs inside (other vendors may call this the detailed solution guide) You'll also get the Bootcamp 4.0 Workbook (again with worked out labs and explanations) and initial configurations for the Bootcamp Workbook labs.  These workbooks are actively being worked on and Narbik was mentioning that there are some new sections being added which will be get access to (that said these are very comprehensive as is)

The reality though, is that this is a 5 day bootcamp and there is no way to cover this all in depth, so only a limited list of the blueprint topics can be covered.  That's not to say that Narbik wont talk to you about an area you are having issues with - its just that to cover all of the topics listed with the depth required, it's impossible to touch everything (I think if you were to look at the Services portion of the blueprint you could have a dedicated bootcamp just for that)

Day 1 (Monday)
Just about everyone shows up just before 9am bright-eyed and bushy-tailed ready to get into it, if they are like me they have heard all the good things about Narbik and are wondering if he's 10 feet tall and can fix faults by staring hard at routers.  Well, he's not 10 feet tall and I don't think he stares hard at routers to fix problems, though perhaps he looks intently at telnet sessions - he's a very likable and knowledgable person.  He very quickly goes into his background and puts down what the agenda for the following days are going to be and then goes around the room asking us to talk briefly about ourselves, what we do, where we are at with our studies, if we have a lab date scheduled yet, if we have already attempted the lab and what we want to get out of the bootcamp.  If I remember correctly, at our bootcamp Multicast and MPLS were hot topics (mine was L2 QoS)

After that we pretty much jumped into the Cisco 360 lab racks and started on an 8 hour Assessment lab exam.  It wasn't the hardest of exams but I think it was good thing to do, as it helps Narbik gauge what else the class might be have problems with and can ensure that those topics are particularly re-enforced throughout the bootcamp.  Also for myself it helped to clarify what topics I might need to attention on.  The Cisco 360 Assessment Labs are automatically graded but besides being sent the answer key (this is the file that can only be opened in windows) their is a web page that takes your configs and compares them with the reference configs, allowing you to see side-by-side of what you did vs what the recommended method was.  Besides that, you could simulate the commands "show ip route", "show ip bgp summary" and many many more on the simulation and see what you had and should have seen, very nice but probably better if you had more time to review.

After the Assessment Lab we started on the switching lecture, there's a level of expectation here, so we dont go into STP and basic stuff like that, rather it's more on the security related features including port-security and dhcp snooping and a good session on Private VLANs and dot1x.  The next session goes into Frame-Relay topics.  All of these lectures are Narbik standing at a whiteboard going through the topics in what is obviously orchestrated to go through a sequence (to ensure nothing gets left out) but is certainly free-flow with active discussion with the class.  One of the things that comes to light early on is that Narbik has a great passion to teach and has a great desire to ensure that any uncertainties someone has, things are clarified using different examples until they comprehend a topic.  I think one of the things that Narbik enjoys is seeing someone's expression change from looking quizzical about something to looking satisfied that they understand the how and why.

At around 9:30pm we leave the class to go home or back to our hotels with the advice to do some of the Frame-Relay labs from the Advanced CCIE Routing and Switching lab book

Day 2 (Tuesday)
Most people show up around 9am a little tired and ready to get into it - some of the guys probably weren't quite prepared for the length of the day (and if you factor in the travel time they might have had, they would have had very little sleep if they spent a couple of hours or labs) which is really just sign of things to come.

We spent a few hours in the morning doing some more Frame Relay Labs.  The point of the bootcamp is not just to get a face full of lecture but to also play with the equipment, observe the behaviour, verify it and take advantage of Narbik (and the other students) to help clarify your problems or misunderstanding (as a student I find trying to explain how things work to someone else, helps me ensure that I actually know it)

The two topics under discussion are OSPF and EIGRP with most of the day associated with OSPF.  To be honest coming in I thought I knew OSPF pretty well but now I understand it a lot better, particularly in areas around filtering and summarisation.  Narbik made it a point to talk about things from an IOS point of view rather than what RFCs say.  Even then sometimes IOS documentation can be ambiguous at best or misleading in other cases, so whatever Narbik described as a behaviour, he was able to back what he was saying with labs in his workbook (the whole bootcamp is done that way, so he can back up anything he says with a demonstration)

At around 9:30pm we leave the class to go back and do some OSPF labs.

Day 3 (Wednesday)
Not too long after 9am most people are ready to start, the enthusiam is still there but people are starting to look a little weary but Narbik is like the Energizer Bunny and is ready to go like its his first day.  After doing more OSPF and EIGRP labs in the morning.  The main topics for the day are BGP, MPLS and L3 VPNs including the different CE-PE routing protocols and possibly implications you need to be aware of.

We finish up the around 9:00pm as Thursday was going to be a long day indeed...


Day 4 (Thursday)
Everyone is looking a little rough around the edges and kind of dreading what a long day it's going to be, that is everyone except for Narbik who is fresh as a daisy.  I've had so much coffee, that the guy at the coffee shop starts making my coffee before I order it - I've become a regular there in less than a week!

After a few hours of labbing, the QoS lectures kick in.  Starting with MLS QoS which was a particular topic I don't know if I was looking forward to as such but certainly wanted to ensure that I would help get some clarification as to how things work, particular around the srr-queue area.  The lecture was great, I've taken a lot of notes in that area now and have access to some good labs to help crystalise my thoughts further.  We then go stuck into L3 QoS primarily on FRTS and CBWFQ.

At around 6pm we started the troubleshooting portion of the Cisco 360 Assessment lab which went for about two and a half hours.

After this we had a dinner break and the Assement lab Configuration portion started at 10pm with 6 hours to complete the tasks. There was enough time for some of the locals to head home to do that component.  I stayed at the class because my hotel internet connection was pretty average and it was a short walk from the class room to the hotel.  At around 4am I saved my configs and walk to the hotel and organised a wake up call for 7:30am since I needed to get ready for check out to fly home on Friday evening.  After being pumped up on caffeine and sugar for most of the night, towards the end I was stuck in a task because the router was doing what I was telling it do to do, rather than what I though I was telling it to do :)

We were told roughly that if we were to put the level of difficulty of the Monday Assement lab as 3, that this might be considered about a 9, however this was not really analagous to the actual R&S Lab and that doing well in this was not a guarantee that you would pass the R&S Lab exam but to be used as a datapoint in determining your readiness.

That said, one of the advantages of doing this bootcamp is with the association with the Cisco 360 program, if you do well in the assessment labs, you can qualify for a free lab re-take voucher.

Day 5 (Friday)
Around 9am the classroom was half empty and those that are present are looking like extras from a zombie movie except for Narbik who looks like he has just had 2 weeks of vacation.

No one really feels like labbing this morning, so it's mainly chatting about how things went in the Assessment lab until more and more people come back to class.

The topics covered today are Multicast, and the other sub-topics we didn't quite finish on previous days (things like ZBPFW and class based policing)

Upon closing Narbik mentioned about the free re-attendance to the bootcamps for past students in the same track (as I mentioned there were some guys here doing just that) and that he expected that we treat the bootcamp as the beginning of our relationship with him, the expectation of using email and forums with him and other students to help clarify problems or concerns.

He also again re-iterated that the bootcamp didn't cover all of the topics of the blueprint, there just isnt enough time, however the workbooks (which I believe are subject to free electronic updates) do cover these topics.


Overall conclusion
I really enjoyed this bootcamp.  If I were to do things differently, I would probably have looked at attending one of Narbik's bootcamps earlier into my studies, gone through his workbooks and then re-visted his bootcamp within a few weeks of attempting the lab.  This hasn't worked out that way in my case but I think the current preparation that I have done (using material from other vendors as my prime resource) has helped me, with this bootcamp enabling me to solidify what I do know and helping to uncover what I thought I knew but didn't.  I will be attemping my lab exam in a few weeks, and I will most certainly be using the material in the workbooks as well as my notes to do my final bit of preparation.  Should I not pass on the first attempt which is a possibility, I will certainly look at taking advantage of the lab re-take voucher and again make use of the material.


The ratio of labbing to lecture was around 40:60 but that is kind of skewed with the assessment (mock labs).

Being with a bunch of people where you can talk directly and/or draw on a whiteboard your problem and get help either from Narbik or your fellow students just isn't as easy to get from mailing lists and forums (which I most certainly have used)

How do I rate the workbooks?  I haven't had enough use to give a comprehensive review but so far I think they're great, they're different in format to the other vendor's material I've used (Narbik's topology is slightly smaller but I believe is on par with the CCIE R&S Configuration topology), I don't necessarily think they're better or worse, the labs in Narbik's workbooks appear to focus on individual subtopics.

I haven't been on another bootcamp so I cant compare it to anything else.  I have used Video on Demand material which simulated a one week bootcamp and while I did get value from the VoD, a prerecorded non/dynamic environment is just not the same as being able to interact directly.

It does appear to be a program that you could follow with a single investment of the bootcamp which includes the workbooks and a training sessions from an instructor that really knows his stuff, and is passionate about transferring his knowledge to you.


I liked the idea of the mocklabs but that's probably because I'm closer to sitting the exam than most of my other classmates (plus that lab voucher is a good draw card to the bootcamp) however I do see an advantage to covering the same level of material in smaller assessment labs spread out across the days so you don't have such an overload on the second last day.  Spreading it out may mean that there is more information retained on Day 5 - Narbik mentioned that this might be on the cards for future bootcamps so get in touch with Micronics for the latest information in that area.

I kind of hope I don't come across as some kind of religious fanatic praising Narbik or his methods, I'm just attempting to help those that may be trying to determine if they want to go on a bootcamp, what they need to put in to get value, and what they get for their money for the one I went to.  Pass or fail the Lab, it will be down to the individual and not the training vendor, regardless of the training vendor and the material used, all that can do is help unlock the knowledge required but its the perserverance, determination and desire of the CCIE candidate to achieve the status.

I hope that this rather long posting should anyone read it helps someone determine if they want to go on a bootcamp what they might expect, or even if they were interested in starting their CCIE studies, how a program such as this one might help them along.  I think going for a CCIE is possibly like scaling a mountain, it's not really a race, it's something you pretty much have to do yourself but there can be people around to help you out but it's up to you to get to the peak and it's great that there is a whole community of training vendors, CCIE candidates and CCIEs and others out there helping us all to success.