OSPF External Route Summarization

Currently my CCIE studies are taking a slight detour while renew and upgrade one of my Juniper Certifications, so lately my current study area has been focused on Junos.  Just to be a little different today I'm going to discuss an OSPF operation and demonstrate it in IOS and then Junos.

The prime message I want to give here is that summarizing of OSPF external routes in certain circumstances occur at multiple places within the network.  Some people may believe that you can only summarize on the OSPF ASBR, however if you have that ASBR in an OSPF NSSA Area, the ABR can also summarize those external routes.  This is because summarization of external routes can occur on the OSPF router that originates the Type 5 or Type 7 LSAs and the ABR will convert Type 7s to Type 5s.

The demonstration topology, configuration and verification (using IOS first, with Junos following on)  is below:


R1 is an ASBR with Fa0/0 in Area 12 (NSSA)
R2 is an ABR with Fa0/0 in Area 12 (NSSA) and S0/0.23 in Area 0 (Backbone)
R3 is a Regular OSPF Router with S0/0.32 in Area 0

R1
hostname R1
!
interface FastEthernet0/0
 ip address 10.1.12.1 255.255.255.0
 ip ospf 1 area 12
!
router ospf 1
 router-id 1.1.1.1
 area 12 nssa
 redistribute static metric 100 subnets
!
ip route 1.1.1.0 255.255.255.0 Null0

R2
hostname R2
!
interface FastEthernet0/0
 ip address 10.1.12.2 255.255.255.0
 ip ospf 1 area 12
 duplex auto
 speed auto
!
interface Serial0/0
 no ip address
 encapsulation frame-relay
 no frame-relay inverse-arp
!
interface Serial0/0.23 point-to-point
 ip address 10.1.23.2 255.255.255.0
 ip ospf 1 area 0
 frame-relay interface-dlci 203
!
router ospf 1
 router-id 2.2.2.2
 area 12 nssa

R3
hostname R3
!
interface Serial0/0
 no ip address
 encapsulation frame-relay
 clock rate 2000000
 no frame-relay inverse-arp
!
interface Serial0/0.32 point-to-point
 ip address 10.1.23.3 255.255.255.0
 ip ospf 1 area 0
 frame-relay interface-dlci 302
!
router ospf 1
 router-id 3.3.3.3

We'll take a baseline verification to make sure we can see 1.1.1.0/24

R2#sh ip route ospf
     1.0.0.0/24 is subnetted, 1 subnets
O N2    1.1.1.0 [110/100] via 10.1.12.1, 00:01:25, FastEthernet0/0


R3#sh ip route ospf
     1.0.0.0/24 is subnetted, 1 subnets
O E2    1.1.1.0 [110/100] via 10.1.23.2, 00:01:14, Serial0/0.32
     10.0.0.0/24 is subnetted, 2 subnets
O IA    10.1.12.0 [110/74] via 10.1.23.2, 00:19:39, Serial0/0.32

All good and we can see that R2 is performing the Type 7 (N2) to Type 5 (E2) conversion.


Let's summarize R1

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#router ospf 1
R1(config-router)#summary-address 1.1.0.0 255.255.0.0
R1(config-router)#end

R2#sh ip route ospf
     1.0.0.0/16 is subnetted, 1 subnets
O N2    1.1.0.0 [110/100] via 10.1.12.1, 00:00:31, FastEthernet0/0


R3#sh ip route ospf
     1.0.0.0/16 is subnetted, 1 subnets
O E2    1.1.0.0 [110/100] via 10.1.23.2, 00:00:35, Serial0/0.32
     10.0.0.0/24 is subnetted, 2 subnets
O IA    10.1.12.0 [110/74] via 10.1.23.2, 00:20:57, Serial0/0.32

So the point of the exercise is to demonstrate summarization on R2, so let's do it

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#router ospf 1
R2(config-router)#summary-address 1.0.0.0 255.0.0.0
R2(config-router)#end

R3#sh ip route ospf
O E2 1.0.0.0/8 [110/100] via 10.1.23.2, 00:00:10, Serial0/0.32
     10.0.0.0/24 is subnetted, 2 subnets
O IA    10.1.12.0 [110/74] via 10.1.23.2, 00:22:29, Serial0/0.32

Yes it does - R2 is not an ABR but it is originating Type 5 LSAs (when it converts the Type 7s from R1) Therefore we have verified that summarization can occur on OSPF routers that originates Type 5 or Type 7 LSAs


We can demonstrate the same thing using Junos but it is somewhat different in the method execution

R1 is an ASBR with em1.12 in Area 12 (NSSA)
R2 is an ABR with em1.12 in Area 12 (NSSA) and em1.23 in Area 0 (Backbone)
R3 is a Regular OSPF Router with em1.23 in Area 0

Unlike IOS, we need to specifically create a policy to redistribute static routes into OSPF, below are our baseline configurations

root@R1> show configuration | display set
set system host-name R1
set interfaces em1 vlan-tagging
set interfaces em1 unit 12 vlan-id 12
set interfaces em1 unit 12 family inet address 10.1.12.1/24
set routing-options static route 1.1.1.0/24 discard
set routing-options router-id 1.1.1.1
set protocols ospf export ExternalToOSPF
set protocols ospf area 0.0.0.12 nssa
set protocols ospf area 0.0.0.12 interface em1.12
set policy-options policy-statement ExternalToOSPF term Static from protocol static
set policy-options policy-statement ExternalToOSPF term Static to protocol ospf
set policy-options policy-statement ExternalToOSPF term Static then metric 100
set policy-options policy-statement ExternalToOSPF term Static then accept

root@R2> show configuration | display set
set system host-name R2
set interfaces em1 vlan-tagging
set interfaces em1 unit 12 vlan-id 12
set interfaces em1 unit 12 family inet address 10.1.12.2/24
set interfaces em1 unit 23 vlan-id 23
set interfaces em1 unit 23 family inet address 10.1.23.2/24
set routing-options router-id 2.2.2.2
set protocols ospf area 0.0.0.12 nssa
set protocols ospf area 0.0.0.12 interface em1.12
set protocols ospf area 0.0.0.0 interface em1.23

root@R3> show configuration | display set
set system host-name R3
set interfaces em1 vlan-tagging
set interfaces em1 unit 23 vlan-id 23
set interfaces em1 unit 23 family inet address 10.1.23.3/24
set routing-options router-id 3.3.3.3
set protocols ospf area 0.0.0.0 interface em1.23

Lets verify the baseline routing table:

root@R2> show ospf route network
Topology default Route Table:

Prefix             Path  Route      NH       Metric NextHop       Nexthop
                   Type  Type       Type            Interface     Address/LSP
1.1.1.0/24         Ext2  Network    IP          100 em1.12        10.1.12.1
10.1.12.0/24       Intra Network    IP            1 em1.12
10.1.23.0/24       Intra Network    IP            1 em1.23

root@R2> show ospf route network 1.1.1.0/24 extensive
Topology default Route Table:

Prefix             Path  Route      NH       Metric NextHop       Nexthop
                   Type  Type       Type            Interface     Address/LSP
1.1.1.0/24         Ext2  Network    IP          100 em1.12        10.1.12.1
  area 0.0.0.12, origin 1.1.1.1, type 7, P-bit, fwd NZ, priority medium



root@R3> show ospf route network
Topology default Route Table:

Prefix             Path  Route      NH       Metric NextHop       Nexthop
                   Type  Type       Type            Interface     Address/LSP
1.1.1.0/24         Ext2  Network    IP          100 em1.23        10.1.23.2
10.1.12.0/24       Inter Network    IP            2 em1.23        10.1.23.2
10.1.23.0/24       Intra Network    IP            1 em1.23

root@R3> show ospf route network 1.1.1.0/24 extensive
Topology default Route Table:

Prefix             Path  Route      NH       Metric NextHop       Nexthop
                   Type  Type       Type            Interface     Address/LSP
1.1.1.0/24         Ext2  Network    IP          100 em1.23        10.1.23.2
  area 0.0.0.0, origin 2.2.2.2, fwd NZ, priority medium

Without examining the route in more detail we weren't able to determine as easily if the external route was from an NSSA area or not.

Summarization on R1 requires a bit more effort than what we used IOS as well - First we need to set up an aggregate route (in Junos, aggregate routes are considered their own protocol) and redistribute that into OSPF.

root@R1> configure
Entering configuration mode

[edit]
set routing-options aggregate route 1.1.0.0/16
set policy-options policy-statement ExternalToOSPF term Aggregate to protocol ospf
set policy-options policy-statement ExternalToOSPF term Aggregate then metric 100
set policy-options policy-statement ExternalToOSPF term Aggregate then accept

If we leave it like this, we'll find that the 1.1.1.0/24 static route will still be advertised, so we need a method to be able to remove that - In this example we will set up a prefix list that if matched will stop the route being advertised, while still letting other static routes through

[edit]
set policy-options prefix-list Drop 1.1.1.0/24
set policy-options policy-statement ExternalToOSPF term Drop from prefix-list Drop
set policy-options policy-statement ExternalToOSPF term Drop then reject

All that's left to do is to ensure the Drop term gets evaluated first

[edit]
root@R1# insert policy-options policy-statement ExternalToOSPF term Drop before term Static

Let's examine the complete policy

root@R1# show policy-options
prefix-list Drop {
    1.1.1.0/24;
}
policy-statement ExternalToOSPF {
    term Drop {
        from {
            prefix-list Drop;
        }
        then reject;
    }
    term Static {
        from protocol static;
        to protocol ospf;
        then {
            metric 100;
            accept;
        }
    }
    term Aggregate {
        from protocol aggregate;
        to protocol ospf;
        then {
            metric 100;
            accept;
        }
    }
}


looks good, so lets apply it.

[edit]
root@R1# commit and-quit
commit complete

Now to see if the aggregate is appearing beyond R1

root@R2> show ospf route network
Topology default Route Table:

Prefix             Path  Route      NH       Metric NextHop       Nexthop
                   Type  Type       Type            Interface     Address/LSP
1.1.0.0/16         Ext2  Network    IP          100 em1.12        10.1.12.1
10.1.12.0/24       Intra Network    IP            1 em1.12
10.1.23.0/24       Intra Network    IP            1 em1.23

root@R2> show ospf route network 1.1.0.0/16 extensive
Topology default Route Table:

Prefix             Path  Route      NH       Metric NextHop       Nexthop
                   Type  Type       Type            Interface     Address/LSP
1.1.0.0/16         Ext2  Network    IP          100 em1.12        10.1.12.1
  area 0.0.0.12, origin 1.1.1.1, type 7, P-bit, fwd NZ, priority medium


root@R3> show ospf route network
Topology default Route Table:

Prefix             Path  Route      NH       Metric NextHop       Nexthop
                   Type  Type       Type            Interface     Address/LSP
1.1.0.0/16         Ext2  Network    IP          100 em1.23        10.1.23.2
10.1.12.0/24       Inter Network    IP            2 em1.23        10.1.23.2
10.1.23.0/24       Intra Network    IP            1 em1.23

root@R3> show ospf route network 1.1.0.0/16 extensive
Topology default Route Table:

Prefix             Path  Route      NH       Metric NextHop       Nexthop
                   Type  Type       Type            Interface     Address/LSP
1.1.0.0/16         Ext2  Network    IP          100 em1.23        10.1.23.2
  area 0.0.0.0, origin 2.2.2.2, fwd NZ, priority medium

Believe it or not, performing the summarization on R2 for R3 is very simple.

root@R2> configure
Entering configuration mode

[edit]
root@R2# set protocols ospf area 12 nssa area-range 1.0.0.0/8

[edit]
root@R2# commit and-quit
commit complete
Exiting configuration mode

To verify:

root@R3> show ospf route network
Topology default Route Table:

Prefix             Path  Route      NH       Metric NextHop       Nexthop
                   Type  Type       Type            Interface     Address/LSP
1.0.0.0/8          Ext2  Network    IP          101 em1.23        10.1.23.2
10.1.12.0/24       Inter Network    IP            2 em1.23        10.1.23.2
10.1.23.0/24       Intra Network    IP            1 em1.23

root@R3> show ospf route network 1.0.0.0/8 extensive
Topology default Route Table:

Prefix             Path  Route      NH       Metric NextHop       Nexthop
                   Type  Type       Type            Interface     Address/LSP
1.0.0.0/8          Ext2  Network    IP          101 em1.23        10.1.23.2
  area 0.0.0.0, origin 2.2.2.2, priority medium

So there you go, I'm not sure how useful this actually is in a production environment but in a lab exam, it may be an extra tool to get out of a sticky situation...

MFR (Multilink Frame Relay FRF.16) Revisited

Some time ago I blogged briefly about MFR but only had it working with back-to-back routers because at the time, I couldn't work out how to get it to work through a Frame-Relay Switch.  After catching this statement on CCO, I finally understood that the FR Switch actually had to be MFR enabled as well.

Because of this, you can do some interesting things.

This topology will have 3 devices (R1, R2 and a FrameSwitch)

R1==MFR==FRSW--FR--R2

My crude diagram above is meant to indicate that R1 has two physical links to FRSW and is using MFR, while R2 has a single physical link to FRSW using regular frame-relay.

Why might you do this?  R1 could be a hub site serving multiple spokes like R2, and the Service Provider for whatever reason cant deliver big pipes to the hub site.  As opposed to having R1 and FRSW having just two links with the spokes distributed across the interfaces, besides any bandwidth gains, MFR also enables some resiliency (losing a single physical link can still keep all spokes connected at diminished capacity)


These are the configurations:

R1
hostname R1
!
interface MFR1
 no ip address
 frame-relay multilink bid R1-MFR1
 no frame-relay inverse-arp
!
interface MFR1.12 point-to-point
 ip address 10.1.12.1 255.255.255.0
 snmp trap link-status
 frame-relay interface-dlci 102
!
interface Serial0/0
 description FRSW S0/0
 no ip address
 encapsulation frame-relay MFR1
 no arp frame-relay
 frame-relay multilink lid R1-S0/0
!
interface Serial0/1
 description FRSW S0/1
 no ip address
 encapsulation frame-relay MFR1
 no arp frame-relay
 frame-relay multilink lid R1-S0/1

FRSW
hostname FRSW
frame-relay switching
!
interface MFR1
 no ip address
 frame-relay multilink bid FRSW-MFR1
 frame-relay intf-type dce
 frame-relay route 102 interface Serial1/0 201
!
interface Serial0/0
 description R1 S0/0
 no ip address
 encapsulation frame-relay MFR1
 clock rate 2000000
 no arp frame-relay
 frame-relay multilink lid FRSW-S0/0
!
interface Serial0/1
 description R1 S0/1
 no ip address
 encapsulation frame-relay MFR1
 clock rate 2000000
 no arp frame-relay
 frame-relay multilink lid FRSW-S0/1
!
interface Serial1/0
 description R2 S0/0
 no ip address
 encapsulation frame-relay
 serial restart-delay 0
 frame-relay intf-type dce
 frame-relay route 201 interface MFR1 102


R2
hostname R2
!
interface Serial0/0
 description FRSW S1/0
 no ip address
 encapsulation frame-relay
 no fair-queue
 no frame-relay inverse-arp
!
interface Serial0/0.21 point-to-point
 ip address 10.1.12.2 255.255.255.0
 frame-relay interface-dlci 201
!



Within the MFR configuration we have added some optional statements which identify the MFR bundle:

interface MFR
 frame-relay multilink bid MFRBundleIDString


And the links which make up the bundle:

interface Serial
 frame-relay multilink lid LinkIDString


This kind of thing can help with Layer 2 troubleshooting if you have a lot of peers.

R1#sh frame multi de | i BID|Peer|LID
 BID = R1-MFR1
 No. of bundle links = 2, Peer's bundle-id = FRSW-MFR1
  Serial0/1, HW state = up, link state = Up, LID = R1-S0/1
    Peer LID = FRSW-S0/1, RTT = 20 ms
  Serial0/0, HW state = up, link state = Up, LID = R1-S0/0
    Peer LID = FRSW-S0/0, RTT = 24 ms

FRSW#sh frame multi de | i BID|Peer|LID
 BID = FRSW-MFR1
 No. of bundle links = 2, Peer's bundle-id = R1-MFR1
  Serial0/1, HW state = up, link state = Up, LID = FRSW-S0/1
    Peer LID = R1-S0/1, RTT = 0 ms
  Serial0/0, HW state = up, link state = Up, LID = FRSW-S0/0
    Peer LID = R1-S0/0, RTT = 0 ms


So from above, we can see that  that the bundle is up with both links and we appear to be connected to the right peer interfaces.


So lets see the PVC status on the Frame Switch:

FRSW#sh frame pvc

PVC Statistics for interface Serial1/0 (Frame Relay DCE)

              Active     Inactive      Deleted       Static
  Local          0            0            0            0
  Switched       1            0            0            0
  Unused         0            0            0            0

DLCI = 201, DLCI USAGE = SWITCHED, PVC STATUS = ACTIVE, INTERFACE = Serial1/0

  input pkts 33            output pkts 29           in bytes 7428
  out bytes 4708           dropped pkts 12          in pkts dropped 12
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 0         out bcast bytes 0
  30 second input rate 0 bits/sec, 0 packets/sec
  30 second output rate 0 bits/sec, 0 packets/sec
  switched pkts 24
  Detailed packet drop counters:
  no out intf 0            out intf down 0          no out PVC 0
  in PVC down 0            out PVC down 12          pkt too big 0
  shaping Q full 0         pkt above DE 0           policing drop 0
  pvc create time 00:10:17, last time pvc status changed 00:06:07

PVC Statistics for interface MFR1 (Frame Relay DCE)

              Active     Inactive      Deleted       Static
  Local          0            0            0            0
  Switched       1            0            0            0
  Unused         0            0            0            0

DLCI = 102, DLCI USAGE = SWITCHED, PVC STATUS = ACTIVE, INTERFACE = MFR1

  input pkts 30            output pkts 24           in bytes 5029
  out bytes 4494           dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 0         out bcast bytes 0
  30 second input rate 0 bits/sec, 0 packets/sec
  30 second output rate 0 bits/sec, 0 packets/sec
  switched pkts 30
  Detailed packet drop counters:
  no out intf 0            out intf down 0          no out PVC 0
  in PVC down 0            out PVC down 0           pkt too big 0
  shaping Q full 0         pkt above DE 0           policing drop 0
  pvc create time 00:06:14, last time pvc status changed 00:05:54

FRSW#sh frame route
Input Intf      Input Dlci      Output Intf     Output Dlci     Status
Serial1/0       201             MFR1            102             active
MFR1            102             Serial1/0       201             active

Looks good here, lets see what R1 and R2 have to say

R1#sh frame pvc

PVC Statistics for interface MFR1 (Frame Relay DTE)

              Active     Inactive      Deleted       Static
  Local          1            0            0            0
  Switched       0            0            0            0
  Unused         0            0            0            0

DLCI = 102, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = MFR1.12

  input pkts 12            output pkts 19           in bytes 2802
  out bytes 3929           dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 9         out bcast bytes 2889
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  pvc create time 00:08:34, last time pvc status changed 00:04:20
R1#sh frame map
MFR1.12 (up): point-to-point dlci, dlci 102(0x66,0x1860), broadcast
          status defined, active

R2#sh frame pvc

PVC Statistics for interface Serial0/0 (Frame Relay DTE)

              Active     Inactive      Deleted       Static
  Local          1            0            0            0
  Switched       0            0            0            0
  Unused         0            0            0            0

DLCI = 201, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0.21

  input pkts 18            output pkts 22           in bytes 3608
  out bytes 6062           dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 17        out bcast bytes 5542
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  pvc create time 00:09:40, last time pvc status changed 00:05:30
R2#sh frame map
Serial0/0.21 (up): point-to-point dlci, dlci 201(0xC9,0x3090), broadcast
          status defined, active

And lets verify layer 3 connectivity:

R2#ping 10.1.12.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.12.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/12/44 ms


There are more things you can do with MFR which include the specifying the method of using the bundles (variable bandwidth class support). By default, IOS will use bandwidth class a (the MFR will stay active with downed link members as long as a single link is up),  other options include class b (all links must be active to keep the MFR interface up) or class c (a minimum threshold of links to keep the MFR interface up)

interface MFR
 frame-relay multilink bandwidth-class BandwidthClass LinkThresholdForClassC


Load Balancing:
When the MFR interface is not using FIFO queuing, the router will simly pick the interface with the empty or shortest queue to send traffic to.  If FIFO is being used then there is a default output-threshold of 300 bytes for each link, once this is exceeded, the next interface is selected.  It is possible to modify this value to a different level (this is not FRF.12 frame-relay fragmentation)

interface MFR
 frame-relay multilink output-threshold Threshold

Multilink Frame Relay (MFR) and Multilink PPP (ML-PPP)

This post is about non-etherchannel link bundling a way to increase bandwidth and resiliency with the layer 3 result appearing as a single Layer 3 link between nodes using multiple WAN links.

So lets start with Routers R4 and R7, each has 4 serial interfaces that are back to back (R4 s0/0 to R7 s0/0, R4 s0/1 to R7 s0/1, R4 s1/0 to R7 s1/0 and R4 s1/1 to R7 s1/1)

Start with a multilink frame relay (MFR) for the first two links (since we're not using a Frame Switch here so we're going to turn FR LMI off)

R4
hostname R4
int mfr 1
 no keepalive
 no shut
int s0/0
 shutdown
 encapsulation frame-relay mfr 1
 no shut
int s0/1
 shutdown
 encapsulation frame-relay mfr 1
 no shut
int mfr1.47 point-to-point
 ip address 47.47.47.4 255.255.255.0
 frame-relay interface-dlci 47


R7
hostname R7
int mfr 1
 no keepalive
 no shut
int s0/0
 shutdown
 encapsulation frame-relay mfr 1
 no shut
int s0/1
 shutdown
 encapsulation frame-relay mfr 1
 no shut
int mfr1.47 point-to-point
 ip address 47.47.47.7 255.255.255.0
 frame-relay interface-dlci 47


So lets check that is operating.

R4#sh frame pvc

PVC Statistics for interface MFR1 (Frame Relay DTE)

              Active     Inactive      Deleted       Static
  Local          1            0            0            0
  Switched       0            0            0            0
  Unused         0            0            0            0

DLCI = 47, DLCI USAGE = LOCAL, PVC STATUS = STATIC, INTERFACE = MFR1.47

  input pkts 3             output pkts 3            in bytes 963
  out bytes 963            dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 3         out bcast bytes 963
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  pvc create time 00:00:28, last time pvc status changed 00:00:14
R4#ping 47.47.47.7

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 47.47.47.7, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/16 ms
R4#sh frame-relay multilink
Bundle: MFR1, State = up, class = A, fragmentation disabled
 BID = MFR1
 Bundle links:
  Serial0/1, HW state = up, link state = Up, LID = Serial0/1
  Serial0/0, HW state = up, link state = Up, LID = Serial0/0

Okay, so multilink frame relay is relatively straight forward to get going.  Lets park that for the moment and look at getting multilink PPP operational

R4
int mfr1.47 point-to-point
 no ip address
interface Multilink1
 ip address 47.47.47.4 255.255.255.0
 ppp multilink
 ppp multilink group 1
int s1/0
 shutdown
 encapsulation ppp
 ppp multilink
 ppp multilink group 1
 no shut
int s1/1
 shutdown
 encapsulation ppp
 ppp multilink
 ppp multilink group 1
 no shut

R7
int mfr1.47 point-to-point
 no ip address
interface Multilink1
 ip address 47.47.47.7 255.255.255.0
 ppp multilink
 ppp multilink group 1
int s1/0
 shutdown
 encapsulation ppp
 ppp multilink
 ppp multilink group 1
 no shut
int s1/1
 shutdown
 encapsulation ppp
 ppp multilink
 ppp multilink group 1
 no shut

So lets check that this is working.
 
R7#ping 47.47.47.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 47.47.47.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/9/24 ms
R7#sh ppp multi

Multilink1
  Bundle name: R4
  Remote Endpoint Discriminator: [1] R4
  Local Endpoint Discriminator: [1] R7
  Bundle up for 00:00:26, total bandwidth 3088, load 1/255
  Receive buffer limit 24000 bytes, frag timeout 1000 ms
    0/0 fragments/bytes in reassembly list
    0 lost fragments, 7 reordered
    0/0 discarded fragments/bytes, 0 lost received
    0xF received sequence, 0xD sent sequence
  Member links: 2 active, 0 inactive (max not set, min not set)
    Se1/0, since 00:00:21
    Se1/1, since 00:00:20
No inactive multilink interfaces

No problems, we can see that the bandwidth is 3088kbps (1544 kbps default per serial interface). Ok, so just for fun, how about including the Frame Relay links with this bundle?  It's certainly possible, we just need to put ppp over frame and include it in the multilink group.

R4/47
interface Virtual-Template1
 no ip address
 ppp multilink
 ppp multilink group 1
interface MFR1.47 point-to-point
 frame-relay interface-dlci 47 ppp Virtual-Template1

However it's not quite as simple as it seems to get it going..

*Mar  1 00:14:13.559: %LINK-3-UPDOWN: Interface Virtual-Access2, changed state to up
*Mar  1 00:14:25.547: %FR-3-MLPOFR_ERROR: MLPoFR not configured properly on Link Virtual-Access2 Bundle Multilink1 :Frame Relay traffic shaping must be enabled
*Mar  1 00:14:26.539: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to up

We need some traffic shaping on the FR (MFR1) interface - remember if we just turn traffic shaping on, all PVCs will be shaped at 56kbps.  For this example, we'll assume that there is 256kbps per PVC available

R4/47
map-class frame-relay MFR
 frame-relay traffic-rate 512000
interface MFR1
 frame-relay traffic-shaping
interface MFR1.47 point-to-point
 frame-relay interface-dlci 47 ppp Virtual-Template1
 class MFR

Lets bounce the link and see if it's happy this time round.

R7(config-if)#int mfr1
R7(config-if)#shut
*Mar  1 00:22:30.247: %LINK-5-CHANGED: Interface MFR1, changed state to administratively down
*Mar  1 00:22:30.247: %LINK-3-UPDOWN: Interface Virtual-Access2, changed state to down
*Mar  1 00:22:31.247: %LINEPROTO-5-UPDOWN: Line protocol on Interface MFR1, changed state to down
*Mar  1 00:22:31.251: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to down
*Mar  1 00:22:33.255: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, changed state to down
*Mar  1 00:22:33.259: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, changed state to down
R7(config-if)#no shut
*Mar  1 00:22:46.619: %LINK-3-UPDOWN: Interface MFR1, changed state to down
*Mar  1 00:22:54.215: %LINK-3-UPDOWN: Interface MFR1, changed state to up
*Mar  1 00:22:54.263: %LINK-3-UPDOWN: Interface Virtual-Access2, changed state to up
*Mar  1 00:22:55.207: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, changed state to up
*Mar  1 00:22:55.215: %LINEPROTO-5-UPDOWN: Line protocol on Interface MFR1, changed state to up
*Mar  1 00:22:55.219: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, changed state to up
*Mar  1 00:22:55.263: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to up

No complaints this time...

R7(config-if)#do ping 47.47.47.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 47.47.47.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/6/16 ms
R7(config-if)#do sh users
    Line       User       Host(s)              Idle       Location
*  0 con 0                idle                 00:00:00

  Interface    User               Mode         Idle     Peer Address
  Vi2                             PPPoFR       00:01:06
  Se1/0                           Sync PPP     00:00:05
  Se1/1                           Sync PPP     00:00:05
  Mu1                             Sync PPP     00:00:54 47.47.47.4

Looks good so far, lets confirm what the multilink PPP interface thinks its bandwidth is (it should be 1544 * 2 for the direct PPP links and 512 for the MFR link)

R7(config-if)#do sh ppp multi

Multilink1
  Bundle name: R4
  Remote Endpoint Discriminator: [1] R4
  Local Endpoint Discriminator: [1] R7
  Bundle up for 00:24:11, total bandwidth 3344, load 1/255
  Receive buffer limit 36000 bytes, frag timeout 1000 ms
    0/0 fragments/bytes in reassembly list
    0 lost fragments, 13 reordered
    0/0 discarded fragments/bytes, 0 lost received
    0x32 received sequence, 0x30 sent sequence
  Member links: 3 active, 1 inactive (max not set, min not set)
    Se1/0, since 00:24:06, 5790 weight, 1496 frag size
    Se1/1, since 00:24:05, 5790 weight, 1496 frag size
    Vi2, since 00:04:21, 960 weight, 950 frag size
    Vt1 (inactive)
No inactive multilink interfaces

We can see that the multilink interface thinks it has 3344 kbps of bandwidth and not 3600.

R7#sh int virtual-access 2
Virtual-Access2 is up, line protocol is up
  Hardware is Virtual Access interface
  MTU 1500 bytes, BW 256 Kbit/sec, DLY 100000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Open
  Link is a member of Multilink bundle Multilink1
  PPPoFR vaccess, cloned from Virtual-Template1
  Vaccess status 0x44
  Bound to MFR1.47 DLCI 47, Cloned from Virtual-Template1, loopback not set
  Keepalive set (10 sec)
  DTR is pulsed for 5 seconds on reset
  Last input 00:02:02, output never, output hang never
  Last clearing of "show interface" counters 00:20:59
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     253 packets input, 5890 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     254 packets output, 5393 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

Virtual-Access2 believes that there is 256kbps of bandwidth available rather than the expected 512kbps.  Lets modify the bandwidth associated with Virtual-Template 1 (since the Virtual-Access interface inherits data from it) and see if that helps.

R7(config-if)#int virtual-template 1
R7(config-if)#bandwidth 512
R7(config-if)#int mfr1
R7(config-if)#shut
R7(config-if)#no shut
*Mar  1 00:40:48.283: %LINK-5-CHANGED: Interface MFR1, changed state to administratively down
*Mar  1 00:40:48.287: %LINK-3-UPDOWN: Interface Virtual-Access2, changed state to down
*Mar  1 00:40:49.295: %LINEPROTO-5-UPDOWN: Line protocol on Interface MFR1, changed state to down
*Mar  1 00:40:49.299: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to down
*Mar  1 00:40:49.647: %LINK-3-UPDOWN: Interface MFR1, changed state to down
*Mar  1 00:40:51.295: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, changed state to down
*Mar  1 00:40:51.299: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, changed state to down
*Mar  1 00:40:58.259: %LINK-3-UPDOWN: Interface MFR1, changed state to up
*Mar  1 00:40:58.311: %LINK-3-UPDOWN: Interface Virtual-Access2, changed state to up
*Mar  1 00:40:59.251: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, changed state to up
*Mar  1 00:40:59.259: %LINEPROTO-5-UPDOWN: Line protocol on Interface MFR1, changed state to up
*Mar  1 00:40:59.267: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, changed state to up
*Mar  1 00:40:59.311: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to up
R7(config-if)#do sh ppp multi

Multilink1
  Bundle name: R4
  Remote Endpoint Discriminator: [1] R4
  Local Endpoint Discriminator: [1] R7
  Bundle up for 00:32:49, total bandwidth 3600, load 1/255
  Receive buffer limit 36000 bytes, frag timeout 1000 ms
    0/0 fragments/bytes in reassembly list
    0 lost fragments, 13 reordered
    0/0 discarded fragments/bytes, 0 lost received
    0x3A received sequence, 0x38 sent sequence
  Member links: 3 active, 1 inactive (max not set, min not set)
    Se1/0, since 00:32:44, 5790 weight, 1496 frag size
    Se1/1, since 00:32:43, 5790 weight, 1496 frag size
    Vi2, since 00:00:09, 1920 weight, 1496 frag size
    Vt1 (inactive)
No inactive multilink interfaces

Okay, repeat the same thing on R4 and that should be it.  Would anyone want to do this in the real world?  I certainly hope not!