Thursday, February 20, 2020

Troubleshooting Metric Redistribution

Redistribution is a difficult topic to master. The configuration is quite easy but if you are not careful you will end up with sub-optimal routing and/or routing loops.
When it comes to redistribution issues, there are two possible problems that we can encounter:
In this lesson we’ll take a look at metric based redistribution problems and how to fix them. To demonstrate this I will use the following topology:
metric based redistribution scenario
In this topology we have 4 routers. All routers are running RIPv2 while R3 and R4 are also running OSPF on the link between them.

R1 is only used to advertise a network (1.1.1.0 /24) into RIP. This is what the routing configuration looks like right now (without redistribution):
R1#show run | section rip
router rip
 version 2
 offset-list 0 out 5
 network 1.0.0.0
 network 192.168.12.0
 no auto-summary
R2#show run | section rip
router rip
 version 2
 network 192.168.12.0
 network 192.168.23.0
 network 192.168.24.0
 no auto-summary
R3#show run | section rip
router rip
 version 2
 network 192.168.23.0
 no auto-summary
R4#show run | section rip
router rip
 version 2
 network 192.168.24.0
 no auto-summary
Each router runs RIP version 2, nothing special. Here’s the OSPF configuration of R3 and R4:
R3#show run | section ospf
router ospf 1
 log-adjacency-changes
 network 192.168.34.0 0.0.0.255 area 0
R4#show run | section ospf
router ospf 1
 log-adjacency-changes
 network 192.168.34.0 0.0.0.255 area 0
Let’s take a look what the routing tables of R2, R3 and R4 look like now. Let’s look at the RIP routes first:
R2#show ip route rip
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/6] via 192.168.12.1, 00:00:09, FastEthernet1/0
R3#show ip route rip
R    192.168.12.0/24 [120/1] via 192.168.23.2, 00:00:27, FastEthernet0/0
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/7] via 192.168.23.2, 00:00:27, FastEthernet0/0
R    192.168.24.0/24 [120/1] via 192.168.23.2, 00:00:27, FastEthernet0/0
R4#show ip route rip
R    192.168.12.0/24 [120/1] via 192.168.24.2, 00:00:02, FastEthernet0/0
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/7] via 192.168.24.2, 00:00:02, FastEthernet0/0
R    192.168.23.0/24 [120/1] via 192.168.24.2, 00:00:02, FastEthernet0/0
R2 has learned about network 1.1.1.0 /24 from R1 with a hop count of 6. I used an offset-list on R1 to increase the hop count for this topology, I’ll show you why in a bit.
R3 and R4 also learn about this network and the links in between R1/R2, R2/R3 and R/4. Nothing special so far…
Since R3 and R4 only run OSPF on the directly connected link they don’t have any OSPF routes right now:
R3#show ip route ospf
R4#show ip route ospf
So far so good. Now we will continue by configuring redistribution. I’ll do this step-by-step so you can see what will happen. First we will redistribute RIP into OSPF on R3 and R4:
Redistribute RIP into OSPF
R3 & R4#
(config)#router ospf 1
(config-router)#redistribute rip subnets
The command above redistribute everything from RIP into OSPF. Let’see what R3 and R4 now have in their routing tables:
R3#show ip route ospf
O E2 192.168.24.0/24 [110/20] via 192.168.34.4, 00:01:02, FastEthernet0/1
R3 has now learned through OSPF to reach network 192.168.24.0 /24 with R4 as its next hop. Before we configured redistribution, it had a RIP route for this network with R2 as its next hop. The reason that the RIP route has been removed is because OSPF has a better administrative distance (120 vs 110):
R3 RIP OSPF Route Receive
Is this an issue? For this particular topology right now it won’t cause any issues. R3 can reach this network through R2 or R4. Let’s take a look at R4 now:
R4#show ip route ospf
O E2 192.168.12.0/24 [110/20] via 192.168.34.3, 00:01:14, FastEthernet0/1
     1.0.0.0/24 is subnetted, 1 subnets
O E2    1.1.1.0 [110/20] via 192.168.34.3, 00:01:14, FastEthernet0/1
O E2 192.168.23.0/24 [110/20] via 192.168.34.3, 00:01:14, FastEthernet0/1
The output of R4 is a bit more interesting. Take a close look at the first two entries:
  • 192.168.12.0 /24 with R3 as its next hop.
  • 1.1.1.0 /24 with R3 as its next hop.
R4 used to have RIP routes for these two networks but they have been replaced with these OSPF entries. Is this a problem? It won’t cause any issues in this topology but this is what we call sub-optimal routing:
metric based redistribution before after
We have connectivity but since R4 prefers OSPF (AD 110) over RIP (AD 120) we will send all traffic destined for 192.168.12.0 /24 and 1.1.1.0 /24 through R3. It works, but it’s not optimal.
Why did R4 learn about these routes through R3 and not the other way around (R3 learning those two networks from R4)? R3 and R4 are both redistributing all RIP routes into OSPF so why is there a difference in the output of their routing tables?
It depends on which router has converged first. In my example I started with the redistribution configuration on R3. In this case, R3 will redistribute the RIP routes into OSPF and advertises them to R4. R4 will remove its RIP routes for these networks and installs the OSPF routes.
Since R4 doesn’t have a RIP route for 192.168.12.0 /24 and 1.1.1.0 /24 in its routing table, they can’t be redistributed into OSPF and advertised to R3. If I would have configured R4 first (or reset the OSPF process on R3) then R3 would learn about the redistributed RIP routes through OSPF.
We will forget about this sub-optimal routing issue for now, it won’t cause any issues for this topology and the goal of this lesson is to explain you the redistribution metric problem. In my second post I will explain how to fix this issue. In case you are curious, the short answer is that you have to set the AD of the OSPF entries for 192.168.12.0 /24 and 1.1.1.0 /24 to a value higher than 120. This causes R3 and R4 to use the RIP routes instead of the OSPF routes.
Let’s continue by redistributing OSPF into RIP, that’s where the real fun starts. We’ll do this on R3 and R4:
R3 & R4#
(config)#router rip
config-router)#redistribute ospf 1 metric 1
We redistribute everything from OSPF into RIP. Now take a look what happens with R2:
R2#show ip route rip
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/1] via 192.168.24.4, 00:00:05, FastEthernet0/1
R    192.168.34.0/24 [120/1] via 192.168.24.4, 00:00:05, FastEthernet0/1
                     [120/1] via 192.168.23.3, 00:00:01, FastEthernet0/0
Take a good look at the entry for 1.1.1.0 /24. R2 installed this route with R4 as its next hop…this will cause issues, check out this traceroute:
R2#traceroute 1.1.1.1

Type escape sequence to abort.
Tracing the route to 1.1.1.1

  1 192.168.24.4 1388 msec 620 msec 744 msec
  2 192.168.34.3 2552 msec 2308 msec 1904 msec
  3 192.168.23.2 56 msec 48 msec 48 msec
  4 192.168.24.4 76 msec 76 msec 72 msec
  5 192.168.34.3 112 msec 124 msec 100 msec
  6 192.168.23.2 96 msec 100 msec 92 msec
  7 192.168.24.4 124 msec 140 msec 124 msec
  8 192.168.34.3 156 msec 168 msec 176 msec
  9 192.168.23.2 152 msec 136 msec 156 msec
 10 192.168.24.4 176 msec 184 msec 172 msec
That’s not looking good, we now have a routing loop! Traffic is sent like this:
R2 R3 R4 routing loop
So why exactly is this happening? This is the metric redistribution issue that this lesson is named after. Let’s zoom in on R2:
R2 receives two rip routes
R2 is receiving a routing update for network 1.1.1.0 /24 on two interfaces. The update from R1 has a hop count of 6 while the update from R3 has a hop count of 1. R2 selects the route with the lowest hop count so it installs the route towards R3.
RIP doesn’t see the difference between an “internal” and “redistributed” route so it just selects the one with the lowest metric. What about other protocols like OSPF or EIGRP? Are they also vulnerable to this “metric” problem?
  • OSPF doesn’t have this issue because it always prefers internal “O” and “O IA” routes over O “E1” and “E2” routes.
  • EIGRP doesn’t have this issue because it always prefers “internal” routes (AD 90) over “external” routes (AD 170). The only exception to this rule is when the original route is an EIGRP external route. In this case, both routes have an AD of 170.
Before we start looking at different solutions to fix this problem, let’s pause for a moment, take a deep breath and think about the “core” issue of the problem that we have here. The problem with the metric that R2 is facing happens because we redistributed the 1.1.1.0 /24 network from RIP into OSPF and then back into RIP:
Redistribution OSPF RIP R3 R4
Advertising a prefix from one routing protocol into another and then back into the first routing protocol is a bad idea. There’s no point feeding this information back into the routing protocol that originated the prefix.
You should never redistribute prefixes like this:
Routing Protocol X > Y > X
This is a very important redistribution rule, in fact it’s so important that I’ll add it extra large:
Redistribution Rule: Never advertise prefixes from routing protocol X into Y and then back into X.
If we would prevent R4 (and R3) from redistributing the 1.1.1.0 /24 network back into RIP then R2 would never learn this network with that low hop count and there would be no routing loop.
Keep this redistribution rule in mind while we look at the different solutions to fix this problem…

Solutions

There are many different methods to solve the “metric” redistribution issue that you just witnessed. I’ll show you the different methods and their (dis)advantages. It’s important to know some of the different methods if you are preparing for the CCIE R&S lab. Let’s look at the first method…





Redistribution Filtering

Instead of just redistributing everything from one routing protocol into another, we’ll use some filters to select the prefixes we want to redistribute.
We should only redistribute the prefixes that originated in the routing protocol. For example, when we look at OSPF the only network that originated in OSPF is 192.168.34.0 /24 (the link between R3 and R4). This is the only network that we should redistribute into RIP.
Let me show you how to configure a filter when you redistribute:
R3 & R4#
(config)#ip access-list standard NATIVE_OSPF_ROUTES
(config-std-nacl)#permit 192.168.34.0 0.0.0.255

(config)#route-map NATIVE_OSPF permit 10
(config-route-map)#match ip address NATIVE_OSPF_ROUTES
(config-route-map)#route-map NATIVE_OSPF deny 20

(config)#router rip
(config-router)#redistribute ospf 1 metric 1 route-map NATIVE_OSPF
First we create an access-list that matches the network that we want to redistribute. Secondly we configure a route-map that matches this access-list and finally we attach the route-map to the redistribute command.  This fixes our problem since R2 will never learn about network 1.1.1.0 /24 through R3 or R4. Take a look at its routing table:
R2#show ip route rip
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/6] via 192.168.12.1, 00:00:21, FastEthernet1/0
Above you can see that R2 now uses the original (and correct) route for network 1.1.1.0 /24. Great! We just solved our first redistribution problem.
Instead of using an access-list, you can also use a prefix-list. This might be useful if your CCIE lab has a requirement that tells you not to use access-lists or something. Let me change the route-map:
R3 & R4#
(config)#ip prefix-list NATIVE_OSPF_ROUTES permit 192.168.34.0/24
(config-route-map)#no match ip address NATIVE_OSPF_ROUTES
(config-route-map)#match ip address prefix-list NATIVE_OSPF_ROUTES
The end result will be the same.
Sometimes it helps to use clear ip route * to speed up convergence. It’s also useful to enable debug ip routing to see changes to the routing table on your console.
The solution I just showed you meets the “don’t redistribute routing protocol X > Y > X” rule but it has one limitation…it’s not a very scalable solution. When you advertise new prefixes into OSPF, you’ll have to add them to your access-list or prefix-list.
We attached a route-map to the redistribution from OSPF into RIP but we didn’t do this for redistribution for RIP into OSPF. Technically it doesn’t matter much since OSPF guards itself against this redistribution metric problem…it will always prefer internal prefixes over external prefixes. Still, it would be a good idea to prevent prefixes going from OSPF > RIP > OSPF.
Let’s take a look at another solution which is more scalable…

Redistribution Route Tagging

Instead of manually selecting the prefixes we want to redistribute we can also “tag” the prefixes. What this means is that whenever we redistribute a prefix, it will be tagged. Here’s a illustration:
R3 R4 Route Tagged
Above is an example for network 1.1.1.0 /24. When R3 redistributes this network into OSPF, it will add a tag to it. When R4 sees this tag, it won’t redistribute it from OSPF to RIP.
This way a router will know if the prefix has already been redistributed or not. Let me show you how to do this, first let’s get rid of the route-map we used in the previous example:
R3 & R4#
(config)router rip
(config-router)#no redistribute ospf 1 metric 1 route-map NATIVE_OSPF
(config-router)#redistribute ospf 1 metric 1
I removed the route-map and added the old redistribution command again.
This time we’ll create a route-map that tags all prefixes when they are redistributed from RIP into OSPF. Let’s do this step-by-step:
R3 & R4#
(config)#route-map RIP_TO_OSPF permit 10
(config-route-map)#set tag 100

(config)#router ospf 1
(config-router)#redistribute rip metric 1 subnets route-map RIP_TO_OSPF
First we create a route-map that permits everything and sets the tag to 100. You can pick whatever value you like. Secondly we apply this route-map to the redistribution of RIP prefixes into OSPF. Now take a look at the OSPF database of R3 and R4:
R3#show ip ospf database | begin Type-5
                Type-5 AS External Link States

Link ID         ADV Router      Age         Seq#       Checksum Tag
1.1.1.0         192.168.34.4    6           0x80000001 0x0001BB 100
192.168.12.0    192.168.34.3    107         0x80000004 0x00EE59 100
192.168.23.0    192.168.34.3    107         0x80000004 0x0075C7 100
192.168.24.0    192.168.34.4    59          0x80000004 0x0064D6 100
R4#show ip ospf database | begin Type-5
                Type-5 AS External Link States

Link ID         ADV Router      Age         Seq#       Checksum Tag
192.168.12.0    192.168.34.3    99          0x80000004 0x00EE59 100
192.168.23.0    192.168.34.3    99          0x80000004 0x0075C7 100
192.168.24.0    192.168.34.4    50          0x80000004 0x0064D6 100
You can see that the RIP prefixes have been redistributed into OSPF and are tagged with “100”. Our next move is to tell R3 and R4 that when we redistribute from OSPF into RIP that we have to ignore the prefixes that are tagged with 100. Here’s how to do it:
R3 & R4#
(config)#route-map OSPF_TO_RIP deny 10
(config-route-map)#match tag 100
(config-route-map)#exit
(config)#route-map OSPF_TO_RIP permit 20

(config-route-map)#router rip
(config-router)#redistribute ospf 1 metric 1 route-map OSPF_TO_RIP
We create a new route-map that denies everything with tag 100 while everything else is permitted. This route-map is attached to the redistribute command under the RIP process. This meets our “don’t redistribute from routing protocol X > Y back to X” rule. Take a look at R2:
R2#show ip route rip
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/6] via 192.168.12.1, 00:00:13, FastEthernet1/0
Since R2 never learns about the 1.1.1.0 /24 prefix from OSPF, it will prefer the original route from R1…way to go!
Now you understand route tagging, let me show you an even better solution. We can create a single route-map that can be used for redistribution in both directions. Here’s what it looks like:
R3 & R4#
(config)#route-map TAGGING deny 10
(config-route-map)#match tag 1234
(config-route-map)#exit
(config)#route-map TAGGING permit 20
(config-route-map)#set tag 1234

(config)#router rip
(config-router)#redistribute ospf 1 metric 1 route-map TAGGING

(config)#router ospf 1
(config-router)#redistribute rip subnets route-map TAGGING
This route-map won’t redistribute a prefix if it has a tag of ‘1234’. If the prefix doesn’t have a tag then we will permit it and set the tag to ‘1234’. The route-map has been attached to OSPF and RIP.
The logic behind this route-map is that when something has been tagged then it must have been redistributed already and we should ignore it. If there is no tag, we set one and redistribute the prefix. The end result is the same:
R2#show ip route rip
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/6] via 192.168.12.1, 00:00:06, FastEthernet1/0
Instead of only looking at the routing table, enabling a debug is also a nice method to see in- and outgoing routing updates:
R2#debug ip rip
RIP protocol debugging is on

RIP: received v2 update from 192.168.24.4 on FastEthernet0/1
     192.168.34.0/24 via 0.0.0.0 in 1 hops

RIP: received v2 update from 192.168.23.3 on FastEthernet0/0
     192.168.34.0/24 via 0.0.0.0 in 1 hops
This tells us that R3 and R4 are only redistributing 192.168.34.0 /24 towards R2.
Route tagging is the most convenient method to enforce the redistribution rule of “Never advertise prefixes from routing protocol X into Y and then back into X”. If you advertise new prefixes in RIP or OSPF then they’ll be automatically tagged.
We just used a route-map in combination with an access-list, prefix-list and route tagging. These are the most common methods but you can get pretty creative with the match options that a route-map offers. Some of the examples are matching on the “next hop IP address” or “match interface”. To get an idea of the possibilities, take a look at this Cisco article.
Let’s take a look at some other options to solve our metric redistribution problem…

Redistribution Seed Metric

Forget about the route-maps for now, let’s get back to the original scenario. The problem with R2 accepting the 1.1.1.0 /24 network from R4 (or R3) is that the hop count was “an offer it couldn’t refuse”. Take a look at this picture to refresh your memory:
R2 receives two rip routes
To make R2 prefer the original information from R1 we could change the seed metric. This is kinda a dirty trick but it’s a possibility:
R3 & R4#
(config)#router rip
(config-router)#redistribute ospf 1 metric 10
By setting the hop count for the redistributed routes into RIP to 10, R2 will prefer the information from R1. This solution doesn’t meet our holy redistribution rule and it can be dangerous. When someone increases the hop count on R1 to 11 or higher, we’ll have the same problem again.
Still, on a CCIE R&S lab you can expect anything so it’s good to understand the different possibilities of fixing a problem. Let’s take a look at the next solution which is similar…

Offset-List

In the previous solution we changed the seed metric. What if R3 and R4 are pre-configured routers and outside of our control? When this is the case, we can use an offset-list on R2 to increase the metric:
R2 offset-list rip
Here’s what the configuration looks like:
R2(config)#ip access-list standard LOOPBACK_R1
R2(config-std-nacl)#permit 1.1.1.0 0.0.0.255
First we create an access-list that matches the 1.1.1.0 /24 network. Our next move is to attach it to an offset-list:
R2(config)#router rip
R2(config-router)#offset-list LOOPBACK_R1 in 10 FastEthernet0/1
The metric from R1 will now be better than whatever R3 or R4 are offering:
R2#show ip route rip
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/6] via 192.168.12.1, 00:00:13, FastEthernet1/0
Just like changing the seed metric, this is a fix that could be used when R3 and R4 are outside of our control. Let’s check out another solution…

Distribute-list Filtering

To prevent routes from entering the routing table, we can also use a distribute-list. When we get back to the original problem, this is what the routing table of R2 looks like:
R2#show ip route rip
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/1] via 192.168.23.3, 00:00:09, FastEthernet0/0
Above you can see that R2 accepted the 1.1.1.0 /24 prefix from R3. We can use distribute-lists to block certain routing updates:
R2 distribute list not accepting network
Here’s how to configure this:
R2(config)#ip access-list standard BLOCK_LOOPBACK_R1
R2(config-std-nacl)#deny 1.1.1.0 0.0.0.255
R2(config-std-nacl)#permit any
I will use an access-list to filter the prefix but you can use some other options as well:
R2(config-router)#distribute-list ?
  <1-199>      IP access list number
  <1300-2699>  IP expanded access list number
  WORD         Access-list name
  gateway      Filtering incoming updates based on gateway
  prefix       Filter prefixes in routing updates
Let’s attach the access-list to the distribute-list and to the two interfaces pointing to R3 and R4:
R2(config-router)#distribute-list BLOCK_LOOPBACK_R1 in FastEthernet 0/0
R2(config-router)#distribute-list BLOCK_LOOPBACK_R1 in FastEthernet 0/1
R2 will now not accept the information from R3 or R4 and as a result, it will use the information from R1:
R2#show ip route rip
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/6] via 192.168.12.1, 00:00:01, FastEthernet1/0
Problem solved! Just like the seed metric and offset-list solutions, this one doesn’t meet our redistribution rule. However, changing the metric on R1 won’t accept R2’s decisions since it’s not accepting 1.1.1.0 /24 from R3 and R4 at all.
I have one more solution for you, the next one is a bit different!

Administrative Distance

Back to the original problem, R2 has an incorrect route again from R3:
R2#show ip route rip
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/1] via 192.168.23.3, 00:00:12, FastEthernet0/0
We can change the administrative distance for certain prefixes. Let’s do this for network 1.1.1.0 /24:
R2(config)#ip access-list standard LOOPBACK_R1
R2(config-std-nacl)#permit 1.1.1.0 0.0.0.255

R2(config)#router rip
R2(config-router)#distance 255 192.168.23.3 0.0.0.0 LOOPBACK_R1
R2(config-router)#distance 255 192.168.24.4 0.0.0.0 LOOPBACK_R1
When R2 receives RIP information about 1.1.1.0 /24 from R3 or R4 then it will set the administrative distance to 255.  Setting it to a value of 121 or higher would do the job but setting it to 255 prevents R2 from installing it in its routing table at all.
Here’s the end result:
R2#show ip route rip
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/6] via 192.168.12.1, 00:00:13, FastEthernet1/0
Problem solved! Instead of changing the administrative distance on R2, we can also do this on R3 or R4. Let’s get rid of the configuration we just did on R2:
R2(config)#router rip
R2(config-router)#no distance 255 192.168.23.3 0.0.0.0 LOOPBACK_R1
R2(config-router)#no distance 255 192.168.24.4 0.0.0.0 LOOPBACK_R1
Take a look at R3, this is currently the router that is responsible for redistributing 1.1.1.0 /24 into RIP:
R3#show ip route ospf
     1.0.0.0/24 is subnetted, 1 subnets
O E2    1.1.1.0 [110/1] via 192.168.34.4, 00:36:18, FastEthernet0/1
R3 RIP OSPF Route Receive
Let’s increase the administrative distance for 1.1.1.0 /24 on R3 and R4:
R3 & R4#
(config)#ip access-list standard LOOPBACK_R1
(config-std-nacl)# permit 1.1.1.0 0.0.0.255

(config)#router ospf 1
(config-router)#distance 255 0.0.0.0 255.255.255.255 LOOPBACK_R1
R3 and R4 are now unable to install the 1.1.1.0 /24 OSPF route in their routing tables. They are forced to use the RIP information:
R3#show ip route | incl 1.1.1.0
R       1.1.1.0 [120/7] via 192.168.23.2, 00:00:19, FastEthernet0/0
R4#show ip route rip | incl 1.1.1.0
R       1.1.1.0 [120/7] via 192.168.24.2, 00:00:26, FastEthernet0/0
Above you can see that R3 and R4 are now using the information from RIP. Since they don’t have a OSPF entry for this network, they also can’t redistribute it into RIP.
The solution above is also the solution to get rid of the sub-optimal routing problem. By changing the administrative distance, R3 and R4 are forced to use the original routing information instead of the redistributed routing information.
This ensures that R2 won’t learn it:
R2#show ip route rip
     1.0.0.0/24 is subnetted, 1 subnets
R       1.1.1.0 [120/6] via 192.168.12.1, 00:00:12, FastEthernet1/0
Problem solved!

Conclusion

Redistribution is a complex topic…keep the redistribution rule in mind that you should never redistribute routing information from routing protocol X into Y and then back into X. When you do face metric-related redistribution issues, you are now able to fix them using a variety of techniques.
In the next lesson we will look at another redistribution problem that is caused by the administrative distance. If you have any questions, feel free to leave a comment!

No comments:

Post a Comment